Traefik sometimes randomly stops reacting

Hey traefik community,

I wanted to ask for some help. I have traefik installed on an unRAID server and it sometimes stops working. That means that I can't reach the dashboard (because of too many redirects) anymore. Moreover, some of my services won't be reachable as well. Here is the odd part, I'm pretty sure that my home assistant container is always reachable but my pihole e.g. won't be reachable at that times. I think when that happens, the docker health of my pihole container shows "unhealthy" which I can't explain either. Does traefik rely on any of the docker healthchecks?
Unfortunately, I can't test all that atm, as traefik currently is working as it is supposed to.

 

In the traefik logs I saw the following message which could be related to that:

panic: Timeout while stopping traefik, killing instance ✝

goroutine 5772 [running]:
github.com/traefik/traefik/v3/pkg/server.(*Server).Close.func1({0x70a24a8, 0xc000747110})
        github.com/traefik/traefik/v3/pkg/server/server.go:87 +0x9d
created by github.com/traefik/traefik/v3/pkg/server.(*Server).Close in goroutine 1
        github.com/traefik/traefik/v3/pkg/server/server.go:82 +0x96

 

In the unRAID docker config, I just bind volumes and add the docker.sock, my config files are the following:
docker label-file:

# > Traefik
traefik.enable=true
traefik.http.routers.api.service=api@internal
traefik.http.routers.traefik.middlewares=traefikAuth@file,default@file

config.yml:

# Traefik dynamic configuration file
# See https://www.benjaminrancourt.ca/a-complete-traefik-configuration/
# and https://blog.lrvt.de/nginx-proxy-manager-versus-traefik/
# and https://doc.traefik.io/traefik/getting-started/configuration-overview/#the-dynamic-configuration

http:
  middlewares:
    # A basic authentification middleware, to protect the Traefik dashboard to anyone except myself
    # Use with traefik.http.routers.myRouter.middlewares: "traefikAuth@file"
    traefikAuth:
      basicAuth:
        removeHeader: true
        usersFile: "/etc/traefik/usersfile"               # Contains the users with a hashed password

    # Recommended default middleware for most of the services
    # Use with traefik.http.routers.myRouter.middlewares: "default@file"
    # Equivalent of traefik.http.routers.myRouter.middlewares: "default-security-headers@file,error-pages@file,gzip@file"
    # Or enable as default for an entryPoint in the static config
    default:
      chain:
        middlewares:
          - default-security-headers
          - gzip

    # Add automatically some security headers
    # Use with traefik.http.routers.myRouter.middlewares: "default-security-headers@file"
    default-security-headers:
      headers:
        browserXssFilter: true                            # X-XSS-Protection=1; mode=block
        contentTypeNosniff: true                          # X-Content-Type-Options=nosniff
        forceSTSHeader: true                              # Add the Strict-Transport-Security header even when the connection is HTTP
        frameDeny: true                                   # X-Frame-Options=deny
        referrerPolicy: "strict-origin-when-cross-origin"
        sslRedirect: true                                 # Allow only https requests
        stsIncludeSubdomains: true                        # Add includeSubdomains to the Strict-Transport-Security header
        stsPreload: true                                  # Add preload flag appended to the Strict-Transport-Security header
        stsSeconds: 63072000                              # Set the max-age of the Strict-Transport-Security header (63072000 = 2 years)

    # Enables the GZIP compression (https://docs.traefik.io/middlewares/compress/)
    #   if the response body is larger than 1400 bytes
    #   if the Accept-Encoding request header contains gzip
    #   if the response is not already compressed (Content-Encoding is not set)
    # Use with traefik.http.routers.myRouter.middlewares: "gzip@file"
    gzip:
      compress: {}

tls:
  stores:
    # Add bassmann.intra certificate to be the standard one
    default:
      defaultCertificate:
        certFile: xxxx.crt
        keyFile: xxxx.pem
  options:
    default:
      minVersion: "VersionTLS13"                          # Minimum TLS Version

traefik.yml:

# Traefik static configuration file (/etc/traefik/traefik.yml)
# See https://www.benjaminrancourt.ca/a-complete-traefik-configuration/
# and https://blog.lrvt.de/nginx-proxy-manager-versus-traefik/
# and https://doc.traefik.io/traefik/getting-started/configuration-overview/#the-static-configuration
# and https://doc.traefik.io/traefik/reference/static-configuration/cli/

api:
  dashboard: true                             # Enable the dashboard
  insecure: false                             # Don't expose the dashboard through :8080

global:
  checknewversion: true                       # Periodically check if a new version has been released.
  sendAnonymousUsage: true

entryPoints:
  http:
    address: :80                              # Create the HTTP entrypoint on port 80
    http:
      redirections:                           # HTTPS redirection (80 to 443)
        entryPoint:
          to: https                           # The target element
          scheme: https                       # The redirection target scheme
  https:
    address: :443                             # Create the HTTPS entrypoint on port 443
    asDefault: true                           # Use https as default entrypoint
    http:
      tls: true                               # Enable TLS by default
      middlewares:
        - default@file                        # Set 'default' middleware from config.yml as default

# serversTransports:
#   insecureSkipVerify: true                              # Allow insecure backend connections

log:
  level: INFO                                 # DEBUG, INFO, WARN, ERROR, FATAL, PANIC
  # Uncomment if normal logs should be written to a file instead of std_out thus not being able to view in unRAID console
  # filePath: "/logs/traefik.log"               # Path to the log file

accessLog:
  filePath: "/logs/access.log"                # Path to the file
  format: json                                
  bufferingSize: 0                            # How many logs should be buffered until written to the file
  filters:
    statusCodes:                              # Limit logging to the following codes
      - "200"
      - "300-302"
    retryAttempts: true                       # Keep the access logs when at least one retry has happened
    minDuration: "10ms"                       # Keep access logs when requests take longer than the specified duration

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"   # Listen to the UNIX Docker socket
    exposedByDefault: false                   # Only expose container that are explicitly enabled (using label traefik.enabled)
    network: "traefik-net"                    # Default network to use for connections to all containers.
    defaultRule: "Host(`{{ .Name }}.xxxx`)" # If no URL is given, use the Docker service name
  
  file:
    filename: "/etc/traefik/config.yml"       # Link to the dynamic configuration
    watch: true                               # Watch for modifications
  providersThrottleDuration: 10               # Configuration reload frequency

 

Thanks for any help in advance :slight_smile:

Something needs to send "too many redirects", probably Traefik. Enable and check Traefik debug log (doc) and Traefik access log in JSON format (doc).

Also check browser developer tools network tab.

I read that sometimes people experience issues when using compress.

I actually just have the problem again. Although this time, the only service not reacting is the dashboard. Pihole e.g. works completely and is reachable. So maybe these two are separate problems which sometimes occur at the same time.

So my access log spams 15 lines of me trying to connect to the dashboard although I only tried it once.

{"ClientAddr":"192.168.x.x:65335","ClientHost":"192.168.x.x","ClientPort":"65335","ClientUsername":"xxxx","DownstreamContentSize":17,"DownstreamStatus":301,"Duration":2018269,"OriginContentSize":17,"OriginDuration":617355,"OriginStatus":301,"Overhead":1400914,"RequestAddr":"traefik.xxxx","RequestContentSize":0,"RequestCount":33,"RequestHost":"traefik.xxxx","RequestMethod":"GET","RequestPath":"/","RequestPort":"-","RequestProtocol":"HTTP/2.0","RequestScheme":"https","RetryAttempts":0,"RouterName":"traefik@docker","ServiceAddr":"10.0.4.2:80","ServiceName":"traefik@docker","ServiceURL":"http://10.0.4.2:80","StartLocal":"2025-01-17T15:24:11.09062467+01:00","StartUTC":"2025-01-17T14:24:11.09062467Z","TLSCipher":"TLS_AES_128_GCM_SHA256","TLSVersion":"1.3","entryPointName":"https","level":"info","msg":"","time":"2025-01-17T15:24:11+01:00"}

Except for the "Duration", "OriginDuration", "Overhead" and the timestamps, everything of these 15 messages stayed the same.

The log spams the following:

2025-01-17T15:19:45+01:00 DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:228 > Serving default certificate for request: "traefik.xxxx"
2025-01-17T15:19:45+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/compress/compress.go:127 > Unable to parse MIME type error="mime: no media type" middlewareName=gzip@file middlewareType=Compress
2025-01-17T15:19:45+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/auth/basic_auth.go:89 > Authentication succeeded middlewareName=traefikAuth@file middlewareType=BasicAuth
2025-01-17T15:19:45+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/auth/basic_auth.go:97 > Removing authorization header middlewareName=traefikAuth@file middlewareType=BasicAuth
2025-01-17T15:19:45+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/compress/compress.go:127 > Unable to parse MIME type error="mime: no media type" middlewareName=gzip@file middlewareType=Compress
2025-01-17T15:19:45+01:00 DBG github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:207 > Service selected by WRR: 3612d554f9f36a5c

That on the other hand could mean that you are right about the compressing, as it fails here.

What do you think?

Just remove compress and try it :slightly_smiling_face:

It seems your Traefik labels are missing a router rule, compare to simple Traefik example:

    labels:
      - traefik.enable=true
      - traefik.http.routers.mydashboard.rule=Host(`traefik.example.com`)
      - traefik.http.routers.mydashboard.service=api@internal
      - traefik.http.routers.mydashboard.middlewares=myauth
      - traefik.http.middlewares.myauth.basicauth.users=test:$$apr1$$H6uskkkW$$IgXLP6ewTrSuBkTrqE8wj/

Actually, the router rule is getting set by the default rule at the bottom in my traefik.yml:

providers:
  docker:
    ...
    defaultRule: "Host(`{{ .Name }}.xxxx`)" # If no URL is given, use the Docker service name

So in my case, the name of my docker container is traefik which translates my router rule to traefik.xxxx with xxxx being censored for private reasons.

For the compress part: the part with the wrong MIME-Type might have been another problem but I now completely deleted the gzip and am still getting the same error. I also removed the default-security-headers which also had no effect on the error.
Currently I'm trying it without the traefikAuth and that didn't throw the error so far. But as I said, sometimes the error occurs and at other times it wont. But still, do you see something wrong about the traefikAuth in my config.yml?

indicates the dashboard application wants to redirect you to a different page.

It’s probably sending the client/browser from / to /dashboard/.

Check browser developer tools network tab to see details.

I checked that whilst I had the problem. It told me it was redirecting from / to / which I thought would be weird.
Anyhow, I can't test this atm, as the dashboard is still working as it is supposed to. So far, I think that honestly something about the traefikAuth could be wrong or broken. Again, do you see something I do wrong here?