Traefik restarts/crashes after Pterodactyl wings request for a list of servers in the panel

For a couple of days, I've been trying hard to have Pterodactyl wings get a list of servers successfully. None of the attempts were successful. On my fourth attempt, I realized that traefik kept restarting after wings sent a request to retrieve a list of servers. Wings returned a 401 HTTP error regarding the Authorization header was missing. I checked the traefik logs and traefik didn't send the debug logs of the incoming HTTP request, indicating that traefik must've restarted. To be sure that the header wasn't missing for other docker services proxied through traefik, I ran them (my rest API services) and the results did not appear them having issues with the Authentication header. These attempts were able to send the debug incoming HTTP request without a problem. So far I'm not sure if traefik randomly crashes with Pterodactyl wings or if traefik is overloaded with listening to too many docker services at once.

Here's my docker compose file:

services:
  database:
    image: mariadb:10.5
    restart: always
    command: --default-authentication-plugin=mysql_native_password
    volumes:
      - "/srv/pterodactyl/database:/var/lib/mysql"
    environment:
      <<: *db-environment
      MYSQL_DATABASE: "panel"
      MYSQL_USER: "pterodactyl"
  cache:
    image: redis:alpine
    restart: always
  panel:
    image: ghcr.io/pterodactyl/panel:latest
    restart: always
    expose:
      - 80
    links:
      - database
      - cache
    volumes:
      - "/srv/pterodactyl/var/:/app/var/"
      - "/srv/pterodactyl/nginx/:/etc/nginx/http.d/"
      - "/srv/pterodactyl/certs/:/etc/letsencrypt/"
      - "/srv/pterodactyl/logs/:/app/storage/logs"
    networks:
      - default
      - traefik-network
    environment:
      <<: [*panel-environment, *mail-environment]
      DB_PASSWORD: *db-password
      APP_ENV: "production"
      APP_ENVIRONMENT_ONLY: "false"
      CACHE_DRIVER: "redis"
      SESSION_DRIVER: "redis"
      QUEUE_DRIVER: "redis"
      REDIS_HOST: "cache"
      DB_HOST: "database"
      DB_PORT: "3306"
    labels:
      traefik.enable: true
      # traefik.http.middlewares.game_daemon-headers.headers.accesscontrolallowheaders: Authorization
      # traefik.http.middlewares.game_daemon-headers.headers.accesscontrolalloworiginlist: "*"
      # traefik.http.routers.game_daemon.entrypoints: websecure
      # traefik.http.routers.game_daemon.middlewares: game_daemon-headers
      traefik.http.middlewares.minecraftpanel-accesscontrol.headers.accesscontrolallowheaders: Authorization
      traefik.http.middlewares.minecraftpanel-accesscontrol.headers.accesscontrolalloworiginlist: "*"

      traefik.http.routers.minecraftpanel-web.rule: Host(`panel.minecraft.ee.lol`) # <== Your Domain Name goes here for the http rule
      traefik.http.routers.minecraftpanel-web.entrypoints: web # <== Defining the entrypoint for http, **ref: line 30
      traefik.http.routers.minecraftpanel-web.middlewares: redirect-to-https # <== This is a middleware to redirect to https
      
      traefik.http.routers.minecraftpanel-secured.rule: Host(`panel.minecraft.ee.lol`) # <== Your Domain Name for the https rule
      traefik.http.routers.minecraftpanel-secured.entrypoints: websecure # <== Defining entrypoint for https, **ref: line 31
      traefik.http.routers.minecraftpanel-secured.middlewares: minecraftpanel-accesscontrol
      traefik.http.routers.minecraftpanel-secured.tls.certresolver: letsencrypt # <== Defining certsresolvers for https
      traefik.http.routers.minecraftpanel-secured.service: minecraftpanel-httpservice
      traefik.http.services.minecraftpanel-httpservice.loadbalancer.server.port: 80
networks:
  default:
    ipam:
      config:
        - subnet: 172.20.0.0/16
  traefik-network:
    external: true

The log that came after wings sent a Http GET request:

{Status:die ID:361b04bf96b77d372a8013b1e054f72d8496ab7bef28d379467ff9cec83ae100 From:ghcr.io/pterodactyl/wings:latest Type:container Action:die Actor:{ID:361b04bf96b77d372a8013b1e054f72d8496ab7bef28d379467ff9cec83ae100 Attributes:map[com.docker.compose.config-hash:6273458a31fcbe5f7592aacfc033379afd5caa2f121999921180912a8b501d3b com.docker.compose.container-number:1 com.docker.compose.depends_on: com.docker.compose.image:sha256:6591782522dadd4cba5ebc208111fdc61c644bae4889f9ff87c65172de263321 com.docker.compose.oneoff:False com.docker.compose.project:wings com.docker.compose.project.config_files:/home/minecraft/minecraft-server/pterodactl/wings/docker-compose.yml com.docker.compose.project.working_dir:/home/minecraft/minecraft-server/pterodactl/wings com.docker.compose.service:wings com.docker.compose.version:2.14.1 exitCode:2 image:ghcr.io/pterodactyl/wings:latest name:wings-wings-1 org.opencontainers.image.created:2022-12-05T00:44:40.428Z org.opencontainers.image.description:The server control plane for Pterodactyl Panel. Written from the ground-up with security, speed, and stability in mind. org.opencontainers.image.licenses:MIT org.opencontainers.image.revision:e4d790ea40185c620049effeb9e4cf64f4c55e62 org.opencontainers.image.source:https://github.com/pterodactyl/wings org.opencontainers.image.title:wings org.opencontainers.image.url:https://github.com/pterodactyl/wings org.opencontainers.image.version:v1.11.0]} Scope:local Time:1673405138 TimeNano:1673405138093098968}" providerName=docker
- This log indicated the error encountered after the wings service failed to retrieve a list of servers through the Pterodactyl panel domain that is proxied through CloudFlare

Which Traefik version do you use? Did you enable Traefik debug log?

Can you share your Traefik static/dynamic configuration and docker-compose.yml file, if used?

I use v2.9.6.

version: "3.8"

services:
  reverse-proxy:
    # The official v2 Traefik docker image
    image: traefik
    # Enables the web UI and tells Traefik to listen to docker
    env_file:
      - .env
    ports:
      - "80:8080"
      - "443:8443"
    command:
      ## API Settings - https://docs.traefik.io/operations/api/, endpoints - https://docs.traefik.io/operations/api/#endpoints ##
      # - --api.insecure=true # <== Enabling insecure api, NOT RECOMMENDED FOR PRODUCTION
      - --api.dashboard=true # <== Enabling the dashboard to view services, middlewares, routers, etc...
      - --api.debug=true # <== Enablnig additional endpoints for debugging and profiling
      ## Log Settings (options: ERROR, DEBUG, PANIC, FATAL, WARN, INFO) - https://docs.traefik.io/observability/logs/ ##
      - --log.level=DEBUG # <== Setting the level of the logs from traefik
      ## Provider Settings - https://docs.traefik.io/providers/docker/#provider-configuration ##
      - --providers.docker=true # <== Enabling docker as the provider for traefik
      - --providers.docker.exposedbydefault=false # <== Don't expose every container to traefik, only expose enabled ones
      - --providers.docker.network=traefik-network # <== Operate on the docker network named web
      ## Entrypoints Settings - https://docs.traefik.io/routing/entrypoints/#configuration ##
      - --entrypoints.web.address=:8080/tcp # <== Defining an entrypoint for port :80 named web
      - --entrypoints.websecure.address=:8443/tcp # <== Defining an entrypoint for https on port :443 named web-secured
      # Allow cloudflare IPs to forward headers
      - --entrypoints.websecure.forwardedHeaders.trustedIPs=173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/13,104.24.0.0/14,172.64.0.0/13,131.0.72.0/22,10.0.0.0/8,172.16.0.0/16,192.168.0.0/16,fc00::/7 # <== Defining an entrypoint for https on port :443 named web-secured
      ## Certificate Settings (Let's Encrypt) -  https://docs.traefik.io/https/acme/#configuration-examples ##
      - --certificatesresolvers.letsencrypt.acme.dnschallenge=true
      - --certificatesresolvers.letsencrypt.acme.dnschallenge.provider=cloudflare
      - --certificatesresolvers.letsencrypt.acme.email=${CERTBOT_EMAIL} # <== Setting email for certs
      - --certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json # <== Defining acme file to store cert information
      ## Log4j vulnerability mitigation
      - --pilot.token=xxx
      - --experimental.plugins.log4shell.modulename=github.com/traefik/plugin-log4shell
      - --experimental.plugins.log4shell.version=v0.1.2
      - --entrypoints.websecure.http.middlewares=log4j-mitigation
      CF_DNS_API_TOKEN: ${CF_UNVERSAL_TOKEN}
      CF_ZONE_API_TOKEN: ${CF_UNVERSAL_TOKEN}
    volumes:
      # So that Traefik can listen to the Docker events
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./letsencrypt:/letsencrypt
    # restart: unless-stopped
    networks:
      - traefik-network
    labels:
      traefik.enable: true
      traefik.http.middlewares.redirect-to-https.redirectscheme.scheme: https
      traefik.http.middlewares.log4j-mitigation.plugin.log4shell.errorCode: 200
      
      # dashboard
      traefik.http.routers.traefik.rule: Host(`${DASHBOARD_HOST}`)
      traefik.http.routers.traefik.entrypoints: websecure
      traefik.http.routers.traefik.service: api@internal
      traefik.http.routers.traefik.tls.certresolver: letsencrypt
networks:
  traefik-network:
    name: traefik-network

Try running another service to see if Traefik works correctly.

services:
  …

  whoami:
    image: traefik/whoami:v1.8
    networks:
      - traefik-network
    labels:
      - traefik.enable=true
      - traefik.http.routers.mywhoami.entrypoints=websecure
      - traefik.http.routers.mywhoami.rule=Host(`traefik.example.com`)
      - traefik.http.routers.mywhoami.tls.certresolver=letsencrypt
      - traefik.http.services.mywhoami.loadbalancer.server.port=80

(post deleted by author)

Did Traefik really crash? What does docker ps tell you?

Have you tried disabling experimental features?

Have you looked at Traefik access logs?

Unfortunately, Traefik was not loading the certificate for the whoami service, leading me to the insecure Cloudflare page:

Great! Forget the post I replied you earlier. The Cloudflare error seemed to become a problem due to invalid SSL certificate.

Traefik didn't crash.

Disabling the log4j experimental feature didn't cease the problem. This was the log that began restarting traefik:

{"ClientAddr":"*****:50002","ClientHost":"[my vps ip]","ClientPort":"50002","ClientUsername":"-","DownstreamContentSize":197,"DownstreamStatus":401,"Duration":94180166,"OriginContentSize":197,"OriginDuration":94152835,"OriginStatus":401,"Overhead":27331,"RequestAddr":"panel.minecraft.ee.lol","RequestContentSize":0,"RequestCount":106,"RequestHost":"panel.minecraft.ee.lol","RequestMethod":"GET","RequestPath":"/api/remote/servers?page=0\u0026per_page=50","RequestPort":"-","RequestProtocol":"HTTP/2.0","RequestScheme":"https","RetryAttempts":0,"RouterName":"minecraftpanel-secured@docker","ServiceAddr":"192.168.16.5:80","ServiceName":"minecraftpanel-httpservice@docker","ServiceURL":{"Scheme":"http","Opaque":"","User":null,"Host":"192.168.16.5:80","Path":"","RawPath":"","OmitHost":false,"ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""},"StartLocal":"2023-01-16T18:47:34.534800012Z","StartUTC":"2023-01-16T18:47:34.534800012Z","TLSCipher":"TLS_AES_128_GCM_SHA256","TLSVersion":"1.3","entryPointName":"websecure","level":"info","msg":"","time":"2023-01-16T18:47:34Z"}

Determining the situation problem, I discovered that going to https://panel.minecraft.ee.lol/api/remote/servers?page=0\u0026per_page=50 on my VPS was making traefik restart after returning. I then tried going to that link on my home IP and no restart was reproduced. Is this a faulty error in the VPS backend?