I'm utilizing Podman on my Fedora server (192.168.10.200) where Traefik is working okay, but after adding new entries for containers hosted on the TrueNAS (192.168.10.20), Traefik is not redirecting correctly. I am able to get to my TrueNAS webpage and Plex which are hosted on my TrueNAS system through Traefik which were working fine before this issue arose.
I'm utilizing Docker on my TrueNAS server where I'm trying to migrate my services to. After troubleshooting some more yesterday, I'm experiencing issues with Traefik on both systems where it appears the dynamic file is not working properly.
I'm currently routing most traffic through my TrueNAS server at 192.168.10.20 via DNS. I currently have DNS for my "tdarr.domain.net" and "sonarr4k.mydomain.net" entries routing through my TrueNAS server. Sonarr4k was an established service on my Fedora server which I moved to TrueNAS and is now giving a "Bad Gateway" page. Tdarr is a new service only established on TrueNAS which is giving me a 404 error no matter which Traefik instance (Fedora or TrueNAS) I run it through.
Here is my dashboard running through my TrueNAS instance. Most containers still live on my Fedora server which are working, so I'm focusing on those that are not which are tdarr and sonarr4k, both hosted on the TrueNAS system.
I'm hopeful these logs show that Traefik is not updating my configuration as it shows above. They clearly show Traefik is trying to reach my Fedora server instead of my TrueNAS server IP.
I cannot enable Debug for my access log as it crashes the container and it won't start with that line item. I already had debug enabled for traefik.log via my requested configuration. If I check my access.log for tdarr.mydomain.net, there are no entries at all. But if I check for sonarr4k.galmotech.net, I see new entries pointing to my "old" Fedora server IP 192.168.10.200 instead of 192.168.10.20 of my TrueNAS that I've moved the container to.
Here are those entries:
Line 33: 172.16.0.1 - - [27/Jan/2025:14:09:56 +0000] "GET / HTTP/2.0" 502 11 "-" "-" 37 "sonarr4k@file" "http://192.168.10.200:8990" 1ms
Line 35: 172.16.0.1 - - [27/Jan/2025:14:09:56 +0000] "GET /favicon.ico HTTP/2.0" 502 11 "-" "-" 38 "sonarr4k@file" "http://192.168.10.200:8990" 0ms
Line 36: 192.168.10.200 - - [27/Jan/2025:14:10:00 +0000] "GET /api/v3/queue?apikey=a2c7a0399be743f3a8faf4d5756fdf79&includeEpisode=true HTTP/1.1" 502 11 "-" "-" 42 "sonarr4k@file" "http://192.168.10.200:8990" 1ms
Here are sonarr4k records from traefik.log:
Line 186128: 2025-01-27T09:09:34-05:00 WRN github.com/traefik/traefik/v3/pkg/provider/file/file.go:422 > HTTP router already configured, skipping filename=routers.yml providerName=file routerName=sonarr4k
Line 186168: 2025-01-27T09:09:34-05:00 WRN github.com/traefik/traefik/v3/pkg/provider/file/file.go:438 > HTTP service already configured, skipping filename=services.yml providerName=file serviceName=sonarr4k
Line 186187: 2025-01-27T09:09:34-05:00 DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{"middlewares":{"addprefix-pihole1":{"redirectRegex":{"regex":"^https?://pihole1.mydomain.net/$","replacement":"https://pihole1.mydomain.net/admin/"}},"addprefix-pihole2":{"redirectRegex":{"regex":"^https?://pihole2.mydomain.net/$","replacement":"https://pihole2.mydomain.net/admin/"}},"allow-origin-header":{"headers":{"accessControlAllowHeaders":["*"],"accessControlAllowMethods":["GET","OPTIONS","PUT"],"accessControlAllowOriginList":["https://*.mydomain.net","https://gameon.mydomain.net","https://gameona.mydomain.net"],"accessControlMaxAge":100,"addVaryHeader":true}},"authelia":{"forwardAuth":{"address":"http://192.168.10.200:9091/api/verify?rd=https://login.mydomain.net","authResponseHeaders":["adminGroup","Remote-User","Remote-Groups"],"maxBodySize":-1,"trustForwardHeader":true}},"chain-authelia":{"chain":{"middlewares":["middlewares-rate-limit","nextcloud-middlewares-secure-headers","authelia"]}},"chain-no-auth":{"chain":{"middlewares":["middlewares-rate-limit","middlewares-secure-headers"]}},"default-headers":{"headers":{"browserXssFilter":true,"contentTypeNosniff":true,"customFrameOptionsValue":"SAMEORIGIN","customRequestHeaders":{"X-Forwarded-Proto":"https"},"forceSTSHeader":true,"frameDeny":true,"referrerPolicy":"strict-origin-when-cross-origin","stsIncludeSubdomains":true,"stsPreload":true,"stsSeconds":15552000}},"https-redirect":{"redirectScheme":{"permanent":true,"port":"443","scheme":"https"}},"https-redirect-noport":{"redirectScheme":{"permanent":true,"scheme":"https"}},"middlewares-rate-limit":{"rateLimit":{"average":100,"burst":50,"period":"1s"}},"my-auth":{"basicAuth":{"headerField":"X-WebAuth-User","removeHeader":true,"users":["username:${pass}"]}},"nextcloud-middlewares-secure-headers":{"headers":{"accessControlMaxAge":100,"browserXssFilter":true,"contentTypeNosniff":true,"customFrameOptionsValue":"SAMEORIGIN","customResponseHeader ...
Line 186222: 2025-01-27T09:09:34-05:00 DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:313 > Creating load-balancer entryPointName=websecure routerName=sonarr4k@file serviceName=sonarr4k@file
Line 186223: 2025-01-27T09:09:34-05:00 DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:350 > Creating server entryPointName=websecure routerName=sonarr4k@file serverName=9bde6258e42b47ab serviceName=sonarr4k@file target=http://192.168.10.200:8990
Line 186390: 2025-01-27T09:09:34-05:00 DBG github.com/traefik/traefik/v3/pkg/server/router/tcp/manager.go:237 > Adding route for sonarr4k.mydomain.net with TLS options default entryPointName=websecure
Line 186563: 2025-01-27T09:09:34-05:00 DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:313 > Creating load-balancer entryPointName=websecure routerName=sonarr4k@file serviceName=sonarr4k@file
Line 186565: 2025-01-27T09:09:34-05:00 DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:350 > Creating server entryPointName=websecure routerName=sonarr4k@file serverName=9bde6258e42b47ab serviceName=sonarr4k@file target=http://192.168.10.200:8990
Line 186679: 2025-01-27T09:09:34-05:00 DBG github.com/traefik/traefik/v3/pkg/server/router/tcp/manager.go:237 > Adding route for sonarr4k.mydomain.net with TLS options default entryPointName=websecure
Here are tdarr log entries from only traefik.log. As mentioned, there are no logs in access.log file for this service/router:
Line 186187: 2025-01-27T09:09:34-05:00 DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{"middlewares":{"addprefix-pihole1":{"redirectRegex":{"regex":"^https?://pihole1.mydomain.net/$","replacement":"https://pihole1.mydomain.net/admin/"}},"addprefix-pihole2":{"redirectRegex":{"regex":"^https?://pihole2.mydomain.net/$","replacement":"https://pihole2.mydomain.net/admin/"}},"allow-origin-header":{"headers":{"accessControlAllowHeaders":["*"],"accessControlAllowMethods":["GET","OPTIONS","PUT"],"accessControlAllowOriginList":["https://*.mydomain.net","https://gameon.mydomain.net","https://gameona.mydomain.net"],"accessControlMaxAge":100,"addVaryHeader":true}},"authelia":{"forwardAuth":{"address":"http://192.168.10.200:9091/api/verify?rd=https://login.mydomain.net","authResponseHeaders":["adminGroup","Remote-User","Remote-Groups"],"maxBodySize":-1,"trustForwardHeader":true}},"chain-authelia":{"chain":{"middlewares":["middlewares-rate-limit","nextcloud-middlewares-secure-headers","authelia"]}},"chain-no-auth":{"chain":{"middlewares":["middlewares-rate-limit","middlewares-secure-headers"]}},"default-headers":{"headers":{"browserXssFilter":true,"contentTypeNosniff":true,"customFrameOptionsValue":"SAMEORIGIN","customRequestHeaders":{"X-Forwarded-Proto":"https"},"forceSTSHeader":true,"frameDeny":true,"referrerPolicy":"strict-origin-when-cross-origin","stsIncludeSubdomains":true,"stsPreload":true,"stsSeconds":15552000}},"https-redirect":{"redirectScheme":{"permanent":true,"port":"443","scheme":"https"}},"https-redirect-noport":{"redirectScheme":{"permanent":true,"scheme":"https"}},"middlewares-rate-limit":{"rateLimit":{"average":100,"burst":50,"period":"1s"}},"my-auth":{"basicAuth":{"headerField":"X-WebAuth-User","removeHeader":true,"users":["username:${pass}"]}},"nextcloud-middlewares-secure-headers":{"headers":{"accessControlMaxAge":100,"browserXssFilter":true,"contentTypeNosniff":true,"customFrameOptionsValue":"SAMEORIGIN","customResponseHeader ...
Line 186330: 2025-01-27T09:09:34-05:00 DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:313 > Creating load-balancer entryPointName=websecure routerName=tdarr@file serviceName=tdarr@file
Line 186331: 2025-01-27T09:09:34-05:00 DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:350 > Creating server entryPointName=websecure routerName=tdarr@file serverName=72c666012f67cc01 serviceName=tdarr@file target=https://192.168.10.20:30028
Line 186385: 2025-01-27T09:09:34-05:00 DBG github.com/traefik/traefik/v3/pkg/server/router/tcp/manager.go:237 > Adding route for tdarr.mydomain.net with TLS options default entryPointName=websecure
Line 186474: 2025-01-27T09:09:34-05:00 DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:313 > Creating load-balancer entryPointName=websecure routerName=tdarr@file serviceName=tdarr@file
Line 186475: 2025-01-27T09:09:34-05:00 DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:350 > Creating server entryPointName=websecure routerName=tdarr@file serverName=72c666012f67cc01 serviceName=tdarr@file target=https://192.168.10.20:30028
Line 186668: 2025-01-27T09:09:34-05:00 DBG github.com/traefik/traefik/v3/pkg/server/router/tcp/manager.go:237 > Adding route for tdarr.mydomain.net with TLS options default entryPointName=websecure
If you need anything else or have recommendations I can certainly provide. Thanks for your help!
