Traefik is refusing to proxy to a specific service, the http request reaches the server but traefik does not pass it to the service.
Service compose:
version: '3.9'
networks:
default:
name: 'traefik'
external: true
services:
nekosbestapi:
image: nekosbest-api:latest
environment:
- DATA_API_ADDRESS=http://internal-data-api
- STATS_API_ADDRESS=http://internal-stats-api
- ASPNETCORE_URLS=http://[::]:5678
ports:
- '5678'
labels:
- "traefik.enable=true"
- "traefik.http.routers.nekosbestapi.rule=Host(`${TRAEFIK_DOMAIN}`) && PathPrefix(`/api/v3`)"
- "traefik.http.routers.nekosbestapi.entryPoints=web"
- "traefik.http.services.nekosbestapi.loadbalancer.server.port=5678"
Traefik conf:
providers:
docker:
watch: true
exposedByDefault: false
# For some reason everything breaks when this option is enabled
# swarmMode: false
file:
directory: /etc/traefik/conf.d
watch: true
entryPoints:
web:
address: :80
http:
#middlewares:
# - cloudflarewarp@file
api:
dashboard: true
insecure: true
metrics:
prometheus:
addRoutersLabels: true
addServicesLabels: true
experimental:
plugins:
cloudflarewarp:
moduleName: "github.com/BetterCorp/cloudflarewarp"
version: "v1.3.3"
log:
level: DEBUG
Traefik logs on service create:
Provider event received {Status:start ID:afcf77ed481eb93742b445ee176c36881c05bcc384dd291cf3f4daf5d8544ea5 From:nekosbest-api:latest Type:container Action:start Actor:{ID:afcf77ed481eb93742b445ee176c36881c05bcc384dd291cf3f4daf5d8544ea5 Attributes:map[com.docker.stack.namespace:nekosbest com.docker.swarm.node.id:wjw9izljg7f03k1nr1tvxbmsl com.docker.swarm.service.id:qx8kctljtepvhyg88wb9c6mq3 com.docker.swarm.service.name:nekosbest_nekosbestapi com.docker.swarm.task: com.docker.swarm.task.id:ho5iqzrs3pukmji8u48lgewmu com.docker.swarm.task.name:nekosbest_nekosbestapi.1.ho5iqzrs3pukmji8u48lgewmu image:nekosbest-api:latest name:nekosbest_nekosbestapi.1.ho5iqzrs3pukmji8u48lgewmu traefik.enable:true traefik.http.routers.nekosbestapi.entryPoints:web traefik.http.routers.nekosbestapi.rule:Host(`censored.domain`) && PathPrefix(`/api/v3`) traefik.http.services.nekosbestapi.loadbalancer.server.port:5678]} Scope:local Time:1695735192 TimeNano:1695735192485357853} providerName=docker
docker service ls output:
Received http request:
GET /api/v3/ping HTTP/1.1
Host: censored.domain
Connection: Keep-Alive
accept-encoding: gzip
X-Forwarded-For: XXXXX
CF-RAY: XXXXX
X-Forwarded-Proto: https
CF-Visitor: {"scheme":"https"}
user-agent: insomnia/2023.5.8
accept: */*
CDN-Loop: cloudflare
CF-Connecting-IP: XXXX
CF-IPCountry: XXX
Note: We have services that traefik does proxy correctly to, they are together with traefik in the compose file, here is one of their configs:
portainer:
image: "portainer/portainer-ee:2.19.0"
command: -H unix:///var/run/docker.sock # --log-level=DEBUG # To enable DEBUG logging (undocumented option)
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- portainer_data:/data
labels:
- "traefik.enable=true"
- "traefik.http.routers.portainer.rule=Host(`sysnya.${TRAEFIK_DOMAIN}`) && PathPrefix(`/portainer`)"
- "traefik.http.routers.portainer.middlewares=portainer-stripprefix@file"
- "traefik.http.routers.portainer.service=portainer"
- "traefik.http.routers.portainer.entryPoints=web"
- "traefik.http.services.portainer.loadbalancer.server.port=9000"
What’s the response to the request?
What is Traefik Dashboard telling you?
I personally prefer to use an explicit network (not default), see simple Traefik Swarm example.
Additional thought: how many containers are you running? Any Docker network only allows for ~250 IPs by default.
The response I should get from the url I'm testing (https://censored.domain/api/v3/ping) is a 204 empty response, I'm getting a timeout.
The service is online and working correctly.
The traefik dashboard shows the proxy correctly
There are about 7 containers running.
Please share your Traefik static and dynamic config, and Traefik's docker-compose.yml.
traefik.yml
providers:
docker:
watch: true
exposedByDefault: false
# For some reason everything breaks when this option is enabled
# swarmMode: false
file:
directory: /etc/traefik/conf.d
watch: true
entryPoints:
web:
address: :80
http:
#middlewares:
# - cloudflarewarp@file
api:
dashboard: true
# TODO: Consider removing this and make the dashboard (http://localhost:8080) accessible only via a VPN
# We could set up Wireguard and create a «fake domain» pointing to localhost:8080 by using a custom DNS server for easy access
insecure: true
metrics:
prometheus:
addRoutersLabels: true
addServicesLabels: true
experimental:
plugins:
cloudflarewarp:
moduleName: "github.com/BetterCorp/cloudflarewarp"
version: "v1.3.3"
log:
level: DEBUG
conf.d/middlewares/auth.yml
version: "3.8"
http:
middlewares:
admin-auth:
basicauth:
removeHeader: true
users:
- "USERNAME:XXXX"
registry-auth:
basicauth:
removeHeader: true
users:
- "USERNAME:XXXX"
conf.d/middlewares/general.yml
version: "3.8"
http:
middlewares:
cloudflarewarp:
plugin:
cloudflarewarp:
disableDefault: false
# Public services should use this middleware to prevent big bodies
limit-body:
buffering:
maxRequestBodyBytes: 2000000 # 2 Megabytes
# Used by admin > registry (unused currently)
registry-headers:
headers:
customRequestHeaders:
Docker-Distribution-Api-Version: registry/2.0
X-Forwarded-Proto: https
customResponseHeaders:
Docker-Distribution-Api-Version: registry/2.0
conf.d/middlewares/strip-prefix.yml
version: "3.8"
http:
middlewares:
traefik-stripprefix:
stripprefix:
prefixes:
- "/traefik"
portainer-stripprefix:
stripprefix:
prefixes:
- "/portainer"
grafana-stripprefix:
stripprefix:
prefixes:
- "/grafana"
traefik docker-compose (partial)
version: "3.8"
networks:
default:
name: traefik
external: true
volumes:
grafana_data:
external: true
portainer_data:
external: true
services:
reverse-proxy:
image: "traefik:v3.0"
ports:
- "80:80"
- "443:443"
- "8080:8080"
labels:
- "traefik.enable=true"
- "traefik.http.routers.traefik-dashboard.entryPoints=web"
- "traefik.http.routers.traefik-dashboard.service=api@internal"
- "traefik.http.routers.traefik-dashboard.middlewares=admin-auth@file, traefik-stripprefix@file"
- "traefik.http.routers.traefik-dashboard.rule=Host(`XXX.${TRAEFIK_DOMAIN:?TRAEFIK_DOMAIN is not set}`) && (PathPrefix(`/traefik`) || Header(`Referer`, `https://XXX.${TRAEFIK_DOMAIN}/traefik/dashboard/`))"
- "traefik.http.services.traefik-dashboard.loadbalancer.server.port=9999" # Any port will work here
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:ro"
- "./traefik/:/etc/traefik/:ro"
Interesting trick to use Header(`Referer`, for Traefik Dashboard.
I thought you might have a longer rule like
Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
that would have had higher priority over
Host(`${TRAEFIK_DOMAIN}`) && PathPrefix(`/api/v3`)
We made sure that all rules are unique to their service, nothing can collide.
In addition, the API responds to itself within its container (via docker exec ) and the issue persists even without CF's proxy (by making a request using the server's public IP address and setting the Host header)
What infra & distribution are you running on?
We are on Ubuntu 22.04.3 (GNU/Linux 5.15.0-83-generic x86_64).
No VMs.
No firewall rules.
No VPNs.
In addition, it worked before. So idk why now it decides to give up (no settings were changed)
So the only difference is that the non-working service is in a separate compose file? Have you tried placing it in the same compose file? Have you tried using a dedicated Docker network, not default? See simple Traefik Swarm example.
So I moved it to the same compose, restarted the stack, and it worked.
Then I restarted it again and it stopped working...
I don’t know what special handling Docker has for default, that might just not work.
Try an extra network, check simple Traefik Swarm example.
The file is now using dedicated networks, no change. That specific service does not reply (everything else in the file does)
Change image to traefik/whoami:v1.10 to see if its in general or just with your specific image, you can set the internal port used with env WHOAMI_PORT_NUMBER. (Doc)
