Traefik refuses to proxy to a specific service

Traefik is refusing to proxy to a specific service, the http request reaches the server but traefik does not pass it to the service.

Service compose:

version: '3.9'

networks:
  default:
    name: 'traefik'
    external: true

services:
  nekosbestapi:
    image: nekosbest-api:latest
    environment:
      - DATA_API_ADDRESS=http://internal-data-api
      - STATS_API_ADDRESS=http://internal-stats-api
      - ASPNETCORE_URLS=http://[::]:5678
    ports:
      - '5678'
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.nekosbestapi.rule=Host(`${TRAEFIK_DOMAIN}`) && PathPrefix(`/api/v3`)"
      - "traefik.http.routers.nekosbestapi.entryPoints=web"
      - "traefik.http.services.nekosbestapi.loadbalancer.server.port=5678"

Traefik conf:

providers:
    docker:
        watch: true
        exposedByDefault: false
        # For some reason everything breaks when this option is enabled
        # swarmMode: false
    file:
        directory: /etc/traefik/conf.d
        watch: true

entryPoints:
    web:
        address: :80
        http:
            #middlewares:
            #    - cloudflarewarp@file

api:
    dashboard: true
    insecure: true

metrics:
    prometheus:
        addRoutersLabels: true
        addServicesLabels: true

experimental:
  plugins:
    cloudflarewarp:
      moduleName: "github.com/BetterCorp/cloudflarewarp"
      version: "v1.3.3"

log:
  level: DEBUG

Traefik logs on service create:

Provider event received {Status:start ID:afcf77ed481eb93742b445ee176c36881c05bcc384dd291cf3f4daf5d8544ea5 From:nekosbest-api:latest Type:container Action:start Actor:{ID:afcf77ed481eb93742b445ee176c36881c05bcc384dd291cf3f4daf5d8544ea5 Attributes:map[com.docker.stack.namespace:nekosbest com.docker.swarm.node.id:wjw9izljg7f03k1nr1tvxbmsl com.docker.swarm.service.id:qx8kctljtepvhyg88wb9c6mq3 com.docker.swarm.service.name:nekosbest_nekosbestapi com.docker.swarm.task: com.docker.swarm.task.id:ho5iqzrs3pukmji8u48lgewmu com.docker.swarm.task.name:nekosbest_nekosbestapi.1.ho5iqzrs3pukmji8u48lgewmu image:nekosbest-api:latest name:nekosbest_nekosbestapi.1.ho5iqzrs3pukmji8u48lgewmu traefik.enable:true traefik.http.routers.nekosbestapi.entryPoints:web traefik.http.routers.nekosbestapi.rule:Host(`censored.domain`) && PathPrefix(`/api/v3`) traefik.http.services.nekosbestapi.loadbalancer.server.port:5678]} Scope:local Time:1695735192 TimeNano:1695735192485357853} providerName=docker

docker service ls output:

Received http request:

GET /api/v3/ping HTTP/1.1
Host: censored.domain
Connection: Keep-Alive
accept-encoding: gzip
X-Forwarded-For: XXXXX
CF-RAY: XXXXX
X-Forwarded-Proto: https
CF-Visitor: {"scheme":"https"}
user-agent: insomnia/2023.5.8
accept: */*
CDN-Loop: cloudflare
CF-Connecting-IP: XXXX
CF-IPCountry: XXX

Note: We have services that traefik does proxy correctly to, they are together with traefik in the compose file, here is one of their configs:

  portainer:
    image: "portainer/portainer-ee:2.19.0"
    command: -H unix:///var/run/docker.sock # --log-level=DEBUG # To enable DEBUG logging (undocumented option)
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - portainer_data:/data
    labels:
      - "traefik.enable=true"

      - "traefik.http.routers.portainer.rule=Host(`sysnya.${TRAEFIK_DOMAIN}`) && PathPrefix(`/portainer`)"
      - "traefik.http.routers.portainer.middlewares=portainer-stripprefix@file"
      - "traefik.http.routers.portainer.service=portainer"
      - "traefik.http.routers.portainer.entryPoints=web"

      - "traefik.http.services.portainer.loadbalancer.server.port=9000"

What’s the response to the request?

What is Traefik Dashboard telling you?

I personally prefer to use an explicit network (not default), see simple Traefik Swarm example.

Additional thought: how many containers are you running? Any Docker network only allows for ~250 IPs by default.

The response I should get from the url I'm testing (https://censored.domain/api/v3/ping) is a 204 empty response, I'm getting a timeout.
The service is online and working correctly.
The traefik dashboard shows the proxy correctly

There are about 7 containers running.

Please share your Traefik static and dynamic config, and Traefik's docker-compose.yml.

traefik.yml

providers:
    docker:
        watch: true
        exposedByDefault: false
        # For some reason everything breaks when this option is enabled
        # swarmMode: false
    file:
        directory: /etc/traefik/conf.d
        watch: true

entryPoints:
    web:
        address: :80
        http:
            #middlewares:
            #    - cloudflarewarp@file

api:
    dashboard: true
    # TODO: Consider removing this and make the dashboard (http://localhost:8080) accessible only via a VPN
    # We could set up Wireguard and create a «fake domain» pointing to localhost:8080 by using a custom DNS server for easy access
    insecure: true

metrics:
    prometheus:
        addRoutersLabels: true
        addServicesLabels: true

experimental:
  plugins:
    cloudflarewarp:
      moduleName: "github.com/BetterCorp/cloudflarewarp"
      version: "v1.3.3"

log:
  level: DEBUG

conf.d/middlewares/auth.yml

version: "3.8"

http:
    middlewares:
        admin-auth:
            basicauth:
                removeHeader: true
                users: 
                    - "USERNAME:XXXX"

        registry-auth:
            basicauth:
                removeHeader: true
                users: 
                    - "USERNAME:XXXX"

conf.d/middlewares/general.yml

version: "3.8"

http:
    middlewares:
        cloudflarewarp:
            plugin:
                cloudflarewarp:
                    disableDefault: false

        # Public services should use this middleware to prevent big bodies
        limit-body:
            buffering:
                maxRequestBodyBytes: 2000000 # 2 Megabytes
        
        # Used by admin > registry (unused currently)
        registry-headers:
             headers:
                customRequestHeaders:
                    Docker-Distribution-Api-Version: registry/2.0
                    X-Forwarded-Proto: https
                customResponseHeaders:
                    Docker-Distribution-Api-Version: registry/2.0

conf.d/middlewares/strip-prefix.yml

version: "3.8"

http:
    middlewares:
        traefik-stripprefix:
            stripprefix:
                prefixes: 
                    - "/traefik"

        portainer-stripprefix:
            stripprefix:
                prefixes:
                    - "/portainer"

        grafana-stripprefix:
            stripprefix:
                prefixes:
                    - "/grafana"

traefik docker-compose (partial)

version: "3.8"

networks:
  default:
    name: traefik
    external: true

volumes:
  grafana_data:
    external: true
  portainer_data:
    external: true

services:
  reverse-proxy:
    image: "traefik:v3.0"
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    labels:
      - "traefik.enable=true"

      - "traefik.http.routers.traefik-dashboard.entryPoints=web"
      - "traefik.http.routers.traefik-dashboard.service=api@internal"
      - "traefik.http.routers.traefik-dashboard.middlewares=admin-auth@file, traefik-stripprefix@file"
      - "traefik.http.routers.traefik-dashboard.rule=Host(`XXX.${TRAEFIK_DOMAIN:?TRAEFIK_DOMAIN is not set}`) && (PathPrefix(`/traefik`) || Header(`Referer`, `https://XXX.${TRAEFIK_DOMAIN}/traefik/dashboard/`))"

      - "traefik.http.services.traefik-dashboard.loadbalancer.server.port=9999" # Any port will work here
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "./traefik/:/etc/traefik/:ro"

Interesting trick to use Header(`Referer`, for Traefik Dashboard.

I thought you might have a longer rule like

Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"

that would have had higher priority over

Host(`${TRAEFIK_DOMAIN}`) && PathPrefix(`/api/v3`)

We made sure that all rules are unique to their service, nothing can collide.
In addition, the API responds to itself within its container (via docker exec ) and the issue persists even without CF's proxy (by making a request using the server's public IP address and setting the Host header)

What infra & distribution are you running on?

We are on Ubuntu 22.04.3 (GNU/Linux 5.15.0-83-generic x86_64).

VMs, firewalls, VPNs?

No VMs.
No firewall rules.
No VPNs.

In addition, it worked before. So idk why now it decides to give up (no settings were changed)

So the only difference is that the non-working service is in a separate compose file? Have you tried placing it in the same compose file? Have you tried using a dedicated Docker network, not default? See simple Traefik Swarm example.

So I moved it to the same compose, restarted the stack, and it worked.
Then I restarted it again and it stopped working...

I don’t know what special handling Docker has for default, that might just not work.

Try an extra network, check simple Traefik Swarm example.

The file is now using dedicated networks, no change. That specific service does not reply (everything else in the file does)

Change image to traefik/whoami:v1.10 to see if its in general or just with your specific image, you can set the internal port used with env WHOAMI_PORT_NUMBER. (Doc)