Traefik not routing traffic to backend service

I am using swarm mode, with socket-proxy and letsencrypt. Everything looks good except the traffic is not routed to the backend. It shows 404 not found.

The certificate works
I ssh into the container, the backend service is running/working with content.

Here is my config

version: '3.8'

networks:
  web:
    external: true
  my-socket-proxy:
    external: true

services:  
  socket-proxy:
    image: tecnativa/docker-socket-proxy:latest
    deploy: 
      restart_policy:
        condition: on-failure
      placement:
        constraints: [node.role == manager]
    environment:
      NETWORKS: 1
      SERVICES: 1
      TASKS: 1
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      - my-socket-proxy

  reverse-proxy:  
    deploy:
      replicas: 1
    image: traefik:3.2.1
    command:  
      - "--providers.swarm=true"
      # Use the secure docker socket proxy
      - "--providers.swarm.endpoint=tcp://socket-proxy:2375"
      # Add a constraint to only use services with the label "traefik.constraint-label=cloud_public"
      - "--providers.swarm.constraints=Label(`traefik.constraint-label`, `cloud-public`)"
      # Don't expose containers per default
      - "--providers.swarm.exposedByDefault=false"
      - "--providers.swarm.network=web"
      - "--providers.swarm.defaultRule=Host(`{{ normalize .Name }}`)"
      - "--providers.file.filename=/traefik_conf/dynamic_conf.toml"
      - "--api=false"  
      - "--api.insecure=false"  
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--entrypoints.web.http.redirections.entryPoint.to=websecure"
      - "--entrypoints.web.http.redirections.entryPoint.scheme=https"
      - "--entrypoints.web.http.redirections.entrypoint.permanent=true"
      - "--certificatesresolvers.letsencrypt.acme.tlschallenge=true"
      - "--certificatesresolvers.letsencrypt.acme.email=my@gmail.com"
      - "--certificatesresolvers.letsencrypt.acme.storage=/acme/certs.json"
      - "--global.sendAnonymousUsage=false"
      # Logging
      - "--accesslog"
      - "--log.level=debug"
      - "--log.filePath=/log.txt"
    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
    volumes:
      # storage for the SSL certificates
      - /home/my/data/traefik/acme:/acme
      - /home/my/logs/traefik/log.txt:/log.txt
      # bind mount the directory for your traefik configuration
      - /home/my/data/traefik/conf:/traefik_conf
    networks:
      - web
      - my-socket-proxy


  my-ui:
    image: 'xxxxxxxxxxxxxxxxx:latest'
    deploy:
      replicas: 1
      labels:
        # Add labels at the deploy level instead of service level
        # Router configuration
        - "traefik.http.routers.my-ui.rule=Host(`www.aaaaa.com`, `aaaaa.com`)"
        - "traefik.http.routers.my-ui.entrypoints=websecure"
        - "traefik.http.services.my-ui.loadbalancer.server.port=9796"
        # TLS configuration
        - "traefik.http.routers.my-ui.tls=true"
        - "traefik.http.routers.my-ui.tls.certresolver=letsencrypt"
        - "traefik.http.routers.my-ui.tls.options=default"
        - "traefik.docker.network=web"
        - "traefik.enable=true"
        - "traefik.constraint-label=cloud-public"
    environment:
      JDBC_DATABASE_URL: jdbc:mysql://mysql:3306/db?serverTimezone=UTC
      JDBC_DATABASE_USERNAME: user
      JDBC_DATABASE_PASSWORD: pass
    

    volumes:
      - /home/my/data/logs:/logos
    networks:
      - web

For Traefik v3, you need to use

.rule=Host(`example.com`) || Host(`www.example.com`)

Are you using Docker Swarm with multiple nodes? It seems you only have a single Traefik replica, it uses "host" port mode and is not pinned to a node.

I don’t think that makes sense as Traefik will then listen only on one random node. When using host port mode, you should have multiple replicas pinned to certain nodes and use a loadbalancer in front.

Or you don’t use port host mode, then Docker Swarm will listen on all nodes on the port and forward requests to the one replica.

Also note that Docker Swarm will not create host folders for bind mount volumes automatically. When Traefik container is moved to another node (because of failure), you need to make sure the folders already exist, or the scheduling will fail on that node.

Enable and check Traefik debug log and Traefik access log in JSON format to see what’s happening during requests.

Thanks! That solved the problem.

For the other suggestion, I removed host mode, is this good enough? Or I need to make more changes? I only have one node for now, but will add more later. Thanks!

Without port host mode, you get an "ingress network" which will listen on all available nodes on the ports and forward requests to the one container replica. So this should work for one and more nodes.

I have a similar issue getting 404 trying to route to my local LAN. I have tailscale setup between the VPS and my LAN with subnet routing working.

Here's my traefik compose.yaml with some bits obscured for security:

  traefik:
    image: traefik
    container_name: traefik
    restart: unless-stopped
    command:
      - --log.level=INFO
      - --accesslog=true
      - --api.dashboard=true
      - --providers.docker=true
      - --providers.docker.exposedbydefault=false
      - --entrypoints.http.address=:80/tcp
      - --entrypoints.https.address=:443/tcp
      - --entrypoints.https.http.tls=true
      - --entrypoints.https.http.tls.certresolver=my-cloudflare
      - --serversTransport.insecureSkipVerify=true
      - --certificatesresolvers.my-cloudflare.acme.dnschallenge=true
      - --certificatesresolvers.my-cloudflare.acme.dnschallenge.provider=cloudflare
      - --certificatesresolvers.my-cloudflare.acme.caserver=https://acme-v02.api.letsencrypt.org/directory
      - --certificatesresolvers.my-cloudflare.acme.email=xyz@xyz.org
      - --certificatesresolvers.my-cloudflare.acme.storage=/letsencrypt/acme.json
    ports:
      - 80:80/tcp
      - 443:443/tcp
    environment:
      CF_API_EMAIL: xyz@xyz.org
      CF_API_KEY: xyzxyzxyzxyz
    labels:
      traefik.enable: "true"
      traefik.http.routers.dashboard.entrypoints: http
      traefik.http.routers.dashboard.rule: Host(`dash.xyz.org`)
      traefik.http.routers.dashboard.tls.certresolver: my-cloudflare
      traefik.http.routers.dashboard.service: api@internal
      traefik.http.routers.metube.entrypoints: http
      traefik.http.routers.metube.rule: Host(`xyz.org`) || Host(`metube.xyz.org`)
      traefik.http.routers.metube.tls.certresolver: my-cloudflare
      traefik.http.services.metube-service.loadbalancer.server.url: http://10.1.33.2:3101
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /home/rtr/dockge/traefik:/letsencrypt

http://dash.xyz.org - works
https://dash.xyz.org - 404
http://metube.xyz.org - 404
https://metube.xyz.org - 404

If I shell into the traefik container, traefik can see the LAN host

Connecting to 10.1.33.2:3101 (10.1.33.2:3101)
saving to 'index.html'
index.html           100% |*********************************************************|   710  0:00:00 ETA```

I have a dozen other docker containers on the VPS that work great. They have traefik labels defining routing by hostname like https://jitsi.xyz.org in each individual compose file. The VPS host can see the LAN as well. What am I missing? 10.1.33.2:3101 is http only. I'd like to eventually move most of my services back to my local LAN and greatly downsize the VPS if I can get this working right.

I should add, before I saw this post, I was using this line:

labels:
  traefik.http.routers.metube.rule: Host(`metube.xyz.org`)

And that didn't work either, still gave 404.

Not sure why you hijack this thread, which is about Docker Swarm, which you don’t seem to use.

You assign only web entrypoint to router, so Traefik won’t match on websecure using https.

Note you only need to assign certResolver to entrypoint, not again to router.

Maybe check simple Traefik example.

My apologies, I saw the same 404 error and got so excited that I missed the forum topic. I appreciate your suggestion, thank you.