Traefik v2 404 page not found for docker compose service

Traefik Traefik v2
1

Here is a docker compose file

version: "3"
networks:
  vester-net:

services:
  traefik:
    image: traefik:v2.10
    container_name: traefik
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro    
      - ./apps/traefik/traefik.yaml:/etc/traefik/traefik.yaml:ro  # static traefik configuration
      - ./apps/traefik/dynamic.yaml:/etc/traefik/dynamic.yaml:ro  # dynamic traefik configuration
      - ./apps/traefik/acme.json:/etc/traefik/acme.json           # TLS certificate storage
    networks:
      - vester-net
    ports:
      - "80:80"
      - 443:443
    labels:
      - "traefik.enable=true"
     # define traefik dashboard router and service
      - "traefik.http.routers.traefik.rule=Host(`${FQDN}`)" # change hostname!
      - "traefik.http.routers.traefik.service=api@internal"
      - "traefik.http.routers.traefik.tls.certresolver=tlschallenge"
      - "traefik.http.routers.traefik.entrypoints=web-secure"
      - "traefik.http.routers.traefik.middlewares=secHeaders@file"
      - "traefik.http.services.traefik.loadbalancer.server.port=8080"

  score:
    image: python:3.11.6
    container_name: score
    environment:
      - "PROJECT_NAME=score"
      - "GIT_REPOS=score-engine"
      - "GIT_BRANCH=master"
    env_file:
      - ./.env
    restart: unless-stopped
    volumes:
      - ./apps/score/run_root.sh:/run.sh:rw
    command: ["/bin/bash", "-c", "./run.sh"]
    networks:
      - vester-net
    ports:
      - "8001:8001"
    depends_on:
      - traefik
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.score.rule=(Host(`${FQDN}`) && PathPrefix(`/score`))"
      - "traefik.http.routers.score.entrypoints=web-secure"
      - "traefik.http.routers.score.service=score@docker"
      - "traefik.http.services.score.loadbalancer.server.port=8001"

When I go to https://${FQDN}/, everything goes well: it redirects to traefik dashboard and it works.

When I go to https://${FQDN}/score, it says 404 page not found.

I would love to have the community support on this. Thanks in advance.

2

Where is your Traefik static config (traefik.yml)?

Compare with simple Traefik example.

3

Here is traefik.yaml file

---
log:
  level: WARN  # ERROR, DEBUG, PANIC, FATAL, ERROR, WARN, INFO
providers:
  docker:
    exposedByDefault: false
    endpoint: 'unix:///var/run/docker.sock'
    # network: vester-net
  file:
    filename: /etc/traefik/dynamic.yaml
    watch: true
api:
  dashboard: true # if you don't need the dashboard disable it
entryPoints:
  web:
    address: ':80' # http
    http:
      redirections:
        entryPoint:
          to: web-secure
          scheme: https      
  web-secure:
    address: ':443' # https
certificatesResolvers:
  tlschallenge:
    acme:
      email: contact@vester.ai
      storage: /etc/traefik/acme.json # chmod 600 this file on the host system
      tlsChallenge: {}
global:
  checkNewVersion: true
  sendAnonymousUsage: true

And here is dynamic.yaml

---
# set more secure TLS options,
# see https://doc.traefik.io/traefik/v2.5/https/tls/
tls:
  options:
    default:
      minVersion: VersionTLS12
      cipherSuites:
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
        - TLS_AES_128_GCM_SHA256
        - TLS_AES_256_GCM_SHA384
        - TLS_CHACHA20_POLY1305_SHA256
      curvePreferences:
        - CurveP521
        - CurveP384

http:
  # define middlewares
  middlewares:

    # define some security header options,
    # see https://doc.traefik.io/traefik/v2.5/middlewares/http/headers/
    secHeaders:
      headers:
        browserXssFilter: true
        contentTypeNosniff: true
        frameDeny: true
        stsIncludeSubdomains: true
        stsPreload: true
        stsSeconds: 31536000
        customFrameOptionsValue: "SAMEORIGIN"
        customResponseHeaders:
          # prevent some applications to expose too much information by removing thise headers:
          server: ""
          x-powered-by: ""
4

You only enabled TLS on the Traefik dashboard router (via labels), not on score.

Enable TLS globally on entrypoint, see simple Traefik example.

5

Thanks @bluepuma77 ... something happened (moving forward) but it's not really solved.

Here is the modified docker compose file:

version: "3"
networks:
  vester-net:

services:
  traefik:
    image: traefik:v2.10
    container_name: traefik
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro        # It's not recommended mounting the docker socket into a container -> see https://github.com/wollomatic/traefik2-hardened
      - ./apps/traefik/traefik.yaml:/etc/traefik/traefik.yaml:ro  # static traefik configuration
      - ./apps/traefik/dynamic.yaml:/etc/traefik/dynamic.yaml:ro  # dynamic traefik configuration
      - ./apps/traefik/acme.json:/etc/traefik/acme.json           # TLS certificate storage
    networks:
      - vester-net
    ports:
      - "80:80"
      - "443:443"
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=vester-deploy_vester-net"
      - "traefik.http.routers.traefik.rule=Host(`${FQDN}`)" # change hostname!
      - "traefik.http.routers.traefik.service=api@internal"
      - "traefik.http.routers.traefik.entrypoints=web-secure"
      - "traefik.http.routers.traefik.middlewares=secHeaders@file"
      - "traefik.http.services.traefik.loadbalancer.server.port=8080"

  score:
    image: python:3.11.6
    container_name: score
    environment:
      - "PROJECT_NAME=score"
      - "GIT_REPOS=score-engine"
      - "GIT_BRANCH=master"
    env_file:
      - ./.env
    restart: unless-stopped
    volumes:
      - ./apps/score/run_root.sh:/run.sh:rw
    command: ["/bin/bash", "-c", "./run.sh"]
    networks:
      - vester-net
    ports:
      - "8001:8001"
    depends_on:
      - traefik
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=vester-deploy_vester-net"
      - "traefik.http.routers.score.rule=(Host(`${FQDN}`) && PathPrefix(`/score`))"
      - "traefik.http.routers.score.entrypoints=web-secure"
      - "traefik.http.routers.score.service=score@docker"
      - "traefik.http.services.score.loadbalancer.server.port=8001"

Here is traefik.yaml modification

---
accessLog: {}  # uncomment this line to enable access log
log:
  level: WARN  # ERROR, DEBUG, PANIC, FATAL, ERROR, WARN, INFO
providers:
  docker:
    exposedByDefault: false
    endpoint: 'unix:///var/run/docker.sock'
    # network: vester-net
  file:
    filename: /etc/traefik/dynamic.yaml
    watch: true
api:
  dashboard: true # if you don't need the dashboard disable it
entryPoints:
  web:
    address: ':80' # http
    http:
      redirections:
        entryPoint:
          to: web-secure
          scheme: https      
  web-secure:
    address: ':443' # https
    http:
      tls:
        certresolver: tlschallenge
certificatesResolvers:
  tlschallenge:
    acme:
      email: contact@vester.ai
      storage: /etc/traefik/acme.json # chmod 600 this file on the host system
      tlsChallenge: {}
global:
  checkNewVersion: true
  sendAnonymousUsage: true # disable this if you don't want to send anonymous usage data to traefik

But still when I go to https://${FQDN}/, there is a response: but it loads quite some time, the page seems to be coming, nop it's not.

When I go to https://${FQDN}/score, it says Not Found, The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.

I checked traefik logs and when I try this https://${FQDN}/score, it shows this:

traefik | 102.180.47.83 - - [18/Nov/2023:03:27:26 +0000] "GET /score HTTP/2.0" 404 207 "-" "-" 116 "score@docker" "http://172.22.0.3:8001" 3ms

So it is clearly still 404

6

Optional: use

networks:
  vester-net:
    name: vester-net

to have a fixed network name, you can use that in docker.network.

Enable and check Traefik debug log (doc).

Enable and check Traefik JSON access log (doc), to see OriginStatus (from target service) and DownstreamStatus (from Traefik).

For test, replace your target service with a simple whoami (link), make sure to adapt the port used.

7

Thanks again @bluepuma77 ... but unfortunately, it didn't work despite following I think most of what you said. Let me show it
Here is the docker compose file (with whoami)

version: "3"
networks:
  vester-net:
    name: vester-net

services:
  traefik:
    image: traefik:v2.10
    container_name: traefik
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./apps/traefik/traefik.yaml:/etc/traefik/traefik.yaml:ro  # static traefik configuration
      - ./apps/traefik/dynamic.yaml:/etc/traefik/dynamic.yaml:ro  # dynamic traefik configuration
      - ./apps/traefik/acme.json:/etc/traefik/acme.json           # TLS certificate storage
    networks:
      - vester-net
    ports:
      - 80:80
      - 443:443
    labels:
      - traefik.enable=true
      - traefik.http.routers.traefik.rule=Host(`${FQDN}`) # change hostname!
      - traefik.http.routers.traefik.service=api@internal

  whoami:
    image: traefik/whoami:v1.8
    networks:
      - vester-net
    labels:
      - traefik.enable=true
      - traefik.http.routers.mywhoami.rule=(Host(`${FQDN}`) && PathPrefix(`/whoami`))
      - traefik.http.services.mywhoami.loadbalancer.server.port=80

  score:
    image: python:3.11.6
    container_name: score
    environment:
      - "PROJECT_NAME=score"
      - "GIT_REPOS=score-engine"
      - "GIT_BRANCH=master"
    env_file:
      - ./.env
    restart: unless-stopped
    volumes:
      - ./apps/score/run_root.sh:/run.sh:rw
    command: ["/bin/bash", "-c", "./run.sh"]
    networks:
      - vester-net
    ports:
      - "8001:8001"
    expose:
      - "8001"
    depends_on:
      - traefik
    labels:
      - traefik.enable=true
      - traefik.http.routers.myscore.rule=(Host(`${FQDN}`) && PathPrefix(`/score`))
      - traefik.http.services.myscore.loadbalancer.server.port=8001

And here is traefik.yaml

---
accessLog: {}  # uncomment this line to enable access log
log:
  level: DEBUG  # ERROR, DEBUG, PANIC, FATAL, ERROR, WARN, INFO
providers:
  docker:
    exposedByDefault: false
    endpoint: 'unix:///var/run/docker.sock'
    network: vester-net
  file:
    filename: /etc/traefik/dynamic.yaml
    watch: true
api:
  dashboard: true # if you don't need the dashboard disable it
entryPoints:
  web:
    address: ':80' # http
    http:
      redirections:
        entryPoint:
          to: web-secure
          scheme: https      
  web-secure:
    address: ':443' # https
    http:
      tls:
        certresolver: tlschallenge
certificatesResolvers:
  tlschallenge:
    acme:
      email: contact@vester.ai
      storage: /etc/traefik/acme.json # chmod 600 this file on the host system
      tlsChallenge: {}

When I go to https://${FQDN}/, traefik works great.

When I go to https://${FQDN}/whoami, everything works great too

When I go to https://${FQDN}/score, it says still the following

Not Found, The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.

Traefik's logs confirms this too like this:
traefik | 102.180.47.83 - - [18/Nov/2023:21:15:38 +0000] "GET /score HTTP/2.0" 404 207 "-" "-" 20 "web-secure-myscore@docker" "http://172.23.0.4:8001" 3ms

traefik | 102.180.47.83 - - [18/Nov/2023:21:15:30 +0000] "GET /whoami HTTP/2.0" 200 900 "-" "-" 15 "web-secure-mywhoami@docker" "http://172.23.0.2:80" 2ms

From within the server, when I curl http://localhost:8001, the score service works.

So still the same issue as before, but a lot of learning and improvement on the process. Is there something, I am missing ?

8

How does the score service hande the url localhost:8001/score? Could it be the traefik routing works and the 404 is produced by the score service?

In this case, you could fix this by using the StripPrefix-Middleware: Traefik StripPrefix Documentation - Traefik

compose file, service:score snippet (untested):

      - "traefik.http.routers.score.middlewares=secHeaders@file,removescore"
      - "traefik.http.middlewares.removescore.stripprefix.prefixes=/score"
      - "traefik.http.services.traefik.loadbalancer.server.port=8080"

(additionally, I added the secHeaders@file middleware to the router, too)

BTW. Maybe you don't want to expose the port 8001 to the whole network. If you use port 8001 for debugging only on the local server, maybe

    ports:
      - "127.0.0.1:8001:8008"

could be enough.

9

Again: enable and check Traefik access log in JSON format (doc), to see OriginStatus (from target service) and DownstreamStatus (from Traefik). That way you can understand if the 404 is coming from Traefik or your target service.

As @wollomatic mentions, be aware of ports. You don't need ports, Traefik should forward requests internally only via the Docker network. You don't want your service exposed on the port externally, because that could circumvent security middlewares you set up in Traefik.

1 Like