404 Error, not routing to external server

Traefik Traefik v3 (latest)
1

Hi all, I'm running into a problem that I can't seem to figure out and Google, Forums, ChatGPT & Claude AI have not been useless. I'm really hoping someone here can help.

I'm running traefik via docker on host "traefik1" which I'm using as a reverse proxy to serve an app running on an external server "canary1". I'm able to access the traefik dashboard but no matter what I do, I keep getting a 404 when trying to access the app.

Here is the docker-compose file

version: '3'

services:
  traefik:
    image: traefik:v3.1.2
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    networks:
      - traefik
    ports:
      - 80:80
      - 443:443
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./config/traefik.yml:/etc/traefik/traefik.yml:ro
      - ./config/config.yml:/etc/traefik/config.yml:ro
      - ./data/acme.json:/etc/traefik/acme.json
    labels:
      - "traefik.enable=true"
      # Route for Traefik dasboard
      - "traefik.http.routers.traefik.entrypoints=websecure"
      - "traefik.http.routers.traefik.rule=Host(`traefik.example.net`)"
      - "traefik.http.routers.traefik.tls=true"
      - "traefik.http.routers.traefik.tls.certresolver=letsencrypt"
      - "traefik.http.routers.traefik.service=api@internal"
      - "traefik.http.routers.traefik.middlewares=secureHeaders@file"
      # Route for Canary Tokens applicaiton
      - "traefik.http.routers.canary.entrypoints=websecure"
      - "traefik.http.routers.canary.rule=Host(`canary.example.net`)"
      - "traefik.http.routers.canary.tls=true"
      - "traefik.http.routers.canary.tls.certresolver=letsencrypt"
      - "traefik.http.services.canary.loadbalancer.server.url=http://192.1.1.1:8081"
      - "traefik.http.routers.canary.service=canary"
      - "traefik.http.routers.canary.middlewares=secureHeaders@file"

networks:
  traefik:
    external: true

Here is traefik.yml

global:
  checkNewVersion: true
  sendAnonymousUsage: false

api:
  dashboard: true
  insecure: true

entryPoints:
  web:
    address: ":80"
  websecure:
    address: ":443"

certificatesResolvers:
  letsencrypt:
    acme:
      email: info@example.net
      storage: /etc/traefik/acme.json
      httpChallenge:
        entryPoint: web

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
    network: traefik
  file:
    filename: "/etc/traefik/config.yml"
    watch: true

log:
  level: DEBUG

accessLog: {}

Here is my config.yml

http:
  middlewares:
    secureHeaders:
      headers:
        sslRedirect: true
        forceSTSHeader: true
        stsIncludeSubdomains: true
        stsPreload: true
        stsSeconds: 31536000
        customFrameOptionsValue: "SAMEORIGIN"
        contentTypeNosniff: true
        browserXssFilter: true
        referrerPolicy: "strict-origin-when-cross-origin"
        permissionsPolicy: "camera=(), microphone=(), geolocation=(), payment=()"
        customResponseHeaders:
          X-Robots-Tag: "none,noarchive,nosnippet,notranslate,noimageindex"
  services:
    canary:
      loadBalancer:
        servers:
          - url: "http://192.1.1.1:8081"

tls:
  options:
    default:
      minVersion: VersionTLS13
      cipherSuites:
        - TLS_AES_128_GCM_SHA256
        - TLS_AES_256_GCM_SHA384
        - TLS_CHACHA20_POLY1305_SHA256

A couple things I noticed from the logs.

First, I keep seeing this line, which is odd because that IP is the internal IP of the docker container and not the external app I should be connecting to.
"GET / HTTP/2.0" 404 19 "-" "-" 1 "canary@docker" "http://172.20.0.2:80" 1ms

Secondly, I see this error in batches:
http: TLS handshake error from 172.169.206.159:39904: tls: client offered only unsupported versions: [303 302 301]

Any help with this would be greatly appreciated as I've been at this for days now.

2

TLS error could come from people/bots testing the connection with old browsers/clients. You use LetsEncrypt, so the domain was announced and is publicly available.

To see if the 404 error comes from Traefik or the target service, enable access log in JSON format. OriginStatus is from service, DownstreamStatus is what Traefik returns.

The config looks ok, note that the request domain is forwarded by default, make sure your target doesn’t need a special Host header, compare with simple Traefik external example.

3

Thanks for the prompt response!

So I checked the access log as you advised and it seems the 404 is on both sides.

If I run curl http://192.1.1.1:8081 from within the traefik container, I'm able to get a proper response, so I know it's able to communicate with the app running on a different machine.

Another thing I noticed from the access log is that external app's url isn't listed anywhere in the access log (http://192.1.1.1:8081)

Here is some additional context.
The remote app is behind a firewall and doesn't have any exposed ports.
The Traefik machine has port 80 and 443 exposed and connected via vpn to backend app.
SSL is setup via Traefik.

Any ideas?

{
    "ClientAddr": "172.21.0.2:43996",
    "ClientHost": "172.21.0.2",
    "ClientPort": "43996",
    "ClientUsername": "-",
    "DownstreamContentSize": 19,
    "DownstreamStatus": 404,
    "Duration": 34463,
    "GzipRatio": 0,
    "OriginContentSize": 0,
    "OriginDuration": 0,
    "OriginStatus": 0,
    "Overhead": 34463,
    "RequestAddr": "canary.example.net",
    "RequestContentSize": 0,
    "RequestCount": 5,
    "RequestHost": "canary.example.net",
    "RequestMethod": "GET",
    "RequestPath": "/",
    "RequestPort": "-",
    "RequestProtocol": "HTTP/1.1",
    "RequestScheme": "http",
    "RetryAttempts": 0,
    "StartLocal": "2024-08-23T14:45:13.41719474-04:00",
    "StartUTC": "2024-08-23T18:45:13.41719474Z",
    "entryPointName": "web",
    "level": "info",
    "msg": "",
    "time": "2024-08-23T14:45:13-04:00"
}
{
    "ClientAddr": "72.80.144.123:51711",
    "ClientHost": "72.80.144.123",
    "ClientPort": "51711",
    "ClientUsername": "-",
    "DownstreamContentSize": 19,
    "DownstreamStatus": 404,
    "Duration": 1150225,
    "OriginContentSize": 19,
    "OriginDuration": 935762,
    "OriginStatus": 404,
    "Overhead": 214463,
    "RequestAddr": "canary.example.net",
    "RequestContentSize": 0,
    "RequestCount": 4,
    "RequestHost": "canary.example.net",
    "RequestMethod": "GET",
    "RequestPath": "/",
    "RequestPort": "-",
    "RequestProtocol": "HTTP/2.0",
    "RequestScheme": "https",
    "RetryAttempts": 0,
    "RouterName": "canary@docker",
    "ServiceAddr": "172.21.0.2:80",
    "ServiceName": "canary@docker",
    "ServiceURL": "http://172.21.0.2:80",
    "StartLocal": "2024-08-23T14:45:13.416506553-04:00",
    "StartUTC": "2024-08-23T18:45:13.416506553Z",
    "TLSCipher": "TLS_AES_128_GCM_SHA256",
    "TLSVersion": "1.3",
    "entryPointName": "websecure",
    "level": "info",
    "msg": "",
    "time": "2024-08-23T14:45:13-04:00"
}