Can't get traefik to work behind a proxy

Currently, I have a regular webserver running and I want to run traefik behind it.
For some reason though, I always get a 404 (from traefik, not the webserver) when I try to access one of my containers running behind traefik trough my webserver. However, if I hit traefik up directly (i.e. port 2000), I can access those containers.

Any pointers to what I'm doing wrong would be greatly appreciated :slight_smile:

/etc/traefik/traefik.yaml

accessLog:
  format: json
  fields:
    headers:
      defaultMode: keep

global:
  checkNewVersion: true
  sendAnonymousUsage: false

entryPoints:
  web:
    address: :80
    forwardedHeaders:
      insecure: true

api:
  dashboard: true

providers:
  docker:
    exposedByDefault: false
    network: traefik

traefik/docker-compose.yml

networks:
  traefik:
    external: true

services:
  traefik:
    container_name: traefik
    image: traefik:v2.8
    restart: always
    labels:
      traefik.enable: true
      traefik.http.routers.traefik-dashboard.entrypoints: web
      traefik.http.routers.traefik-dashboard.rule: Host(`traefik.domain.de`)
      traefik.http.routers.traefik-dashboard.service: api@internal
      traefik.http.routers.traefik-dashboard.middlewares: traefik-dashboard_auth
      traefik.http.middlewares.traefik-dashboard_auth.basicauth.users: admin:bcrypt_hash
    networks:
      traefik:
        ipv4_address: 172.20.255.254
    ports:
      - 2000:80
    volumes:
      - /etc/traefik:/etc/traefik
      - /var/run/docker.sock:/var/run/docker.sock:ro

some-project/docker-compose.yml

name: some-prefix

networks:
  main:
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 172.28.0.32/28
  traefik:
    external: true

services:
  some-name:
    image: some-image:latest
    user: "${UID}:${GID}"
    restart: unless-stopped
    depends_on:
      - mysql
    networks:
      main:
        ipv4_address: 172.28.0.46
      traefik:
    volumes:
      - ./assets:/assets:ro
    labels:
      traefik.enable: true
      traefik.http.routers.ji.entrypoints: web
      traefik.http.routers.ji.rule: Host(`intra.domain2.de`)
      traefik.http.services.ji.loadbalancer.server.port: 8080

    env_file: .env
    environment:
      # ...

  mysql:
    # ...

Excerpt from an access log for a 404:

{
  "ClientAddr": "172.20.0.1:57772",
  "ClientHost": "ip1",
  "ClientPort": "57772",
  "ClientUsername": "-",
  "DownstreamContentSize": 19,
  "DownstreamStatus": 404,
  "Duration": 43366,
  "OriginContentSize": 19,
  "OriginDuration": 17335,
  "OriginStatus": 404,
  "Overhead": 26031,
  "RequestAddr": "localhost:2000",
  "RequestContentSize": 0,
  "RequestCount": 1,
  "RequestHost": "localhost",
  "RequestMethod": "GET",
  "RequestPath": "/",
  "RequestPort": "2000",
  "RequestProtocol": "HTTP/1.1",
  "RequestScheme": "http",
  "RetryAttempts": 0,
  "StartLocal": "2023-02-13T09:47:53.571138751Z",
  "StartUTC": "2023-02-13T09:47:53.571138751Z",
  "downstream_Content-Type": "text/plain; charset=utf-8",
  "downstream_X-Content-Type-Options": "nosniff",
  "entryPointName": "web",
  "level": "info",
  "msg": "",
  "origin_Content-Type": "text/plain; charset=utf-8",
  "origin_X-Content-Type-Options": "nosniff",
  "request_Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8",
  "request_Accept-Encoding": "gzip, deflate",
  "request_Accept-Language": "en-GB,en;q=0.5",
  "request_Cache-Control": "max-age=0",
  "request_Connection": "close",
  "request_Cookie": "flash=MTY3NjI4MTQ1MnxEdi1CQkFFQ180SUFBUkFCRUFBQU12LUNBQUVHYzNSeWFXNW5EQW9BQ0hKbFpHbHlaV04wRGx0ZGFXNTBaWEptWVdObElIdDlfNE1DQVFMX2hBQUJFQUFBTmYtRU1nQUVCbk4wY21sdVp3d0RBQUV2Qm5OMGNtbHVad3dEQUFFdkJuTjBjbWx1Wnd3REFBRXZCbk4wY21sdVp3d0RBQUV2fK8SNx2C18YW6E2RcHlwstwXqF4NGDy6MJwaTH0OmRjD",
  "request_Sec-Gpc": "1",
  "request_Upgrade-Insecure-Requests": "1",
  "request_User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36",
  "request_X-Forwarded-For": "ip1",
  "request_X-Forwarded-Host": "intra.domain2.de",
  "request_X-Forwarded-Port": "2000",
  "request_X-Forwarded-Proto": "http",
  "request_X-Forwarded-Server": "fb4130d20b08",
  "request_X-Real-Ip": "172.20.0.1",
  "time": "2023-02-13T09:47:53Z"
}

Just for understanding: you have something like apache or nginx and it forwards requests for certain domains/paths to Traefik on port 2000? And Traefik should forward it again to your services?

What does Traefik debug log tell you?

What does http://traefik.domain.de/dashboard/ tell you?

Just for understanding: you have something like apache or nginx and it forwards requests for certain domains/paths to Traefik on port 2000? And Traefik should forward it again to your services?

Correct. Apache is running on :80 and forwards (through a reverse proxy) requests for certain domains to traefik, so that traefik can distribute them to my services.

What does Traefik debug log tell you?

Unfortunately, nothing.

time="2023-02-13T11:00:10Z" level=info msg="Configuration loaded from file: /etc/traefik/traefik.yml"
time="2023-02-13T11:00:10Z" level=info msg="Traefik version 2.8.8 built on 2022-09-30T12:20:13Z"
time="2023-02-13T11:00:10Z" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"web\":{\"address\":\":80\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}}},\"providers\":{\"providersThrottleDuration\":\"2s\",\"docker\":{\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"network\":\"traefik\",\"swarmModeRefreshSeconds\":\"15s\"}},\"api\":{\"dashboard\":true},\"log\":{\"level\":\"DEBUG\",\"format\":\"common\"},\"accessLog\":{\"format\":\"json\",\"filters\":{},\"fields\":{\"defaultMode\":\"keep\",\"headers\":{\"defaultMode\":\"keep\"}}},\"pilot\":{\"dashboard\":true}}"
time="2023-02-13T11:00:10Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\n"
time="2023-02-13T11:00:10Z" level=warning msg="Traefik Pilot is deprecated and will be removed soon. Please check our Blog for migration instructions later this year."
time="2023-02-13T11:00:10Z" level=info msg="Starting provider aggregator aggregator.ProviderAggregator"
time="2023-02-13T11:00:10Z" level=debug msg="Starting TCP Server" entryPointName=web
time="2023-02-13T11:00:10Z" level=info msg="Starting provider *traefik.Provider"
time="2023-02-13T11:00:10Z" level=debug msg="*traefik.Provider provider configuration: {}"
time="2023-02-13T11:00:10Z" level=info msg="Starting provider *docker.Provider"
time="2023-02-13T11:00:10Z" level=debug msg="*docker.Provider provider configuration: {\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"network\":\"traefik\",\"swarmModeRefreshSeconds\":\"15s\"}"
time="2023-02-13T11:00:10Z" level=info msg="Starting provider *acme.ChallengeTLSALPN"
time="2023-02-13T11:00:10Z" level=debug msg="*acme.ChallengeTLSALPN provider configuration: {}"
time="2023-02-13T11:00:10Z" level=debug msg="Configuration received: {\"http\":{\"services\":{\"api\":{},\"dashboard\":{},\"noop\":{}},\"serversTransports\":{\"default\":{\"maxIdleConnsPerHost\":200}}},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=internal
time="2023-02-13T11:00:10Z" level=debug msg="No default certificate, generating one" tlsStoreName=default
time="2023-02-13T11:00:10Z" level=debug msg="Provider connection established with docker 23.0.0 (API 1.42)" providerName=docker
time="2023-02-13T11:00:10Z" level=debug msg="Filtering disabled container" providerName=docker container=mysql-some-prefix-c1a6706903f58f4d68ae08acacd33bd03c591018ebd0706a631d62d91d3d4cfd
time="2023-02-13T11:00:10Z" level=debug msg="Filtering disabled container" container=unrelated-container-b8edf405d4c6179aeb75fd26405e5aa437df5981f7799455337a5acf31edba76 providerName=docker
time="2023-02-13T11:00:10Z" level=debug msg="Filtering disabled container" providerName=docker container=unrelated-container-277e0a26325b9cbe1e7b53eaeab60b363e5e99519ecc1f1b751894c8d31b40be
time="2023-02-13T11:00:10Z" level=debug msg="Filtering disabled container" providerName=docker container=unrelated-container-a718fd41eb706fe3858f87f98a28476cfd594c54df8047fb9a342aa718a59807
time="2023-02-13T11:00:10Z" level=debug msg="Filtering disabled container" providerName=docker container=unrelated-container-bf73a4c8294736139ead762a8b487578a71c3ca833fc7df525fc63bb17625b0c
time="2023-02-13T11:00:10Z" level=debug msg="Filtering disabled container" providerName=docker container=unrelated-container-a395725a6b77197290e5ee4b0019503627678f1cb0793d4cf1b01e0c420f2558
time="2023-02-13T11:00:10Z" level=debug msg="Filtering disabled container" container=unrelated-container-2cbc65fdeed98ef4b914ad914e7ed100f2a71ae34067877035c0fc9fd6119792 providerName=docker
time="2023-02-13T11:00:10Z" level=debug msg="Filtering disabled container" providerName=docker container=unrelated-container-1c3c18360f5f775a606755f11c2b81f26e1963689d654ea50e42d4e4de2d3e1b
time="2023-02-13T11:00:10Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"ji\":{\"entryPoints\":[\"web\"],\"service\":\"ji\",\"rule\":\"Host(`intra.domain2.de`)\"},\"traefik-dashboard\":{\"entryPoints\":[\"web\"],\"middlewares\":[\"traefik-dashboard_auth\"],\"service\":\"api@internal\",\"rule\":\"Host(`traefik.domain.de`)\"}},\"services\":{\"ji\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.20.0.2:8080\"}],\"passHostHeader\":true}},\"traefik-traefik\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.20.255.254:80\"}],\"passHostHeader\":true}}},\"middlewares\":{\"traefik-dashboard_auth\":{\"basicAuth\":{\"users\":[\"admin:hash\"]}}}},\"tcp\":{},\"udp\":{}}" providerName=docker
time="2023-02-13T11:00:11Z" level=debug msg="No default certificate, generating one" tlsStoreName=default
time="2023-02-13T11:00:11Z" level=debug msg="Creating middleware" serviceName=ji entryPointName=web routerName=ji@docker middlewareName=pipelining middlewareType=Pipelining
time="2023-02-13T11:00:11Z" level=debug msg="Creating load-balancer" entryPointName=web routerName=ji@docker serviceName=ji
time="2023-02-13T11:00:11Z" level=debug msg="Creating server 0 http://172.20.0.2:8080" serviceName=ji serverName=0 entryPointName=web routerName=ji@docker
time="2023-02-13T11:00:11Z" level=debug msg="child http://172.20.0.2:8080 now UP"
time="2023-02-13T11:00:11Z" level=debug msg="Propagating new UP status"
time="2023-02-13T11:00:11Z" level=debug msg="Added outgoing tracing middleware ji" middlewareName=tracing entryPointName=web routerName=ji@docker middlewareType=TracingForwarder
time="2023-02-13T11:00:11Z" level=debug msg="Added outgoing tracing middleware api@internal" entryPointName=web routerName=traefik-dashboard@docker middlewareName=tracing middlewareType=TracingForwarder
time="2023-02-13T11:00:11Z" level=debug msg="Creating middleware" routerName=traefik-dashboard@docker middlewareName=traefik-dashboard_auth@docker middlewareType=BasicAuth entryPointName=web
time="2023-02-13T11:00:11Z" level=debug msg="Adding tracing to middleware" entryPointName=web routerName=traefik-dashboard@docker middlewareName=traefik-dashboard_auth@docker
time="2023-02-13T11:00:11Z" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=web

What does http://traefik.domain.de/dashboard/ tell you?

I would assume this is an apache configuration problem. Try running a whoami container (traefik/whoami:v1.8, port 2000:80) instead of Traefik to see if you get a response through your proxy.

whoami works and renders:

Hostname: baee7c75cbab
IP: 127.0.0.1
IP: 172.17.0.2
RemoteAddr: 172.17.0.1:36832
GET / HTTP/1.1
Host: 172.17.0.2
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: en-GB,en;q=0.9
Connection: close
Cookie: flash=MTY3NjI4NjgzNXxEdi1CQkFFQ180SUFBUkFCRUFBQUJQLUNBQUE9fKxVLfWZKNPKrqx4bI_p4W2mxcA0WAi-KfzVG_g2lM4m; session=MTY3NjI4NzIwNnxKUC1CQXdFQkIxTmxjM05wYjI0Ql80SUFBUUVCQ2tWdGNHeHZlV1ZsU1VRQkJnQUFBQVhfZ2dFQkFBPT18GbcBszxpsYn1jbotMguaERGLRJNKSe2v9GwzcmPL_IE=
Sec-Gpc: 1
Upgrade-Insecure-Requests: 1
X-Forwarded-For: myIP
X-Forwarded-Host: intra.domain2.de
X-Forwarded-Server: intra.domain2.de

intra.domain2.de being mapped to 172.17.0.2:80. domain2.de:2000 also works

Quick note though: When I used traefik, apache correctly proxied traffic through to traefik, since I could see traefik's (as opposed to apache's) 404 page when I tried to visit intra.domain2.de.

The problem is that Host: is not set to a domain. That header is checked by Traefik to match a route and then forward to the service.

Did you access your proxy with a full domain name (http://intra.domain2.de:2000)? If yes, than you have to check why your other reverse proxy or apache are not forwarding the original headers.

Hmm, I was under the impression that with

forwardedHeaders:
  insecure: true

traefik would use X-Forwarded-* Headers and take the host from X-Forwarded-Host. Is that not the case?