404 when running traefik in front of docker containers

Hello!

TL:DR: I get a 404 when running containerized traefik as a proxy before other docker containers serving different domains.

Here is an image of what I want to achive. Is this a good solution for hosting multiple domains on one server or do you generally recommend something else?

My current setup

/srv/acme.json
it contains the necessary info for Let's Encrypt, did a chmod 600 to it

-/srv/traefik.toml

[entryPoints]
  [entryPoints.web]
    address = ":80"
    [entryPoints.web.http.redirections.entryPoint]
      to = "websecure"
      scheme = "https"

  [entryPoints.websecure]
    address = ":443"

[api]
  dashboard = true

[certificatesResolvers.lets-encrypt.acme]
  email = "my@mail.tld"
  storage = "acme.json"
  [certificatesResolvers.lets-encrypt.acme.tlsChallenge]

[providers.docker]
  watch = true
  network = "web"

[providers.file]
  filename = "traefik_dynamic.toml"

-/srv/traefik_dynamic.toml

[http.middlewares.simpleAuth.basicAuth]
  users = [
    "user:pw"
  ]


[http.routers.api]
  rule = "Host(`monitor.mydomain.tld`)"
  entrypoints = ["websecure"]
  middlewares = ["simpleAuth"]
  service = "api@internal"
  [http.routers.api.tls]
    certResolver = "lets-encrypt"

--/srv/lcmp/docker-compose.yml

version: "3.8"

networks:
  web:
    external: true
  internal:
    external: false

services:
  # PHP Service
  php:
    build: './php_docker/'
    volumes:
      - ./www/:/var/www/html/
    #labels:
      #- traefik.enable=true
      #- traefik.http.routers.shop.rule=Host(`mydomain.tld`)
      #- traefik.port=443
    networks:
      - internal
      #- web

  # Caddy Service
  caddy:
    build: './caddy_docker/'
    depends_on:
      - php
    restart: unless-stopped
    #ports:
    #  - "80:80"
    #  - "443:443"
    #  - "443:443/udp"
    volumes:
      - ./www/:/var/www/html/
      - ./caddy_docker/Caddyfile:/etc/caddy/Caddyfile
      - caddy_data:/data
      - caddy_config:/config
    labels:
      - traefik.enable=true
      - traefik.http.routers.caddy.rule=Host(`mydomain.tld`)
      #- traefik.http.routers.caddy.tls=true
      #- traefik.http.routers.caddy.tls.certresolver=lets-encrypt
      - traefik.port=80
    networks:
      - internal
      - web

  # MySQL Service
  mysql:
    image: mysql:8.0
    environment:
      MYSQL_ROOT_PASSWORD: rootpw
    volumes:
      - mysqldata:/var/lib/mysql
    networks:
      - internal
    labels:
      - traefik.enable=false

  # phpMyAdmin Service
  phpmyadmin:
    image: phpmyadmin/phpmyadmin:latest
    ports:
      - 8080:80
    environment:
      PMA_HOST: mysql
    networks:
      - internal
    depends_on:
      - mysql

# Volumes
volumes:
  mysqldata:
  caddy_data:
  caddy_config:

--/srv/lcmp/caddy_docker
---/srv/lcmp/caddy_docker/Caddyfile

(common) {
        header /* {
                -Server
        }
}

mydomain.tld {
        encode gzip zstd
        root * /var/www/html/public
        php_fastcgi php:9000
        file_server
        header {
                -server
                -Link
                -X-Powered-By

                # disable FLoC tracking
                #Permissions-Policy interest-cohort=()

                # enable HSTS
                Strict-Transport-Security max-age=31536000;

                # disable clients from sniffing the media type
                X-Content-Type-Options nosniff

                # clickjacking protection
                X-Frame-Options DENY

                # keep referrer data off of HTTP connections
                Referrer-Policy no-referrer-when-downgrade
        }
}

www.mydomain.tld {
        redir https://mydomain.tld{uri}
}

---/srv/lcmp/caddy_docker/Dockerfile

# Use the official Caddy Docker image
FROM caddy:latest

# Update package index and upgrade installed packages
RUN apk update && apk upgrade

# Copy Caddyfile to configure Caddy server
COPY Caddyfile /etc/caddy/Caddyfile

--/srv/lcmp/php_docker
---/srv/lcmp/php_docker/Dockerfile

FROM php:8.2-fpm-alpine

# Downloading install-php-extensions script and making it executable
ADD https://github.com/mlocati/docker-php-extension-installer/releases/latest/download/install-php-extensions /usr/local/bin/

# Making install-php-extensions script executable
RUN chmod +x /usr/local/bin/install-php-extensions

# Installing PHP extensions using install-php-extensions script
RUN install-php-extensions \
    mysqli \
    pdo \
    pdo_mysql \
    gd \
    zip \
    intl \
    xml \
    curl \
    dom \
    fileinfo \
    iconv \
    json \
    libxml \
    mbstring \
    openssl \
    pcre \
    phar \
    simplexml \
    zlib

# Setting PHP directives
RUN echo 'memory_limit = 512M' > /usr/local/etc/php/conf.d/memory-limit.ini
RUN echo 'max_execution_time = 300' > /usr/local/etc/php/conf.d/max-execution-time.ini
RUN echo 'max_input_vars = 10000' > /usr/local/etc/php/conf.d/max-input-vars.ini
RUN echo 'max_input_time = 300' > /usr/local/etc/php/conf.d/max-input-time.ini
RUN echo 'opcache.revalidate_freq=0' > /usr/local/etc/php/conf.d/opcache-revalidate-freq.ini
RUN echo 'opcache.validate_timestamps=0' > /usr/local/etc/php/conf.d/opcache-validate-timestamps.ini
RUN echo 'opcache.max_accelerated_files=7963' > /usr/local/etc/php/conf.d/opcache-max-accelerated-files.ini
RUN echo 'opcache.memory_consumption=256' > /usr/local/etc/php/conf.d/opcache-memory-consumption.ini
RUN echo 'opcache.interned_strings_buffer=16' > /usr/local/etc/php/conf.d/opcache-interned-strings-buffer.ini
RUN echo 'opcache.fast_shutdown=1' > /usr/local/etc/php/conf.d/opcache-fast-shutdown.ini

# Cleaning up
RUN rm -rf /var/cache/apk/*

--/srv/lcmp/www
---/srv/lcmp/www/public

Without traefik, just by running the docker-compose.yml in /srv/lcmp and activating the ports in the in the docker-compose.yml in the caddy service section, the website, which is located in /srv/lcmp/www/public ( a Shopware 6 installation)

It's also not clear to me whether to use Let's Encrypt to enable https:// for mydomain.tld or Caddy's built in encryption service, which works flawlessly without the traefik docker container in front of it. The Traefik dashboard works fine with my current setup at monitor.mydomain.tld/dashboard.

The network has been created with this command

docker network create web

The traefik container has been created using this command

docker run -d \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v $PWD/traefik.toml:/traefik.toml \
  -v $PWD/traefik_dynamic.toml:/traefik_dynamic.toml \
  -v $PWD/acme.json:/acme.json \
  -p 80:80 \
  -p 443:443 \
  --network web \
  --name traefik \
  traefik:v2.2

Here is the output of docker ps

CONTAINER ID   IMAGE                          COMMAND                  CREATED       STATUS       PORTS                                                                      NAMES
58e9eb2ad4f2   lcmp-caddy                     "caddy run --config …"   2 hours ago   Up 2 hours   80/tcp, 443/tcp, 2019/tcp, 443/udp                                         lcmp-caddy-1
aa7f81337b50   phpmyadmin/phpmyadmin:latest   "/docker-entrypoint.…"   2 hours ago   Up 2 hours   0.0.0.0:8080->80/tcp, :::8080->80/tcp                                      lcmp-phpmyadmin-1
e79fc710c52f   lcmp-php                       "docker-php-entrypoi…"   2 hours ago   Up 2 hours   9000/tcp                                                                   lcmp-php-1
f269056c4eb5   mysql:8.0                      "docker-entrypoint.s…"   2 hours ago   Up 2 hours   3306/tcp, 33060/tcp                                                        lcmp-mysql-1
fa29e420f5a1   traefik:v2.2                   "/entrypoint.sh trae…"   4 hours ago   Up 4 hours   0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp   traefik

I followed this tutorial, which works when I reproduce it

[How To Use Traefik v2 as a Reverse Proxy for Docker Containers on Ubuntu 20.04 | DigitalOcean](https://Traefik Docker Digital Ocean)

Compare to simple Traefik example.

Thank you very much for your quick reply, I've tried it but now I'm running into a redirect loop when trying to open my domain vanill.at see this output of

curl -vL vanill.at

* TLSv1.2 (IN), TLS header, Unknown (23):
< HTTP/2 308
< alt-svc: h3=":443"; ma=2592000
< date: Fri, 22 Mar 2024 18:32:53 GMT
< location: https://vanill.at/
< server: Caddy
< server: Caddy
< content-length: 0
<
* Connection #1 to host vanill.at left intact
* Maximum (50) redirects followed
curl: (47) Maximum (50) redirects followed

Also, I'm seeing this error when running the logs command on the lcmp-caddy-1 container.
This might be because Caddy also automatically issues certificate for the domain.

docker logs -f 1876714962c4

time="2024-03-21T19:20:58Z" level=error msg="Unable to obtain ACME certificate for domains \"vanill.at\": unable to generate a certificate for the domains [vanill.at]: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many certificates (5) already issued for this exact set of domains in the last 168 hours: vanill.at, retry after 2024-03-22T21:13:49Z: see https://letsencrypt.org/docs/duplicate-certificate-limit/" rule="Host(vanill.at)" providerName=lets-encrypt.acme ACME CA="https://acme-v02.api.letsencrypt.org/directory" routerName=caddy@docker

have a nice weekend!

Congratulation for digging up the almost antique Traefik v2.2 from Sep 7, 2020 :joy:

Recommend to use a current release (link).

I've since updated to the latest Traefik version, here is the current docker-compose.yml

version: '3.9'

services:
  traefik:
    image: traefik:latest
    ports:
      - 80:80
      - 443:443
    networks:
      - caddy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - letsencrypt:/letsencrypt
      - $PWD/traefik.toml:/traefik.toml
      - $PWD/traefik_dynamic.toml:/traefik_dynamic.toml
      - /var/log:/var/log
    command:
      - --api.dashboard=true
      - --log.level=INFO
      - --log.filepath=/var/log/traefik.log
      - --accesslog=true
      - --accesslog.filepath=/var/log/traefik-access.log
      - --providers.docker.network=web
      - --providers.docker.exposedByDefault=false
      - --entrypoints.web.address=:80
      - --entrypoints.web.http.redirections.entrypoint.to=websecure
      - --entryPoints.web.http.redirections.entrypoint.scheme=https
      - --entrypoints.websecure.address=:443
      - --entrypoints.websecure.http.tls.certresolver=myresolver
      - --certificatesresolvers.myresolver.acme.email=office@fastsite.click
      - --certificatesresolvers.myresolver.acme.tlschallenge=true
      - --certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json
    labels:
      - traefik.enable=true

networks:
  caddy:
    external: true

volumes:
  letsencrypt:
    name: letsencrypt

Here is the docker-compose.yml for running Caddy, MySQL, PHP and phpMyAdmin

version: "3.9"

networks:
  caddy:
    external: true
  internal:
    external: false

services:
  # PHP Service
  php:
    build: './php_docker/'
    volumes:
      - ./www/:/var/www/html/
    #labels:
      #- traefik.enable=true
      #- traefik.http.routers.php.rule=Host(`vanill.at`)
      #- traefik.http.routers.php.tls=true
      #- traefik.http.routers.php.tls.certresolver=lets-encrypt
      #- traefik.port=9000
   # networks:
      #- internal
      #- caddy

  # Caddy Service
  caddy:
    build: './caddy_docker/'
    depends_on:
      - php
    restart: unless-stopped
    #ports:
      #- "80:80"
      #- "443:443"
      #- "443:443/udp"
    volumes:
      - ./www/:/var/www/html/
      - ./caddy_docker/Caddyfile:/etc/caddy/Caddyfile
      - caddy_data:/data
      - caddy_config:/config
    labels:
      - traefik.enable=true
      - traefik.tcp.routers.caddy.entrypoints.websecure
      - traefik.tcp.routers.caddy.rule=HostSNI(`*`)
      - traefik.tcp.services.caddy.loadbalancer.server.port=80
      - traefik.http.routers.caddy.rule=Host(`vanill.at`)
      - traefik.http.routers.caddy.tls=true
      - traefik.http.routers.caddy.tls.certresolver=lets-encrypt
      - traefik.port=443
      - traefik.port=80
    networks:
      - internal
      - caddy

  # MySQL Service
  mysql:
    image: mysql:8.0
    environment:
      MYSQL_ROOT_PASSWORD: <redacted>
    volumes:
      - mysqldata:/var/lib/mysql
    networks:
      - internal
    #labels:
    #  - traefik.enable=false

  # phpMyAdmin Service
  phpmyadmin:
    image: phpmyadmin/phpmyadmin:latest
    ports:
      - 8080:80
    environment:
      PMA_HOST: mysql
    networks:
      - internal
    depends_on:
      - mysql

# Volumes
volumes:
  mysqldata:
  caddy_data:
  caddy_config:

Here is the output of nmap -Pn -p80,443 vanill.at

Starting Nmap 7.92 ( https://nmap.org ) at 2024-03-22 22:06 CET
Nmap scan report for vanill.at (185.164.4.42)
Host is up (0.00012s latency).
rDNS record for 185.164.4.42: nezgif.myvserver.online

PORT    STATE SERVICE
80/tcp  open  http
443/tcp open  https

Nmap done: 1 IP address (1 host up) scanned in 0.10 seconds

The Traefik dashboard still works perfectly fine at monitor.vanill.at but I get an error when curl to vanill.at see

*   Trying 185.164.4.42:80...
* Connected to vanill.at (185.164.4.42) port 80 (#0)
> GET / HTTP/1.1
> Host: vanill.at
> User-Agent: curl/7.76.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 308 Permanent Redirect
< Connection: close
< Location: https://vanill.at/
< Server: Caddy
< Date: Fri, 22 Mar 2024 21:07:50 GMT
< Content-Length: 0
<
* Closing connection 0
* Clear auth, redirects to port from 80 to 443Issue another request to this URL: 'https://vanill.at/'
*   Trying 185.164.4.42:443...
* Connected to vanill.at (185.164.4.42) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS header, Unknown (21):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: self-signed certificate
* Closing connection 1
curl: (60) SSL certificate problem: self-signed certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Here is the output of docker logs -f lcmp-caddy-1

{"level":"info","ts":1711141245.238458,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"info","ts":1711141245.2409647,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1711141245.2410948,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1711141245.24111,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1711141245.2411652,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00041fa80"}
{"level":"info","ts":1711141245.241781,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1711141245.241983,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"info","ts":1711141245.242027,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
{"level":"info","ts":1711141245.2420368,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["www.vanill.at","vanill.at"]}
{"level":"warn","ts":1711141245.2428803,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/data/caddy","instance":"1b92eeba-5964-4a28-a239-d65f686f2685","try_again":1711227645.242879,"try_again_in":86399.999999767}
{"level":"info","ts":1711141245.2429504,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":1711141245.2434278,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1711141245.2434392,"msg":"serving initial configuration"}

Here is the error I get in the Browser

ERR_CERT_AUTHORITY_INVALID

Something seems off here:

Traefik and target service should share at least one Docker network and it should match docker network if provided.

corrected, I still get too many redirects

Enable Traefik access log in JSON format to see if the response comes from Traefik itself or the target service.

That shows the status differentiated into OriginStatus (from target) and DownstreamStatus (from Traefik).


You mount a static config file traefik.toml, does that exist? What’s the content? It would completely "overwrite" command: config.


If caddy is behind Traefik, it would need to use httpChallenge or dnsChallenge to get a valid TLS cert.


Traefik forwards traffic to caddy port 80, so caddy is probably sending the redirect to https, trying to be connected on port 444, which will never happen in the current setting.

So either disable the http-to-https redirect in caddy -or- change service port to https.

When still using a custom (self-signed) TLS cert in caddy, add insecureSkipVerify in Traefik.