Traefik 404 error after docker compose up

Traefik Traefik v2
1

I'm trying to set up docker to utilize a reverse proxy on my homelab and after configuring and running docker compose up -d I find a 301 moved permanently error on port 80 and a 404 error on port 443. I am trying to use this guide to utilize ssl certificates and eventually open some of my services up to be used by friends and family. Navigating to the ubuntu server hosting docker at 10.3.14.101 on ports 80 and 443 both yield the 404 error. Running nmap to see if the ports are just unreachable yields that port 80 is being redirected to port 443.

Curl on my main machine:

ptolemy@Phobos:~$ curl 10.3.14.101:443 -i
HTTP/1.1 404 Not Found
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Sat, 03 Feb 2024 11:01:18 GMT
Content-Length: 19
404 page not found

Curl on docker host machine:

ptolemy@titan:~$ curl 10.3.14.101:443 -i
HTTP/1.1 404 Not Found
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Sat, 03 Feb 2024 11:02:14 GMT
Content-Length: 19
404 page not found

I have followed the guide almost to a T and even went as far as retrieving the configuration files from this github page and changing the values to my personal information and following my file hierarchy of:

/home/ptolemy/docker/traefik/acme.json
/home/ptolemy/docker/traefik/config.yml
/home/ptolemy/docker/traefik/traefik.yml

/home/ptolemy/docker-compose/traefik/docker-compose.yml

docker-compose.yml:

ptolemy@titan:~/docker-compose/traefik$ cat docker-compose.yml
version: '3'

services:
  traefik:
    image: traefik:latest
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    networks:
      - proxy
    ports:
      - 80:80
      - 443:443
    environment:
      - CF_DNS_API_TOKEN=<MYAPITOKEN>
      # If you choose to use an API Key instead of a Token, specify your email as well
      # - CF_API_EMAIL=user@example.com
      # - CF_API_KEY=YOUR_API_KEY
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /home/ptolemy/docker/traefik/traefik.yml:/traefik.yml:ro
      - /home/ptolemy/docker/traefik/acme.json:/acme.json
      - /home/ptolemy/docker/traefik/config.yml:/config.yml:ro
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.dashboard.tls=true"
      - "traefik.http.routers.traefik.entrypoints=http"
      - "traefik.http.routers.traefik.rule=Host(`traefik-dashboard.<MYDOMAIN>`)"
      - "traefik.http.middlewares.traefik-auth.basicauth.users=<DASHBOARDLOGINHASH>"
      - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=https"
      - "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
      - "traefik.http.routers.traefik-secure.entrypoints=https"
      - "traefik.http.routers.traefik-secure.rule=Host(`traefik-dashboard.<MYDOMAIN>`)"
      - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
      - "traefik.http.routers.traefik-secure.tls=true"
      - "traefik.http.routers.traefik-secure.tls.certresolver=cloudflare"
      - "traefik.http.routers.traefik-secure.tls.domains[0].main=<MYDOMAIN>"
      - "traefik.http.routers.traefik-secure.tls.domains[0].sans=*.<MYDOMAIN>"
      - "traefik.http.routers.traefik-secure.service=api@internal"

networks:
  proxy:
    external: true

traefik.yml:

api:
  dashboard: true
  debug: true
entryPoints:
  http:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: https
          scheme: https
  https:
    address: ":443"
serversTransport:
  insecureSkipVerify: true
providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
  file:
    filename: /config.yml
certificatesResolvers:
  cloudflare:
    acme:
      email: <MYEMAIL>
      storage: acme.json
      dnsChallenge:
        provider: cloudflare
        #disablePropagationCheck: true # uncomment this if you have issues pulling certificates through cloudflare, By setting this flag to true disables the need to wait for the propagation of the TXT record to all authoritative name servers.
        resolvers:
          - "1.1.1.1:53"
          - "1.0.0.1:53"

config.yml: (empty as I currently don't have any other services to route to)

ptolemy@titan:~/docker/traefik$ cat config.yml
ptolemy@titan:~/docker/traefik$

acme.json:
I'm hesitant to paste here but it does have data in it:

-rw------- 1 ptolemy ptolemy 16K Feb 3 10:28 acme.json

traefik logs:

ptolemy@titan:~/docker/traefik$ sudo docker logs traefik
[sudo] password for ptolemy:
time="2024-02-03T10:33:40Z" level=info msg="Configuration loaded from file: /traefik.yml"

Let me know if you need any more information. Frankly, I'm tired at this point and genuinely thought this would be an easy set up on my test lab before implementing on my albeit barebones main lab.

2

Note: I can provide screenshots of yaml formatting if anyone believes that to be the issue idk why it wouldn't format properly pasting in

3

Please format correctly with 3 backticks before and after config or select config and press </> button, in yaml every space matters.

4

Thank you, I have updated it to such.

5

Of course you will get 404 when you curl with the IP, as Traefik router rule is matching with domain name. Try adding the domain name via

curl -H "Host: traefik-dashboard.<MYDOMAIN>" https://10.3.14.101:443 -i

I would place things like http-to-https redirect and TLS globally on entrypoint, see simple Traefik example.

6

I was using curl to show the 404 error I get on the browser, but also in running the command I get the following output:

ptolemy@titan:~/docker-compose/traefik$ curl -H "Host: traefik-dashboard.<I Put my domain here>" https://10.3.14.101:443 -i
curl: (60) SSL certificate problem: self-signed certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

I was under the impression that with using the acme challenge and cloudflare (who my dns is registered with) that it would provide a legitimate ssl certificate, and that with acme.json being populated I am assigned a valid certificate. Is my thought process incorrect?

7

So it seems Traefik Letsencrypt does not create a valid TLS cert for you. For curl you can use -k or --insecure to connect without verification.

Check the Traefik log for any errors, it helps to set level INFO or DEBUG (doc).

8

Alright I've been busy. I don't know if you want me to go over everything I've done all day but I will say it culminated in me stealing your docker-compose.yml off your github that you linked and renewing my port forwarding rules along with deleting my cloudflare records and remaking them. I'm now able to log in and view the traefik dashboard, but I still get invalid certificate errors when navigating to traefik. because it's using traefik default certificates instead of the let's encrypt certificates specified in your compose file.

9

This is the way:

And share your re-worked config.

10

When starting with traefik, it's better to start with a basic non-ssl config, and then add the SSL after

For Static Config, i prefer the command: in docker-compose.yml, as it allows for ${ENVVARS} to be used via .env

a docker-compose.yml with just enough to get non-ssl traefik working, a dashboard, and default rule to handle most containers without additional tags:

version: "3.7"
services:

  traefik:
    image: traefik:latest
    container_name: "traefik"
    restart: unless-stopped
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro # Access to Docker
      - ./traefik/rules:/rules # Traefik dynamic rules
    ports:
      - 80:80
    command:
      # Traefik settings to get a Dashboard and log settings
      - "--api.dashboard=true"
      - "--api=true"
      - "--api.insecure=true"
      - "--log.format=json"
      - "--log.level=INFO"
      # EntryPoints web ports and automatic redirect
      - "--entryPoints.web.address=:80"
      # Setup the docker provider, and basic rules to grab the docker service name as the host name
      - "--providers.docker=true"
      - "--providers.docker.endpoint=unix:///var/run/docker.sock"
      - "--providers.docker.exposedByDefault=true"
      - "--providers.docker.defaultRule=HostRegexp(`{{ index .Labels \"com.docker.compose.service\" }}.{anydomain:.*}`)"
      # Setup the file provider so we can add dynamic rules as YAML files
      - "--providers.file.directory=/rules"
      - "--providers.file.watch=true"
    labels:
      - "traefik.http.routers.traefik-rtr.service=api@internal"

the .defaultRule=HostRegexp... will pick up the docker service name, and automatically add it to traefik rules for any docker container, but you will still need a DNS name without adding another rule

in most cases this should give you a response to:

  • curl -L http://traefik.localhost
  • curl -L http://traefik.docker.localhost [with GUI docker desktop]
  • curl -L http://traefik.<yourdomain> [with DNS set correctly (Cloudflare/r53/hosts)]

adding SSL with AWS R53:

version: "3.7"
services:
  # This is a traefik setup for a public domain name with LE
  traefik:
    image: traefik:latest
    container_name: "traefik"
    restart: unless-stopped
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro # Access to Docker
      - ./traefik/acme.json:/acme.json # SSL certificates
      - ./traefik/rules:/rules # Traefik configuration
    environment:
      # Required for Route53, replace with whatever DNS providers API key are required
      - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
      - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
      - AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION}
      # For file (Dynamic) provider rules, we need a variable we can use
      - DOMAINNAME=${DOMAINNAME}
    ports:
      - 80:80
      - 443:443
    command:
      # Traefik settings to get a Dashboard
      - "--api.dashboard=true"
      - "--api=true"
      - "--api.insecure=true"
      # Traefik Log Settings
      - "--log.format=json"
      - "--log.level=INFO"
      # EntryPoints ports and redirecton to SSL
      - "--entryPoints.web.address=:80"
      - "--entryPoints.websecure.address=:443"
      - "--entryPoints.web.http.redirections.entryPoint.to=websecure"
      - "--entryPoints.web.http.redirections.entryPoint.scheme=https"
      # For containers that have their own untrusted certs
      - "--serversTransport.insecureSkipVerify=true" 
      # Use the AWS ENV variables to Authenticate to AWS and update Route53
      - "--certificatesResolvers.aws.acme.storage=/acme.json"
      - "--certificatesResolvers.aws.acme.email=${EMAIL}"
      - "--certificatesResolvers.aws.acme.dnsChallenge=true"
      - "--certificatesResolvers.aws.acme.dnschallenge.provider=route53" # Using AWS Route53 here
      # Attach SSL Certs to Primary Domains
      - "--entryPoints.websecure.http.tls.certresolver=aws"
      - "--entrypoints.websecure.http.tls.domains[0].main=${DOMAINNAME}"
      - "--entrypoints.websecure.http.tls.domains[0].sans=*.${DOMAINNAME}"
      # Setup the docker provider, and basic rules to grab the docker service name as the host name
      - "--providers.docker=true"
      - "--providers.docker.endpoint=unix:///var/run/docker.sock"
      - "--providers.docker.exposedByDefault=true"
      - "--providers.docker.defaultRule=Host(`{{ index .Labels \"com.docker.compose.service\" }}.${DOMAINNAME}`)"
      # Setup the file provider so we can add dynamic rules as YAML files, eg OAUTH middleware chains
      - "--providers.file.directory=/rules"
      - "--providers.file.watch=true"
    labels:
      - "traefik.http.routers.traefik-rtr.service=api@internal"

replace aws/r53 with whatever you need, eg cloudflare + ENV vars

Then, after that works, add in the auth with whatever method as a dynamic rule in a yml file, but set a default in the static commands

    command:
...
      # Now set some defaults entrypoint settings so we don't have to set them on any docker services
      - "--entryPoints.websecure.http.middlewares=basic-auth@file"
...

traefik/rules/traefik-auth.yml

http:
  middlewares:
    basic-auth:
      <basic auth settings>:

in most cases, a container you want to proxy wont need any labels to get SSL or Auth as long as your DNS has *.domain set correctly to the traefik IP address