Traefik in Docker Container, Pihole using Bare Metal RPi3B, Let's Encrypt delivers cert to Traefik Dashboard page, however page shows 404 Page not found

Traefik Traefik v3 (latest)
1

As spelled out in the title.

I am running Traefik in a Docker container (Intel NUC / Proxmox / Virtual Machine: Running Docker). I am leveraging the config from TechoTim's You Tube tutorial video for Traefik V3 from earlier this year.

Traefik logs show no errors and I'm leveraging the production link for Let's Encrypt.

When I enter the URL for the Traefik admin dashboard into any browser (have tried multiple), Let's Encrypt Delivers a valid certificate (Let's Encrypt vs. Traefik certificate) however the page delivers a "404 Page not found" message.

Separate from the Intel NUC running the Traefik container (but on the same network and VLAN), I am running Pihole on a Raspberry Pi 3B, with Unbound DNS and Wireguard VPN.

When I run dig "1.1.1.1" on the Docker VM the Pihole/Unbound DNS resolves the request because I have the Pihole set as the Custom DNS in my router (Ubiquiti UDM-Pro).

I am by no means an expert with Traefik nor at network configuration, however believe from what I have researched I should be able to maintain the network configuration (Traefik in Docker container, Pihole/Unbound DNS on bare metal RPi3B) I have in my network and be able to leverage Traefik and access my services externally from my network while I am away from the network.

I am not clear on whether I need to make a configuration change in the Traefik configs, Pihole or my UDM-P router. I have seen recommendations about port forwarding, but do not think this is the solution (but would be happy to be proven wrong!). I do know that both Traefik and Pihole both use port :80 so one of the two applications must have a different port, for example :8080 vs. :80.

I attempted this approach however, this did not result in the Traefik Dashboard to be accessible vs. the "404 Page not found."

One note, at this time I would prefer not to host Pihole in a Docker container, but will entertain it if it is determined as the only path - I have seen several posts on the Traefik support topics addressing this set-up, however would prefer to preserve / maintain the hardware and software configuration I have between Traefik on the Intel NUC and Pihole on the RPI3B.

I would appreciate guidance from the Traefik support team and community to solve this. I am really interested to use Traefik and I think my set up is likely very common and could help many others.

Thanks,

Potos

2

Share your full Traefik static and dynamic config, and docker-compose.yml if used.

Explain again what you want to achieve: which service on which server, using which port?

And what is PiHole using port 80 for, just the dashboard?

3

docker-compose.yml

version: "3.8"

services:
  traefik:
    image: traefik:v3.0
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    networks:
      - proxy
    ports:
      - 80:80
      - 443:443
      # - 443:443/tcp # Uncomment if you want HTTP3
      # - 443:443/udp # Uncomment if you want HTTP3
    environment:
      CF_DNS_API_TOKEN_FILE: /run/secrets/cf_api_token # note using _FILE for docker secrets
      # CF_DNS_API_TOKEN: ${CF_DNS_API_TOKEN} # if using .env
      TRAEFIK_DASHBOARD_CREDENTIALS: ${TRAEFIK_DASHBOARD_CREDENTIALS}
    secrets:
      - cf_api_token
    env_file: .env # use .env
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./data/traefik.yml:/traefik.yml:ro
      - ./data/acme.json:/acme.json
      - ./data/config.yml:/config.yml:ro
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.entrypoints=http"
      - "traefik.http.routers.traefik.rule=Host(`traefik-dashboard.local.example.com`)"
      - "traefik.http.middlewares.traefik-auth.basicauth.users=${TRAEFIK_DASHBOARD_CREDENTIALS}"
      - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=https"
      - "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
      - "traefik.http.routers.traefik-secure.entrypoints=https"
      - "traefik.http.routers.traefik-secure.rule=Host(`traefik-dashboard.local.example.com`)"
      - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
      - "traefik.http.routers.traefik-secure.tls=true"
      - "traefik.http.routers.traefik-secure.tls.certresolver=cloudflare"
      - "traefik.http.routers.traefik-secure.tls.domains[0].main=local.example.com"
      - "traefik.http.routers.traefik-secure.tls.domains[0].sans=*.local.example.com"
      - "traefik.http.routers.traefik-secure.service=api@internal"

secrets:
  cf_api_token:
    file: ./cf_api_token.txt

networks:
  proxy:
    external: true
4

traefik.yml

--

api:
dashboard: true
debug: true
entryPoints:
http:
address: ":80"
http:
redirections:
entryPoint:
to: https
scheme: https
https:
address: ":443"
serversTransport:
insecureSkipVerify: true
providers:
docker:
endpoint: "unix:///var/run/docker.sock"
exposedByDefault: false
file:
filename: /config.yml
certificatesResolvers:
cloudflare:
acme:
email: youremail@email.com
storage: acme.json
caServer: https://acme-v02.api.letsencrypt.org/directory # prod (default)
# caServer: https://acme-staging-v02.api.letsencrypt.org/directory # staging
dnsChallenge:
provider: cloudflare
#disablePropagationCheck: true # uncomment this if you have issues pulling certificates through cloudflare, By setting this flag to true disables the need to wait for the propagation of the TXT record to all authoritative name servers.
#delayBeforeCheck: 60s # uncomment along with disablePropagationCheck if needed to ensure the TXT record is ready before verification is attempted
resolvers:
- "1.1.1.1:53"
- "1.0.0.1:53"

5

traefik.yml

api:
  dashboard: true
  debug: true
entryPoints:
  http:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: https
          scheme: https
  https:
    address: ":443"
serversTransport:
  insecureSkipVerify: true
providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
  file:
    filename: /config.yml
certificatesResolvers:
  cloudflare:
    acme:
      email: youremail@email.com
      storage: acme.json
      caServer: https://acme-v02.api.letsencrypt.org/directory # prod (default)
      # caServer: https://acme-staging-v02.api.letsencrypt.org/directory # staging
      dnsChallenge:
        provider: cloudflare
        #disablePropagationCheck: true # uncomment this if you have issues pulling certificates through cloudflare, By setting this flag to true disables the need to wait for the propagation of the TXT record to all authoritative name servers.
        #delayBeforeCheck: 60s # uncomment along with disablePropagationCheck if needed to ensure the TXT record is ready before verification is attempted 
        resolvers:
          - "1.1.1.1:53"
          - "1.0.0.1:53"
6

config.yml

http:
 #region routers 
  routers:
    proxmox:
      entryPoints:
        - "https"
      rule: "Host(`proxmox.local.example.com`)"
      middlewares:
        - default-headers
        - https-redirectscheme
      tls: {}
      service: proxmox
    pihole:
  
#endregion
#region services
  services:
    proxmox:
      loadBalancer:
        servers:
          - url: "https://192.168.0.17:8006"
        passHostHeader: true
#endregion
  middlewares:
    https-redirectscheme:
      redirectScheme:
        scheme: https
        permanent: true
    default-headers:
      headers:
        frameDeny: true
        browserXssFilter: true
        contentTypeNosniff: true
        forceSTSHeader: true
        stsIncludeSubdomains: true
        stsPreload: true
        stsSeconds: 15552000
        customFrameOptionsValue: SAMEORIGIN
        customRequestHeaders:
          X-Forwarded-Proto: https

    default-whitelist:
      ipAllowList:
        sourceRange:
        - "10.0.0.0/8"
        - "192.168.0.0/16"
        - "172.16.0.0/12"

    secured:
      chain:
        middlewares:
        - default-whitelist
        - default-headers
7

Thank you for the quick response bluepuma77. I have one service in my config file so far which is Proxmox. Proxmox runs on the Intel NUC as the OS supporting the Docker VM instance.

Pihole uses port 80 for the Web UI (dashboard).

I am trying to use both Pihole on my network for network wide blocking (including on Docker containers) and leverage Traefik on the Intel NUC.

I am seeking to run Traefik to support internal and external access to my services on the network which are run as Docker containers (Mealie, Plex, etc.) and Homeassistant which is run as its own VM on Proxmox.

I am using Pihole as my DNS resolver and have a DNS record created for traefik-dashboard.local.example.com (my doman and tld removed for privacy) which points to the IP address of the Docker VM.

8

@bluepuma77 I've provided the requested information. Can you please confirm you are working on a solution and a response for me? Thank you!

9

Not sure you realized this is a community forum. Traefik users supporting Traefik users. For real support, buy Traefik enterprise.

Your Traefik dashboard should be available at https://traefik-dashboard.local.example.com/dashboard/. Enable and check Traefik access log if the request reaches Traefik, more info in JSON format (doc). If not, you must work your way through your custom DNS setup or opening your VM ports.

10

Hi @bluepuma77 I do realize this is a community forum. I had responded with the information you asked and then my post was auto-hid by Askimet (unknown reason why, but I contacted Dylan the moderator). I do appreciate any additional guidance you can provide to me and I have observed your responses on other community posts and know you are very knowledgable with Traefik. I see that you added an additional "/dashboard" at the end of the Traefik dashboard URL. I wanted to confirm that was intentional. Again, I think my set up is very common, so hoping to learn where I need to make configuration adjustments so I can get this awesome setup working. Thank you in advance! :slight_smile:

11

The trailing slash / in /dashboard/ is mandatory

Doc

12

@bluepuma77 Just to eliminate there may have been an issue with the Lets Encrypt certificate - I cleared the old certificate and requested a new one which is received. I am still unable to access the Traefik dashboard even with adding the "/dashboard/" at the end of my Traefik Dashboard URL traefick-dashboard.local.example.com (replacing example with my domain of course). I have taken screen shots of this but I am unable to upload either in PNG or JPEG file format for some reason. They show the 404 page and the valid Let's Encrypt certificate.

13

Enable and check Traefik debug log (doc) and Traefik access log in JSON format (doc).

14

Thank you for sharing those links. I have reviewed them but am somewhat confused how to use the information. I can confirm logging is happening because I have it set "debug" to "true" in the traefik.yml file. I can see the logs when I enter "docker logs traefik". I am unclear how to check the Traefik debug log. I do not know where the file is located as I only have the following in my folder structure:

traefik (directory)

  • cf_api_token.txt
  • data (sub-directory)
  • docker-compose.yml

data (directory)

  • acme.json
  • config.yml
  • traefik.yml

Thank you for your continued help @bluepuma77. I know I have a lot to learn, but I'm excited to keep going and get this working.

15

I think I've been able to get the log information needed to continue to help troubleshoot. I've isolated only the ERR messages from the log and am providing them below. @bluepuma77 please let me know if I delivered what we were looking for:

2024-10-05T12:52:21Z ERR github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:229 > error="accept tcp [::]:80: use of closed network connection" entryPointName=http
2024-10-05T12:52:21Z ERR github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:664 > Error while starting server error="accept tcp [::]:80: use of closed network connection" entryPointName=http
2024-10-05T12:52:21Z ERR github.com/traefik/traefik/v3/pkg/provider/docker/pdocker.go:114 > Failed to list containers for docker error="Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/json": context canceled" providerName=docker
2024-10-05T12:52:47Z ERR github.com/traefik/traefik/v3/pkg/server/router/router.go:150 > error="error while parsing rule Host('traefik-dashboard.local.EXAMPLE.TLD'): 1:6: illegal rune literal" entryPointName=http routerName=traefik@docker
2024-10-05T12:52:47Z ERR github.com/traefik/traefik/v3/pkg/server/router/router.go:150 > error="error while parsing rule Host('proxmox.local.EXAMPLE.TLD'): 1:6: illegal rune literal" entryPointName=https routerName=proxmox@file
2024-10-05T12:52:47Z ERR github.com/traefik/traefik/v3/pkg/server/router/router.go:150 > error="error while parsing rule Host('traefik-dashboard.local.EXAMPLE.TLD'): 1:6: illegal rune literal" entryPointName=https routerName=traefik-secure@docker
2024-10-05T12:52:47Z ERR github.com/traefik/traefik/v3/pkg/server/router/tcp/manager.go:147 > error="invalid rule Host('proxmox.local.31520.me'), error: error while parsing rule Host('proxmox.local.EXAMPLE.TLD'): 1:6: illegal rune literal" entryPointName=https routerName=proxmox@file
2024-10-05T12:52:47Z ERR github.com/traefik/traefik/v3/pkg/server/router/tcp/manager.go:147 > error="invalid rule Host('traefik-dashboard.local.31520.me'), error: error while parsing rule Host('traefik-dashboard.local.EXAMPLE.TLD'): 1:6: illegal rune literal" entryPointName=https routerName=traefik-secure@docker
2024-10-05T13:03:11Z ERR github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:229 > error="accept tcp [::]:443: use of closed network connection" entryPointName=https
2024-10-05T13:03:11Z ERR github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:664 > Error while starting server error="accept tcp [::]:443: use of closed network connection" entryPointName=https
2024-10-05T13:03:11Z ERR github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:229 > error="accept tcp [::]:80: use of closed network connection" entryPointName=http
2024-10-05T13:03:11Z ERR github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:664 > Error while starting server error="accept tcp [::]:80: use of closed network connection" entryPointName=http

16

use of closed network connection usually only happens during container shutdown.

Please remove the container and create a new one for relevant log lines. Not sure if the issue with docker.sock is also happening because of shutdown/restart.