Traefik Dashboard fails to load

Traefik Traefik v2
,
1

Traefik was working fine two weeks ago and now i get 'DNS PROBE POSSIBLE' when trying to load dashboard.

Rebuilt container = no change
Container shows running and i can ping 172.18.0.2 from my machine and within docker.

I only have pihole running as another container on same machine and do have a local DNS record
traefik-dashboard-internal.local..com to 172.18.0.2

I cannot resolve this from my machine or within my server. I see that from my traefik container, i'm unable to reach pihole and think this is the problem (although unsure why my local DNS record configured is not working). I have also rebuilt pihole and using it as my DNS fine.

I tried adding in a route via:

sudo docker exec traefik ip route add 172.16.0.0/24 via 172.18.0.1

but get 'ip: RTNETLINK answers: Operation not permitted'

docker-compose.yml

version: '3'

services:
  traefik:
    image: traefik:latest
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    networks:
      - proxy
    ports:
      - 80:80
      - 443:443
    environment:
      - CF_API_EMAIL=***
      - CF_DNS_API_TOKEN=***
      # - CF_API_KEY=YOU_API_KEY
      # be sure to use the correct one depending on if you are using a token or key
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /mnt/traefik/data/traefik.yml:/traefik.yml:ro
      - /mnt/traefik/data/acme.json:/acme.json
      - /mnt/traefik/data/config.yml:/config.yml:ro
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.entrypoints=http"
      - "traefik.http.routers.traefik.rule=Host(`traefik-dashboard-internal.local.***.com`)"
      - "traefik.http.middlewares.traefik-auth.basicauth.users=admin:$$apr1$$8fbdqs9r$$8SI9LULyBJWXv.BmSXNQq0"
      - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=https"
      - "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
      - "traefik.http.routers.traefik-secure.entrypoints=https"
      - "traefik.http.routers.traefik-secure.rule=Host(`traefik-dashboard-internal.local.***.com`)"
      - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
      - "traefik.http.routers.traefik-secure.tls=true"
      - "traefik.http.routers.traefik-secure.tls.certresolver=cloudflare"
      - "traefik.http.routers.traefik-secure.tls.domains[0].main=local.***.com"
      - "traefik.http.routers.traefik-secure.tls.domains[0].sans=*.local.***.com"
      - "traefik.http.routers.traefik-secure.service=api@internal"

networks:
  proxy:
    external: true


traefik.yml

api:
  dashboard: true
  debug: true
entryPoints:
  http:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: https
          scheme: https
  https:
    address: ":443"
serversTransport:
  insecureSkipVerify: true
providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
  file:
    filename: /config.yml
certificatesResolvers:
  cloudflare:
    acme:
      email: ***
      storage: acme.json
      dnsChallenge:
        provider: cloudflare
        #disablePropagationCheck: true # uncomment this if you have issues pulling certificates through cloudflare, By setting this flag to true disables the need to wait for the propagation of the TXT record to all authoritative name servers.
        resolvers:
          - "1.1.1.1:53"
          - "1.0.0.1:53"

I have my wildcard populating in 'acme.json' file each time i rebuild trying to fix this. Looks to be just internal DNS resolution? any other way i can put a route inside the traefik container?

2

docker network proxy

[
    {
        "Name": "proxy",
        "Id": "b4811b1e7816fd3538b02633f5852d44ce56632f0d3363a2f1cde75130abdf20",
        "Created": "2022-12-03T21:40:49.071359823-10:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.18.0.0/16"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "e2589b42f734bbc63589295cb5c5d814c8ecf6b06a11ae16b7929a9e4afd5155": {
                "Name": "traefik",
                "EndpointID": "5f4419ab84214860e9b1eaeceea0c5c51761fac38b51ed61884e6275b60ac210",
                "MacAddress": "02:42:ac:12:00:02",
                "IPv4Address": "172.18.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]

sudo docker network list
NETWORK ID          NAME                DRIVER              SCOPE
e2a13e2ce119        bridge              bridge              local
0a1ebbb11363        host                host                local
258e4dfaef53        none                null                local
e467245b67f1        pi-hole_default     bridge              local
b4811b1e7816        proxy               bridge              local
3

Did you check your Traefik logs for error?

Do you have two dots in your DNS local..com?

You should place acme.json on a fixed path, like storage: /acme.json, either on a mounted folder or mounted volume. LE has limits how many certs you can create and if you recreate the Traefik container the old one (without path) is currently lost.

I am wondering if Traefik is even reading your static config /traefik.yml, that doesn’t look like a default path. Use command: --configFile= /traefik.yml in docker-compose.yml to load from custom path.

1 Like
4

Hello bluepuma77, yes i checked the logs and only had one entry:
time="2022-12-04T09:32:40-10:00" level=info msg="Configuration loaded from file: /traefik.yml"

Found the problem, I added in a manual reference (not needed with pihole DNS) and also used the wrong name in record.

echo address=/traefik/172.18.0.2 > ~/.firewalla/config/dnsmasq_local/traefik

*if i were to continue echo record in, it should be like this:

echo address=/traefik-dashboard-internal.local.mydomain.com/172.18.0.2 > ~/.firewalla/config/dnsmasq_local/traefik

Prior to my finding, On Firewalla, i discovered in the app that if i added in a custom DNS record here, it worked which had me scratching my head why piholes record wasn't working.

6

Maybe / is configured as home directory in the Traefik container, see default path docs. Then also your acme.json without a path would be in / and persisted over container re-creation.

If you want more information on what Traefik is doing, you can enable Traefik debug log.