Traefik not using wildcard domain certificate, even though acme.json is populated and valid

Traefik Traefik v3 (latest)
1

So, I'm trying to mess around with Traefik since last day, but I just don't understand why it's behaving like this?

Traefik does seems to generate LE certificate both staging and production. But neither of them is actually used at all.

I tried all sorts of method that community said, delete acme.json field recreate it, read the logs check if there is any error. Nothing. At this point I'm just confused my Traefik is doing this to me?

I tried creating a wildcard certificate, It made but didn't used. I tried creating full TLD but it didn't used that also. Can you please go through my config and compose file, and look what might be going so wrong?

  • Traefik is also not serving itself on LE SSL.
  • Traefik is not using any production or staging certificate that it is generating.
  • Replace sub.example.com with a TLD that I own.

Trafeik compose.yaml:

version: "3"

x-podman:
    network: traefik_net
    userns: keep-id

services:
    traefik:
        container_name: traefik
        image: traefik:v3.3.3
        restart: unless-stopped
        security_opt:
            - label=type:container_runtime_t
            - no-new-privileges:true
        # depends_on:
        #     docker-socket-proxy:
        #         condition: service_healthy
        ports:
            - 80:80
            - 81:81
            - 443:443
            - 444:444
            # -- (Optional) Enable Dashboard, don't do in production
            - 8080:8080
        volumes:
            - /run/user/1000/podman/podman.sock:/var/run/docker.sock:ro
            - /var/log/traefik:/var/log/traefik
            - ./config/traefik.yaml:/etc/traefik/traefik.yaml:ro
            - ./config/conf/:/etc/traefik/conf/
            - ./config/certs/:/etc/traefik/certs/
        # -- (Optional) When using Cloudflare as Cert Resolver
        secrets:
            - "cloudflare-token"
            - "cloudflare-email"
        environment:
            - CF_DNS_API_TOKEN_FILE=/run/secrets/cloudflare-token
            - CF_API_EMAIL_FILE=/run/secrets/cloudflare-email
        networks:
            - traefik_net
        labels:
            - traefik.enable=true
            - traefik.http.routers.traefik=true
            - traefik.http.routers.traefik.rule=Host(`traefik.sub.example.com`)
            - traefik.http.middlewares.traefik-secure-redirect.redirectscheme.scheme=https
            - traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=https
            - traefik.http.routers.traefik.entrypoints=websecure
            - traefik.http.routers.traefik.middlewares=traefik-secure-redirect
            - traefik.http.routers.traefik-secure.rule=Host(`traefik.sub.example.com`)
            - traefik.http.routers.traefik.service=api@internal
            - traefik.http.routers.traefik-secure.tls=true
            - traefik.http.routers.traefik-secure.tls.certresolver=production
            - traefik.http.routers.traefik-secure.tls.domains[0].main=sub.example.com
            - traefik.http.routers.traefik-secure.tls.domains[0].sans=*.sub.example.com
            - traefik.http.services.traefik.loadbalancer.server.port=80
        logging:
            options:
                max-size: "10m"
                max-file: "3"
        # resources:
        #     limits:
        #         cpus: "0.001"
        #         memory: 80M

    # docker-socket-proxy:
    #   container_name: docker-socket-proxy
    #   image: tecnativa/docker-socket-proxy
    #   security_opt:
    #     - label=disable
    #   volumes:
    #     - /run/user/1000/podman/podman.sock:/var/run/docker.sock:z
    #   ports:
    #     - 2375:2375
    #   environment:
    #     - CONTAINERS=1
    #   networks:
    #     - traefik_net

secrets:
    cloudflare-email:
        file: "./secrets/cloudflare-email.secret"
    cloudflare-token:
        file: "./secrets/cloudflare-token.secret"

networks:
    traefik_net:
        external: true

traefik.yaml file:

global:
  checkNewVersion: true
  sendAnonymousUsage: false

# -- (Optional) Change Log Level and Format here...
#     - loglevels [DEBUG, INFO, WARNING, ERROR, CRITICAL]
#     - format [common, json, logfmt]
log:
  level: DEBUG
  format: common
  filePath: /var/log/traefik/traefik.log

# -- (Optional) Enable Accesslog and change Format here...
#     - format [common, json, logfmt]
# accesslog:
#   format: common
#   filePath: /var/log/traefik/access.log

# -- (Optional) Enable API and Dashboard here, don't do in production
api:
  dashboard: true
  insecure: true  # Don't do in production
#  debug: true  # Don't do in production

# -- Change EntryPoints here...
entryPoints:
  web:
    address: :80
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
  websecure:
    address: :443
  # https://github.com/JamesTurland/JimsGarage/blob/main/Traefik-Secure/traefik.yaml
  # https://github.com/foureight84/traefik-pihole-doh/blob/master/mgmt/traefik.yaml  
  web-external:
    address: :81
    http:
      redirections:
        entryPoint:
          to: websecure-external
          scheme: https
  websecure-external:
    address: :444

# -- Configure your CertificateResolver here...
certificatesResolvers:
  staging:
    acme:
      email: email@gmail.com
      storage: /etc/traefik/certs/acme.json
      caServer: "https://acme-staging-v02.api.letsencrypt.org/directory"
      dnsChallenge:
        provider: cloudflare
        disablePropagationCheck: true
        delayBeforeCheck: 60s
        resolvers:
          - "1.1.1.1:53"
          - "8.8.8.8:53"
  production:
    acme:
      email: email@gmail.com
      storage: /etc/traefik/certs/acme.json
      caServer: "https://acme-v02.api.letsencrypt.org/directory"
      dnsChallenge:
        provider: cloudflare
        disablePropagationCheck: true
        delayBeforeCheck: 60s
        resolvers:
          - "1.1.1.1:53"
          - "8.8.8.8:53"

tls:
  options:
    default:
      sniStrict: true
      minVersion: VersionTLS13

providers:
  docker:
    # -- (Optional) Enable this, if you want to expose all containers automatically
    exposedByDefault: false
    #endpoint: tcp://docker-socket-proxy:2375  # to access docker via proxy 
  file:
    directory: /etc/traefik
    watch: true

and finally the service that I want to use, the web interface of pihole:

version: "3"

services:
    pihole:
        container_name: pihole
        image: pihole/pihole:latest
        restart: unless-stopped
        ports:
            - 53:53/tcp
            - 53:53/udp
            # # You need to open this when NOT using reverse proxy server
            # - 80:80/tcp
        secrets:
            - piholeweb-password
        environment:
            TZ: "Asia/Kolkata"
            WEBPASSWORD:
            WEBPASSWORD_FILE: /run/secrets/piholeweb-password
        volumes:
            - "pihole_vol:/etc/pihole"
            - "dnsmasq_vol:/etc/dnsmasq.d"
        networks:
            - traefik_net
        # pihole-network:
        # ipv4_address: 192.168.1.3
        cap_add:
            - NET_ADMIN
            - NET_RAW
        labels:
            # Dashboard
            # https://github.com/JamesTurland/JimsGarage/blob/main/Pihole/docker-compose.yml
            - traefik.enable=true
            - traefik.docker.network=traefik_net
            - traefik.http.routers.piholeweb.entrypoints=websecure
            - traefik.http.routers.piholeweb.rule=Host(`pihole.sub.example.com`)
            - traefik.http.routers.piholeweb.tls=true
            - traefik.http.routers.piholeweb.tls.certresolver=production
            - traefik.http.routers.piholeweb.middlewares=piholeweb-redirectregex, piholeweb-secure-redirect

            - traefik.http.middlewares.piholeweb-redirectregex.redirectregex.regex=^https://pihole.sub.example.com/?$$
            - traefik.http.middlewares.piholeweb-redirectregex.redirectregex.replacement=https://pihole.sub.example.com/admin
            - traefik.http.middlewares.piholeweb-secure-redirect.redirectscheme.scheme=https
            - traefik.http.routers.piholeweb.service=piholeweb
            - traefik.http.services.piholeweb.loadbalancer.server.port=80
            # - traefik.http.routers.piholeweb.tls.domains[0].main=sub.example.com
            # - traefik.http.routers.piholeweb.tls.domains[0].sans=*.sub.example.com
        logging:
            options:
                max-size: "10m"
                max-file: "3"
        resources:
            limits:
                cpus: "0.005"
                memory: 50M

secrets:
    piholeweb-password:
        file: ./secrets/piholeweb-password.secret

networks:
    traefik_net:
        external: true

volumes:
    pihole_vol:
    dnsmasq_vol:

In Logs sometimes I see:

 Adding route for pihole.sub.example.com with TLS options default entryPointName=websecure
Looking for provided certificate(s) to validate ["sub.example.com" "*.sub.example.com"]... ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=production.acme
2025-02-09T06:59:24Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:984 > No ACME certificate generation required for domains

Adding certificate for domain(s) *.sub.example.com,sub.example.com
 No default certificate, fallback to the internal generated certificate tlsStoreName=default
2

What does Traefik /dashboard/ tell you?

In general I recommend to clean up your config. http-to-https and TLS can be cetralized on entrypoints, makes dynamic config in labels use a lot less lines. Check simple Traefik example.

3

/dashboard/ lists it is serving TLS, it's using default Traefik certificate. So yeah it's "TLS".

I don't need to do piholeweb-secure-redirect or traefik-secure-redirect if I already have redirect configured in entrypoints, and have entrypoints configured.

Does anyone have a wildcard certificate example for Traefik (like best-practice example)?

4

There is a Traefik dnsChallenge example, which is required for TLS wildcards.

1 Like
5

Still not working. I'm really not understand this. I got the staging certificates.

Yes I tried generating wildcard certificate but instead it generated two certificates on for traefik and one for pihole, but served neighter.

Is there something else at play that I might be missing?

version: "3"

# x-podman:
#     network: traefik_net
#     userns: keep-id

services:
    traefik:
        container_name: traefik
        image: traefik:v3.3.3
        restart: unless-stopped
        security_opt:
            - label=type:container_runtime_t
            - no-new-privileges:true
        # depends_on:
        #     docker-socket-proxy:
        #         condition: service_healthy
        ports:
            - 80:80
            - 81:81
            - 443:443
            - 444:444
            # -- (Optional) Enable Dashboard, don't do in production
            - 8080:8080
        networks:
            - traefik_net
        volumes:
            - /run/user/1000/podman/podman.sock:/var/run/docker.sock:ro
            - /var/log/traefik:/var/log/traefik
            - ./config/letsencrypt:/letsencrypt
            - ./config/conf/:/etc/traefik/conf/
        command:
            - --api=true
            - --api.insecure=true # disable in prod
            - --log.level=DEBUG
            - --accesslog=true
            - --accesslog.filepath=/var/log/traefik-access.log
            - --providers.docker=true
            - --providers.docker.exposedByDefault=false
            - --entrypoints.web.address=:80
            - --entrypoints.web.http.redirections.entrypoint.to=websecure
            - --entryPoints.web.http.redirections.entrypoint.scheme=https
            - --entrypoints.websecure.address=:443
            - --entrypoints.websecure.asDefault=true
            # TODO add web-external entrypoints
            - --entrypoints.websecure.http.tls.certresolver=staging
            - --entrypoints.websecure.http.tls.domains[0].main=sub.example.com
            - --entrypoints.websecure.http.tls.domains[0].sans=*.sub.example.com

            ## Staging certificate resolver
            - --certificatesresolvers.staging.acme.email=anudhadse66@gmail.com
            - --certificatesresolvers.staging.acme.storage=/letsencrypt/acme.json
            # - --certificatesresolvers.staging.acme.storage=/etc/traefik/certs/acme.json
            - --certificatesresolvers.staging.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
            - --certificatesresolvers.staging.acme.dnschallenge=true
            - --certificatesresolvers.staging.acme.dnschallenge.provider=cloudflare
            - --certificatesresolvers.staging.acme.dnschallenge.resolvers=1.1.1.1:53,8.8.8.8:53

            ## Production certificate resolver
            - --certificatesresolvers.production.acme.email=email@gmail.com
            - --certificatesresolvers.production.acme.storage=/letsencrypt/acme.json
            - --certificatesresolvers.production.acme.caserver=https://acme-v02.api.letsencrypt.org/directory
            - --certificatesresolvers.production.acme.dnschallenge=true
            - --certificatesresolvers.production.acme.dnschallenge.provider=cloudflare
            - --certificatesresolvers.production.acme.dnschallenge.resolvers=1.1.1.1:53,8.8.8.8:53
        secrets:
            - "cloudflare-token"
            - "cloudflare-email"
        environment:
            - CF_DNS_API_TOKEN_FILE=/run/secrets/cloudflare-token
            - CF_API_EMAIL_FILE=/run/secrets/cloudflare-email
        labels:
            - traefik.enable=true
            - traefik.http.routers.traefikweb=true
            - traefik.http.routers.traefikweb.service=traefikweb
            - traefik.http.routers.traefikweb.rule=Host(`traefik.sub.example.com`)
            - traefik.http.routers.traefikweb.service=api@internal
            - traefik.http.routers.traefikweb.entrypoints=websecure
            - traefik.http.routers.traefikweb.tls.certresolver=staging
            - traefik.http.services.traefikweb.loadbalancer.server.port=8080
        logging:
            options:
                max-size: "10m"
                max-file: "3"
        # resources:
        #     limits:
        #         cpus: "0.001"
        #         memory: 80M

    # docker-socket-proxy:
    #   container_name: docker-socket-proxy
    #   image: tecnativa/docker-socket-proxy
    #   security_opt:
    #     - label=disable
    #   volumes:
    #     - /run/user/1000/podman/podman.sock:/var/run/docker.sock:z
    #   ports:
    #     - 2375:2375
    #   environment:
    #     - CONTAINERS=1
    #   networks:
    #     - traefik_net

secrets:
    cloudflare-email:
        file: "./secrets/cloudflare-email.secret"
    cloudflare-token:
        file: "./secrets/cloudflare-token.secret"

networks:
    traefik_net:
        external: true
6

Also, this all started happening after I installed k3s. Although now k3s is now turned off.

7

k3s, podman, pihole - any other plans to make it more complex? Oh, yes, I see docker-socket-proxy in the start position :wink:

A simple setup works, check simple Traefik dnsChallenge example. Place your http-to-https and TLS on default "websecure" entrypoint, makes dynamic configs a lot more readable.

Then, for your target service, labels should be a lot simpler:

    labels:
      - traefik.enable=true
      - traefik.http.routers.piholeweb.rule=Host(`pihole.sub.example.com`)
      - traefik.http.routers.piholeweb.middlewares=piholeweb-redir
      - traefik.http.middlewares.piholeweb-redir.redirectregex.regex=^https://pihole.sub.example.com/?$$
      - traefik.http.middlewares.piholeweb-redir.redirectregex.replacement=https://pihole.sub.example.com/admin
      - traefik.http.services.piholeweb.loadbalancer.server.port=80

PS: Not sure if 0.5% of CPU is too small for services to run decently.

8

I finally got everything everything working. It was god damn k3s.

I uninstalled k3s, tried again, everything was up and running with latest traefik.

Then I installed k3s again, now disabling traefik as ingress controller. Seriously I don't know what I was doing at that time, and Anthropic AI helped me:

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--disable=traefik" sh -

Maybe k3s should include documentation, to use a external reverse proxy and using k3s managed reverse proxy, how they might conflict.

As per Claude:




What Happened Behind the Scenes: K3s and Your Reverse Proxy Conflict
You've experienced a classic case of service conflict, and I'll explain exactly what happened at each stage.
When You First Installed K3s
When K3s installs with default settings, several things happen:

Traefik Deployment:

K3s automatically deploys Traefik as its default ingress controller
Traefik binds to ports 80 and 443 on the host machine to handle incoming HTTP/HTTPS traffic
It runs with high privileges using hostNetwork: true


Network Configuration Changes:

K3s adds custom iptables rules to route traffic to cluster services
These rules direct incoming web traffic to Traefik instead of your existing reverse proxy
New network interfaces are created for the Kubernetes networking overlay


Port Binding Conflict:

Network ports can only be bound to one process at a time
Since Traefik claimed ports 80 and 443 first (or with higher privileges), your reverse proxy manager couldn't bind to these ports
When your reverse proxy tried to start, it failed because the ports were already in use



When You Uninstalled K3s

Service Removal:

All K3s services including Traefik were terminated
Ports 80 and 443 were released and became available again


Network Cleanup:

K3s uninstallation removed all the custom iptables rules it had created
Network interfaces created by K3s were removed
The system network configuration reverted to its pre-K3s state


Resource Release:

All the resources that K3s had claimed were freed
This allowed your reverse proxy to successfully bind to the required ports



When You Reinstalled with --disable=traefik

Selective Installation:

K3s installed all its core components (API server, scheduler, controller manager, etc.)
Critically, it skipped the Traefik ingress controller installation


No Port Conflicts:

Without Traefik, K3s didn't attempt to bind to ports 80 and 443
Your existing reverse proxy could continue to use these ports without conflict


Compatible Network Configuration:

While K3s still created its network interfaces and some iptables rules, they didn't intercept HTTP/HTTPS traffic in the way Traefik would have
The networking components focused on pod-to-pod and pod-to-service communication instead



This is why the --disable=traefik flag was the perfect solution - you get to run both K3s and your existing reverse proxy manager without conflicts.
If you later want to use Kubernetes ingress features, you can create custom ingress resources that work with your existing reverse proxy instead of requiring Traefik.