Traefik issues default certificate

Traefik Traefik v2
,
1

I'm running traefik:v2.10.1 with the following docker-compose.yml:

services:
  traefik:
    image: traefik:v2.10.1
    restart: always
    container_name: traefik
    hostname: traefik
    command:
      - "--log.level=DEBUG"
      - "--api.dashboard=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--providers.docker.endpoint=tcp://traefik-socket-proxy:2375"
      - "--entrypoints.websecure.address=:443"
      - "--entrypoints.webunsecure.address=:80"
      - "--entrypoints.websecure.forwardedHeaders.trustedIPs=$CLOUDFLARE_IPS"
      - "--certificatesresolvers.le.acme.dnschallenge=true"
      - "--certificatesresolvers.le.acme.dnschallenge.provider=cloudflare"
      - "--certificatesresolvers.le.acme.email=<LETS ENCRYPT EMAIL>"
      - "--certificatesresolvers.le.acme.storage=/letsencrypt/acme.json"
      - "--certificatesresolvers.le.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
      - "--certificatesresolvers.le.acme.dnschallenge.resolvers=1.1.1.1:53"
      - "--certificatesresolvers.le.acme.dnschallenge.delaybeforecheck=10"
      - "--certificatesresolvers.le-vpn.acme.dnschallenge=true"
      - "--certificatesresolvers.le-vpn.acme.dnschallenge.provider=cloudflare"
      - "--certificatesresolvers.le-vpn.acme.email=<LETS ENCRYPT EMAIL>"
      - "--certificatesresolvers.le-vpn.acme.storage=/letsencrypt/acme-vpn.json"
      - "--certificatesresolvers.le-vpn.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
      - "--certificatesresolvers.le-vpn.acme.dnschallenge.resolvers=1.1.1.1:53"
      - "--certificatesresolvers.le-vpn.acme.dnschallenge.delaybeforecheck=10"
      - "--accesslog=true"
      - "--accesslog.filepath=/log/access.log"
      - "--accesslog.filters.statuscodes=200,300-399,500-599"
      - "--accessLog.bufferingSize=100"
    networks:
      - traefik-network
    ports:
      - 443:443
      - 80:80
    volumes:
      - "./letsencrypt:/letsencrypt"
      - "./log:/log"
    environment:
      - CF_API_EMAIL="<CLOUDFLARE EMAIL>"
      - CF_API_KEY=<CLOUDFLARE API KEY>
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.dashboard.rule=Host(`traefik.<DOMAIN 1>`)"
      - "traefik.http.routers.dashboard.service=api@internal"
      - "traefik.http.routers.dashboard.entrypoints=websecure"
      - "traefik.http.routers.dashboard.tls.certresolver=le"
      - "traefik.http.routers.dashboard.middlewares=authentik-proxy@docker"

  traefik-socket-proxy:
    image: tecnativa/docker-socket-proxy
    container_name: traefik-socket-proxy
    restart: unless-stopped
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      CONTAINERS: 1
    networks:
      - traefik-network


networks:
  traefik-network:
    external: true
    name: traefik

Unfortunately, traefik does not seem to issue a Let's Encrypt certificate, instead I get the default certificate. This is my log file: https://gist.githubusercontent.com/rdmchr/9951c9cff0fef2697ddde98e5ae6f97d/raw/d577de4ae9b23a3a23af9e3c5f9863920f760164/traefik.log
I also noticed that traefik does not even attempt to log in to Cloudflare since Im not getting any error even when using invalid credentials.

2

Why do you use 2 similar certresolvers?

Can you reduce dnsChallenge to the basics:

      - "--certificatesresolvers.myresolver.acme.dnschallenge=true"
      - "--certificatesresolvers.myresolver.acme.dnschallenge.provider=ovh"
      - "--certificatesresolvers.myresolver.acme.email=postmaster@example.com"
      - "--certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json"

What’s inside your acme.json?

Can you use the simpler tlsChallenge?

3

After simplifying my certresolver I got a new error: gist.githubusercontent.com/rdmchr/1792f8d901ffd4bc8f29b77ebc27f599/raw/5e1543553e1dba4bfa1ad9b66160111a39d9e75a/traefik.log

After this error acme times out after 2 minutes without issuing a certificate.

My new configuration is just this:

      - "--certificatesresolvers.le.acme.dnschallenge=true"
      - "--certificatesresolvers.le.acme.dnschallenge.provider=cloudflare"
      - "--certificatesresolvers.le.acme.email=<LETS ENCRYPT EMAIL>"
      - "--certificatesresolvers.le.acme.storage=/letsencrypt/acme.json"
      - "--certificatesresolvers.le.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
      - "--certificatesresolvers.le.acme.dnschallenge.delaybeforecheck=10"

I believe the important part is this:

traefik               | time="2023-06-13T12:01:09Z" level=debug msg="legolog: [WARN] [maven.<DOMAIN 1>] acme: cleaning up failed: cloudflare: unexpected response code 'SERVFAIL' for <DOMAIN 1>. "
traefik               | time="2023-06-13T12:01:09Z" level=debug msg="legolog: [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/<ACME ID>"
traefik               | time="2023-06-13T12:01:09Z" level=error msg="Unable to obtain ACME certificate for domains \"maven.<DOMAIN 1>\": unable to generate a certificate for the domains [maven.<DOMAIN 1>]: error: one or more domains had a problem:\n[maven.<DOMAIN 1>] [maven.<DOMAIN 1>] acme: error presenting token: cloudflare: unexpected response code 'SERVFAIL' for <DOMAIN 1>.\n" rule="Host(`maven.<DOMAIN 1>`)" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=le.acme routerName=reposilite@docker

I tried Googling for that but did not come up with anything useful, but it seems to be related to Cloudflare.
Could this be caused by my DNS setup? I only have a wildcard record that points to my server, ie.

*.<DOMAIN 1> -> 123.123.123.123

I also have some Email related records, but they shouldn't interfere with this. This setup allows me to easily add new services without having to open Cloudflare first.

4

Why not try simple tlsChallenge if your router uses Host(`traefik.<DOMAIN 1>`). Traefik will automatically create a TLS cert for every Host used.

5

I could try that, but even if it worked I have some private services that are only accessible via VPN. Thats the reason why I was trying to get DNS challenge working.

6

Do you have some CNAME?

7

I have checked and I do not have any CNAME records on <DOMAIN 1>, but I do have one record on <DOMAIN 2>, but that record is related to an external service.
To verify I checked using Resolve-DnsName -Name example.<DOMAIN1> -Server 1.1.1.1 -Type CNAME on some of my subdomains and none of them returned any CNAME records only one SOA record is returned.

Furthermore I tried setting the LEGO_DISABLE_CNAME_SUPPORT=true environment variable, but that also didn't help.

On my DNS server I have some CNAME records configured to rewrite some URLs, all of them are related to <DOMAIN 2> which I have commented out in my labels while debugging.

8

So I tried some more things and I got it working once. I even got it working using the production Let's Encrypt servers, but after adding another service I noticed I was not getting a certificate for that service. So I deleted the letsencrypt folder and tried regenerating all certificates.
This messed everything up and now I'm back where I started: no certificate is being issued.

Traefik seems to generate a certificate for each domain separately instead of generating a wildcard certificate (*.<DOMAIN 1>).
This only an assumption based on the logs at hand, the service I added was on <DOMAIN 2> which had only one service at that time so no wildcard certificate would have been necessary at that point.

traefik               | time="2023-06-17T23:54:09Z" level=debug msg="legolog: [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/<ID 3>"
traefik               | time="2023-06-17T23:54:09Z" level=error msg="Unable to obtain ACME certificate for domains \"spotify.<DOMAIN 1>\": unable to generate a certificate for the domains [spotify.<DOMAIN 1>]: error: one or more domains had a problem:\n[spotify.<DOMAIN 1>] time limit exceeded: last error: read udp 172.18.0.12:40313->108.162.193.216:53: i/o 
timeout\n" routerName=spotifyserver@docker rule="Host(`spotify.<DOMAIN 1>`) && PathPrefix(`/api/`)" ACME CA="https://acme-v02.api.letsencrypt.org/directory" providerName=le.acme
traefik               | time="2023-06-17T23:54:09Z" level=debug msg="legolog: [INFO] [whoami.radmacher.me] acme: Waiting for DNS record propagation."
traefik               | time="2023-06-17T23:54:09Z" level=error msg="Unable to obtain ACME certificate for domains \"paperless.<DOMAIN 1>\": unable to generate a certificate for the domains [paperless.<DOMAIN 1>]: error: one or more domains had a problem:\n[paperless.<DOMAIN 1>] time limit exceeded: last error: read udp 172.18.0.12:40691->108.162.193.216:53: i/o timeout\n" routerName=paperless@docker rule="Host(`paperless.<DOMAIN 1>`)" ACME CA="https://acme-v02.api.letsencrypt.org/directory" providerName=le.acme
traefik               | time="2023-06-17T23:54:09Z" level=debug msg="legolog: [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/<ID 2>"
traefik               | time="2023-06-17T23:54:09Z" level=debug msg="legolog: [INFO] [time.<DOMAIN 1>] acme: Waiting for DNS record propagation."
traefik               | time="2023-06-17T23:54:09Z" level=error msg="Unable to obtain ACME certificate for domains \"whoami.<DOMAIN 1>\": unable to generate a certificate for the domains [whoami.<DOMAIN 1>]: error: one or more domains had a problem:\n[whoami.<DOMAIN 1>] time limit exceeded: last error: read udp 172.18.0.12:47806->108.162.193.216:53: i/o timeout\n" ACME CA="https://acme-v02.api.letsencrypt.org/directory" providerName=le.acme routerName=whoami@docker rule="Host(`whoami.<DOMAIN 1>`)"
traefik               | time="2023-06-17T23:54:09Z" level=debug msg="legolog: [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/<ID 1>"
traefik               | time="2023-06-17T23:54:09Z" level=error msg="Unable to obtain ACME certificate for domains \"media.<DOMAIN 1>\": unable to generate a certificate for the domains [media.<DOMAIN 1>]: error: one or more domains had a problem:\n[media.<DOMAIN 1>] time limit exceeded: last error: read udp 172.18.0.12:34937->108.162.193.216:53: i/o timeout\n" routerName=immichweb@docker rule="Host(`media.<DOMAIN 1>`)" ACME CA="https://acme-v02.api.letsencrypt.org/directory" providerName=le.acme

As you can see I'm getting three different Let's Encrypt API ids. This would also explain why the new service was not getting a certificate since traefik probably started the Let's Encrypt process again, which seems quite flaky.

Has anyone else also experienced issues with Cloudflare, traefik and Let's Encrypt?
I think at this point it might make more sense to try another reverse proxy and if I can get that working just switch over...

9

Traefik will use le-go/go-acme to create TLS certs. It will automatically create certs for the domains in Host() and HostSNI() of each router. If you want a wildcard, you need to tell Traefik that, either global or on router. (Doc)

10

I see. The documentation only specifies how to use the file configuration which I would like to avoid, so I adopted it for CLI flags using this article Wildcard LetsEncrypt certificates with Traefik and Cloudflare · Major Hayden.

  traefik:
    image: traefik:latest
    restart: always
    container_name: traefik
    hostname: traefik
    command:
      - "--log.level=DEBUG"
      - "--api.dashboard=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--providers.docker.endpoint=tcp://traefik-socket-proxy:2375"
      - "--entrypoints.websecure.address=:443"
      - "--entrypoints.webunsecure.address=:80"
      - "--entrypoints.websecure.forwardedHeaders.trustedIPs=$CLOUDFLARE_IPS"
      - "--certificatesresolvers.le.acme.dnschallenge=true"
      - "--certificatesresolvers.le.acme.dnschallenge.provider=cloudflare"
      - "--certificatesresolvers.le.acme.email=<LE EMAIL>"
      - "--certificatesresolvers.le.acme.storage=/letsencrypt/acme.json"
      - "--certificatesresolvers.le.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
#      - "--certificatesresolvers.le.acme.dnschallenge.resolvers=1.1.1.1:53"
      - "--certificatesresolvers.le.acme.dnschallenge.delaybeforecheck=30"
      - "--entrypoints.websecure.http.tls=true"
      - "--entrypoints.websecure.http.tls.certResolver=le"
      - "--entrypoints.websecure.http.tls.domains[0].main=<DOMAIN 1>"
      - "--entrypoints.websecure.http.tls.domains[0].sans=*.<DOMAIN 1>"
      - "--entrypoints.websecure.http.tls.domains[1].main=<DOMAIN 2>"
      - "--entrypoints.websecure.http.tls.domains[1].sans=*.<DOMAIN 2>"
      - "--accesslog=true"
      - "--accesslog.filepath=/log/access.log"
      - "--accesslog.filters.statuscodes=200,300-399,500-599"
      - "--accessLog.bufferingSize=100"
    networks:
      - traefik-network
    ports:
      - 443:443
      - 80:80
    volumes:
      - "./letsencrypt:/letsencrypt"
      - "./log:/log"
    environment:
      - CF_API_EMAIL=<CF EMAIL>
      - CF_API_KEY=<CF GLOBAL KEY>
      - LEGO_DISABLE_CNAME_SUPPORT=true
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.dashboard.rule=Host(`traefik.<DOMAIN 1>`)"
      - "traefik.http.routers.dashboard.service=api@internal"
      - "traefik.http.routers.dashboard.entrypoints=websecure"
      - "traefik.http.routers.dashboard.tls.certresolver=le"
      - "traefik.http.routers.dashboard.middlewares=authentik-proxy@docker"

This unfortunately didn't work and looking at my Cloudflare dashboard it seems that it's still trying to generate a certificate for each service/domain since there were more than four TXT records. It also still served the Traefik default certificate. Is there anything I need to change in my service configuration?
All of my services labels are basically just a variation of this:

      - "traefik.http.routers.jellyfin.rule=Host(`watch.<DOMAIN 1>`)"
      - "traefik.http.routers.jellyfin.entrypoints=websecure"
      - "traefik.http.routers.jellyfin.tls.certresolver=le"
      - "traefik.http.routers.jellyfin.tls=true"

Some other blog article (article is in german) had this example, using labels instead of CLI flags, so I'm not sure which is the correct way.

      - "traefik.http.routers.traefik-secure.tls.domains[0].main=euredomain.de"
      - "traefik.http.routers.traefik-secure.tls.domains[0].sans=*.euredomain.de"

If possible I would like to only configure these things in one place, so using the CLI flags seems better to me.

11

You seem to use most of the complications possible: Cloudflare (with sometimes complicated tunneling), LE dnsChallenge, Docker socket proxy.

How about you start easy, simply try tlsChallenge, and work you way up from there. On the way always make use of Traefik dashboard and debug log.

Simple Traefik example.

12

Yeah, I think I'm getting ahead of myself. I have started stripping down components to get a config like the one you linked:

traefik:
    image: traefik:v3.0
    restart: always
    container_name: traefik
    hostname: traefik
    command:
      - "--log.level=DEBUG"
      - "--providers.docker.network=traefik"
      - "--api.dashboard=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
#      - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
#      - "--entryPoints.web.http.redirections.entrypoint.scheme=https"
      - "--entrypoints.websecure.address=:443"
      - "--entrypoints.websecure.asDefault=true"
      - "--entrypoints.websecure.http.tls.certresolver=le"
      - "--certificatesresolvers.le.acme.email=<LE EMAIL>"
      - "--certificatesresolvers.le.acme.storage=/letsencrypt/acme.json"
      - "--certificatesresolvers.le.acme.tlschallenge=true"
      - "--entrypoints.websecure.http.tls.certresolver=le"
      - "--certificatesresolvers.le.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
    networks:
      - traefik-network
    ports:
      - 443:443
      - 80:80
    volumes:
      - "./letsencrypt:/letsencrypt"
      - "./log:/log"
      - "/var/run/docker.sock:/var/run/docker.sock"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.dashboard.rule=Host(`traefik.<DOMAIN 1>`)"
      - "traefik.http.routers.dashboard.service=api@internal"
      - "traefik.http.routers.dashboard.middlewares=traefik-auth"
      - "traefik.http.middlewares.traefik-auth.basicauth.users=<AUTH CONFIG>"

In my Cloudflare configuration, I have set SSL/TLS encryption mode to Off. For now, I have content still proxied through their CDN and would like to keep it that way, but I might have to experiment with turning that off since I'm not getting a certificate at all it seems.
I first tried running Traefik with those two lines not commented out, but I got a redirected too many times error, so I tried commenting out.
Now I'm getting the default Traefik 404 page not found page. The log also contains a error relating to SSL:

traefik  | 2023-06-18T22:35:46Z INF github.com/traefik/traefik/v2/cmd/traefik/traefik.go:100 > Traefik version 3.0.0-beta2 built on 2022-12-07T16:32:34Z version=3.0.0-beta2
traefik  | 2023-06-18T22:35:46Z DBG github.com/traefik/traefik/v2/cmd/traefik/traefik.go:107 > Static configuration loaded [json] staticConfiguration={"api":{"dashboard":true},"certificatesResolvers":{"le":{"acme":{"caServer":"https://acme-staging-v02.api.letsencrypt.org/directory","certificatesDuration":2160,"email":"<LE EMAIL>","keyType":"RSA4096","storage":"/letsencrypt/acme.json","tlsChallenge":{}}}},"entryPoints":{"web":{"address":":80","forwardedHeaders":{},"http":{"redirections":{"entryPoint":{"permanent":true,"priority":2147483646,"scheme":"https","to":"websecure"}}},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s"}},"udp":{"timeout":"3s"}},"websecure":{"address":":443","asDefault":true,"forwardedHeaders":{},"http":{"tls":{"certResolver":"le"}},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s"}},"udp":{"timeout":"3s"}}},"global":{"checkNewVersion":true},"log":{"format":"common","level":"DEBUG"},"providers":{"docker":{"defaultRule":"Host(`{{ normalize .Name }}`)","endpoint":"unix:///var/run/docker.sock","network":"traefik","swarmModeRefreshSeconds":"15s","watch":true},"providersThrottleDuration":"2s"},"serversTransport":{"maxIdleConnsPerHost":200}}
traefik  | 2023-06-18T22:35:46Z INF github.com/traefik/traefik/v2/cmd/traefik/traefik.go:685 >
traefik  | Stats collection is disabled.
traefik  | Help us improve Traefik by turning this feature on :)
traefik  | More details on: https://doc.traefik.io/traefik/contributing/data-collection/
traefik  | 
traefik  | 2023-06-18T22:35:46Z INF github.com/traefik/traefik/v2/pkg/server/configurationwatcher.go:72 > Starting provider aggregator aggregator.ProviderAggregator
traefik  | 2023-06-18T22:35:46Z DBG github.com/traefik/traefik/v2/pkg/server/server_entrypoint_tcp.go:188 > Starting TCP Server entryPointName=web
traefik  | 2023-06-18T22:35:46Z DBG github.com/traefik/traefik/v2/pkg/server/server_entrypoint_tcp.go:188 > Starting TCP Server entryPointName=websecure
traefik  | 2023-06-18T22:35:46Z INF github.com/traefik/traefik/v2/pkg/provider/aggregator/aggregator.go:207 > Starting provider *acme.ChallengeTLSALPN
traefik  | 2023-06-18T22:35:46Z INF github.com/traefik/traefik/v2/pkg/provider/aggregator/aggregator.go:207 > Starting provider *traefik.Provider
traefik  | 2023-06-18T22:35:46Z DBG github.com/traefik/traefik/v2/pkg/provider/aggregator/aggregator.go:208 > *acme.ChallengeTLSALPN provider configuration config={}
traefik  | 2023-06-18T22:35:46Z DBG github.com/traefik/traefik/v2/pkg/provider/aggregator/aggregator.go:208 > *traefik.Provider provider configuration config={}
traefik  | 2023-06-18T22:35:46Z INF github.com/traefik/traefik/v2/pkg/provider/aggregator/aggregator.go:207 > Starting provider *docker.Provider
traefik  | 2023-06-18T22:35:46Z DBG github.com/traefik/traefik/v2/pkg/provider/aggregator/aggregator.go:208 > *docker.Provider provider configuration config={"defaultRule":"Host(`{{ normalize .Name }}`)","endpoint":"unix:///var/run/docker.sock","network":"traefik","swarmModeRefreshSeconds":"15s","watch":true}
traefik  | 2023-06-18T22:35:46Z INF github.com/traefik/traefik/v2/pkg/provider/aggregator/aggregator.go:207 > Starting provider *acme.Provider
traefik  | 2023-06-18T22:35:46Z DBG github.com/traefik/traefik/v2/pkg/provider/aggregator/aggregator.go:208 > *acme.Provider provider configuration config={"HTTPChallengeProvider":{},"ResolverName":"le","TLSChallengeProvider":{},"caServer":"https://acme-staging-v02.api.letsencrypt.org/directory","certificatesDuration":2160,"email":"<LE EMAIL>","keyType":"RSA4096","storage":"/letsencrypt/acme.json","store":{},"tlsChallenge":{}}
traefik  | 2023-06-18T22:35:46Z DBG github.com/traefik/traefik/v2/pkg/provider/acme/provider.go:214 > Attempt to renew certificates "720h0m0s" before expiry and check every "24h0m0s" acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory providerName=le.acme
traefik  | 2023-06-18T22:35:46Z INF github.com/traefik/traefik/v2/pkg/provider/acme/provider.go:798 > Testing certificate renew... acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory providerName=le.acme
traefik  | 2023-06-18T22:35:46Z DBG github.com/traefik/traefik/v2/pkg/server/configurationwatcher.go:217 > Configuration received config={"http":{"middlewares":{"redirect-web-to-websecure":{"redirectScheme":{"permanent":true,"port":"443","scheme":"https"}}},"models":{"websecure":{"tls":{"certResolver":"le"}}},"routers":{"web-to-websecure":{"entryPoints":["web"],"middlewares":["redirect-web-to-websecure"],"priority":2147483646,"rule":"HostRegexp(`^.+$`)","service":"noop@internal"}},"serversTransports":{"default":{"maxIdleConnsPerHost":200}},"services":{"api":{},"dashboard":{},"noop":{}}},"tcp":{},"tls":{},"udp":{}} providerName=internal
traefik  | 2023-06-18T22:35:46Z DBG github.com/traefik/traefik/v2/pkg/server/configurationwatcher.go:217 > Configuration received config={"http":{},"tcp":{},"tls":{},"udp":{}} providerName=le.acme
traefik  | 2023-06-18T22:35:46Z DBG github.com/traefik/traefik/v2/pkg/provider/docker/docker.go:217 > Provider connection established with docker 24.0.2 (API 1.43) providerName=docker
traefik  | 2023-06-18T22:35:46Z DBG github.com/traefik/traefik/v2/pkg/provider/docker/config.go:173 > Filtering disabled container container=mc-minecraft-91bdeb76667e0f7391d745f07d15412f1f810aeca905614dd239140ee2d7a137 providerName=docker
traefik  | 2023-06-18T22:35:46Z DBG github.com/traefik/traefik/v2/pkg/server/configurationwatcher.go:217 > Configuration received config={"http":{"middlewares":{"traefik-auth":{"basicAuth":{"users":["<AUTH CONFIG>"]}}},"routers":{"dashboard":{"middlewares":["traefik-auth"],"rule":"Host(`traefik.<DOMAIN 1>`)","service":"api@internal"}},"services":{"traefik-traefik":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://172.18.0.2:80"}]}}}},"tcp":{},"udp":{}} providerName=docker
traefik  | 2023-06-18T22:35:46Z DBG github.com/traefik/traefik/v2/pkg/tls/tlsmanager.go:294 > No default certificate, fallback to the internal generated certificate tlsStoreName=default
traefik  | 2023-06-18T22:35:46Z DBG github.com/traefik/traefik/v2/pkg/middlewares/tracing/forwarder.go:26 > Added outgoing tracing middleware entryPointName=web middlewareName=tracing middlewareType=TracingForwarder routerName=web-to-websecure@internal serviceName=noop@internal
traefik  | 2023-06-18T22:35:46Z DBG github.com/traefik/traefik/v2/pkg/middlewares/redirect/redirect_scheme.go:23 > Creating middleware entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme routerName=web-to-websecure@internal
traefik  | 2023-06-18T22:35:46Z DBG github.com/traefik/traefik/v2/pkg/middlewares/redirect/redirect_scheme.go:24 > Setting up redirection to https 443 entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme routerName=web-to-websecure@internal
traefik  | 2023-06-18T22:35:46Z DBG github.com/traefik/traefik/v2/pkg/middlewares/tracing/wrapper.go:32 > Adding tracing to middleware entryPointName=web middlewareName=redirect-web-to-websecure@internal routerName=web-to-websecure@internal
traefik  | 2023-06-18T22:35:46Z DBG github.com/traefik/traefik/v2/pkg/middlewares/recovery/recovery.go:22 > Creating middleware entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
traefik  | 2023-06-18T22:35:46Z DBG github.com/traefik/traefik/v2/pkg/server/aggregator.go:47 > No entryPoint defined for this router, using the default one(s) instead entryPointName=["websecure"] routerName=dashboard
traefik  | 2023-06-18T22:35:46Z DBG github.com/traefik/traefik/v2/pkg/tls/tlsmanager.go:294 > No default certificate, fallback to the internal generated certificate tlsStoreName=default
traefik  | 2023-06-18T22:35:46Z DBG github.com/traefik/traefik/v2/pkg/middlewares/tracing/forwarder.go:26 > Added outgoing tracing middleware entryPointName=web middlewareName=tracing middlewareType=TracingForwarder routerName=web-to-websecure@internal serviceName=noop@internal
traefik  | 2023-06-18T22:35:46Z DBG github.com/traefik/traefik/v2/pkg/middlewares/redirect/redirect_scheme.go:23 > Creating middleware entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme routerName=web-to-websecure@internal
traefik  | 2023-06-18T22:35:46Z DBG github.com/traefik/traefik/v2/pkg/middlewares/redirect/redirect_scheme.go:24 > Setting up redirection to https 443 entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme routerName=web-to-websecure@internal
traefik  | 2023-06-18T22:35:46Z DBG github.com/traefik/traefik/v2/pkg/middlewares/tracing/wrapper.go:32 > Adding tracing to middleware entryPointName=web middlewareName=redirect-web-to-websecure@internal routerName=web-to-websecure@internal
traefik  | 2023-06-18T22:35:46Z DBG github.com/traefik/traefik/v2/pkg/middlewares/recovery/recovery.go:22 > Creating middleware entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
traefik  | 2023-06-18T22:35:46Z DBG github.com/traefik/traefik/v2/pkg/middlewares/tracing/forwarder.go:26 > Added outgoing tracing middleware entryPointName=websecure middlewareName=tracing middlewareType=TracingForwarder routerName=dashboard@docker serviceName=api@internal
traefik  | 2023-06-18T22:35:46Z DBG github.com/traefik/traefik/v2/pkg/middlewares/auth/basic_auth.go:33 > Creating middleware entryPointName=websecure middlewareName=traefik-auth@docker middlewareType=BasicAuth routerName=dashboard@docker
traefik  | 2023-06-18T22:35:46Z DBG github.com/traefik/traefik/v2/pkg/middlewares/tracing/wrapper.go:32 > Adding tracing to middleware entryPointName=websecure middlewareName=traefik-auth@docker routerName=dashboard@docker
traefik  | 2023-06-18T22:35:46Z DBG github.com/traefik/traefik/v2/pkg/middlewares/recovery/recovery.go:22 > Creating middleware entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery
traefik  | 2023-06-18T22:35:46Z DBG github.com/traefik/traefik/v2/pkg/server/router/tcp/manager.go:235 > Adding route for traefik.<DOMAIN 1> with TLS options default entryPointName=websecure
traefik  | 2023-06-18T22:35:46Z DBG github.com/traefik/traefik/v2/pkg/provider/acme/provider.go:385 > Trying to challenge certificate for domain [traefik.<DOMAIN 1>] found in HostSNI rule acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory providerName=le.acme routerName=dashboard@docker rule=Host(`traefik.<DOMAIN 1>`)
traefik  | 2023-06-18T22:35:46Z DBG github.com/traefik/traefik/v2/pkg/provider/acme/provider.go:847 > Looking for provided certificate(s) to validate ["traefik.<DOMAIN 1>"]... acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory providerName=le.acme routerName=dashboard@docker rule=Host(`traefik.<DOMAIN 1>`)
traefik  | 2023-06-18T22:35:46Z DBG github.com/traefik/traefik/v2/pkg/provider/acme/provider.go:893 > Domains need ACME certificates generation for domains "traefik.<DOMAIN 1>". acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory domains=["traefik.<DOMAIN 1>"] providerName=le.acme routerName=dashboard@docker rule=Host(`traefik.<DOMAIN 1>`)        
traefik  | 2023-06-18T22:35:46Z DBG github.com/traefik/traefik/v2/pkg/provider/acme/provider.go:621 > Loading ACME certificates [traefik.<DOMAIN 1>]... acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory providerName=le.acme routerName=dashboard@docker rule=Host(`traefik.<DOMAIN 1>`)
traefik  | 2023-06-18T22:35:47Z DBG github.com/traefik/traefik/v2/pkg/provider/acme/provider.go:252 > Building ACME client... providerName=le.acme
traefik  | 2023-06-18T22:35:47Z DBG github.com/traefik/traefik/v2/pkg/provider/acme/provider.go:258 > https://acme-staging-v02.api.letsencrypt.org/directory providerName=le.acme
traefik  | 2023-06-18T22:35:48Z INF github.com/traefik/traefik/v2/pkg/provider/acme/provider.go:372 > Register... providerName=le.acme
traefik  | 2023-06-18T22:35:48Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [INFO] acme: Registering account for <LE EMAIL> lib=lego
traefik  | 2023-06-18T22:35:48Z DBG github.com/traefik/traefik/v2/pkg/provider/acme/provider.go:332 > Using TLS Challenge provider. providerName=le.acme
traefik  | 2023-06-18T22:35:48Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [INFO] [traefik.<DOMAIN 1>] acme: Obtaining bundled SAN certificate lib=lego
traefik  | 2023-06-18T22:35:48Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [INFO] [traefik.<DOMAIN 1>] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/<LE ID> lib=lego
traefik  | 2023-06-18T22:35:48Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [INFO] [traefik.<DOMAIN 1>] acme: use tls-alpn-01 solver lib=lego
traefik  | 2023-06-18T22:35:48Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [INFO] [traefik.<DOMAIN 1>] acme: Trying to solve TLS-ALPN-01 lib=lego
traefik  | 2023-06-18T22:35:48Z DBG github.com/traefik/traefik/v2/pkg/provider/acme/challenge_tls.go:41 > TLS Challenge Present temp certificate for traefik.<DOMAIN 1> providerName=tlsalpn.acme
traefik  | 2023-06-18T22:35:48Z DBG github.com/traefik/traefik/v2/pkg/server/configurationwatcher.go:217 > Configuration received config={"http":{},"tcp":{},"tls":{},"udp":{}} providerName=tlsalpn.acme
traefik  | 2023-06-18T22:35:48Z DBG github.com/traefik/traefik/v2/pkg/server/aggregator.go:47 > No entryPoint defined for this router, using the default one(s) instead entryPointName=["websecure"] routerName=dashboard
traefik  | 2023-06-18T22:35:48Z DBG github.com/traefik/traefik/v2/pkg/tls/certificate.go:158 > Adding certificate for domain(s) acme challenge temp,traefik.<DOMAIN 1>
traefik  | 2023-06-18T22:35:48Z DBG github.com/traefik/traefik/v2/pkg/tls/tlsmanager.go:294 > No default certificate, fallback to the internal generated certificate tlsStoreName=default
traefik  | 2023-06-18T22:35:48Z DBG github.com/traefik/traefik/v2/pkg/middlewares/tracing/forwarder.go:26 > Added outgoing tracing middleware entryPointName=web middlewareName=tracing middlewareType=TracingForwarder routerName=web-to-websecure@internal serviceName=noop@internal
traefik  | 2023-06-18T22:35:48Z DBG github.com/traefik/traefik/v2/pkg/middlewares/redirect/redirect_scheme.go:23 > Creating middleware entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme routerName=web-to-websecure@internal
traefik  | 2023-06-18T22:35:48Z DBG github.com/traefik/traefik/v2/pkg/middlewares/redirect/redirect_scheme.go:24 > Setting up redirection to https 443 entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme routerName=web-to-websecure@internal
traefik  | 2023-06-18T22:35:48Z DBG github.com/traefik/traefik/v2/pkg/middlewares/tracing/wrapper.go:32 > Adding tracing to middleware entryPointName=web middlewareName=redirect-web-to-websecure@internal routerName=web-to-websecure@internal
traefik  | 2023-06-18T22:35:48Z DBG github.com/traefik/traefik/v2/pkg/middlewares/recovery/recovery.go:22 > Creating middleware entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
traefik  | 2023-06-18T22:35:48Z DBG github.com/traefik/traefik/v2/pkg/middlewares/tracing/forwarder.go:26 > Added outgoing tracing middleware entryPointName=websecure middlewareName=tracing middlewareType=TracingForwarder routerName=dashboard@docker serviceName=api@internal
traefik  | 2023-06-18T22:35:48Z DBG github.com/traefik/traefik/v2/pkg/middlewares/auth/basic_auth.go:33 > Creating middleware entryPointName=websecure middlewareName=traefik-auth@docker middlewareType=BasicAuth routerName=dashboard@docker
traefik  | 2023-06-18T22:35:48Z DBG github.com/traefik/traefik/v2/pkg/middlewares/tracing/wrapper.go:32 > Adding tracing to middleware entryPointName=websecure middlewareName=traefik-auth@docker routerName=dashboard@docker
traefik  | 2023-06-18T22:35:48Z DBG github.com/traefik/traefik/v2/pkg/middlewares/recovery/recovery.go:22 > Creating middleware entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery
traefik  | 2023-06-18T22:35:48Z DBG github.com/traefik/traefik/v2/pkg/server/router/tcp/manager.go:235 > Adding route for traefik.<DOMAIN 1> with TLS options default entryPointName=websecure
traefik  | 2023-06-18T22:35:48Z DBG github.com/traefik/traefik/v2/pkg/provider/acme/provider.go:385 > Trying to challenge certificate for domain [traefik.<DOMAIN 1>] found in HostSNI rule acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory providerName=le.acme routerName=dashboard@docker rule=Host(`traefik.<DOMAIN 1>`)
traefik  | 2023-06-18T22:35:48Z DBG github.com/traefik/traefik/v2/pkg/provider/acme/provider.go:847 > Looking for provided certificate(s) to validate ["traefik.<DOMAIN 1>"]... acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory providerName=le.acme routerName=dashboard@docker rule=Host(`traefik.<DOMAIN 1>`)
traefik  | 2023-06-18T22:35:48Z DBG github.com/traefik/traefik/v2/pkg/provider/acme/provider.go:891 > No ACME certificate generation required for domains acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory domains=["traefik.<DOMAIN 1>"] providerName=le.acme routerName=dashboard@docker rule=Host(`traefik.<DOMAIN 1>`)
traefik  | 2023-06-18T22:35:53Z DBG github.com/traefik/traefik/v2/pkg/provider/acme/challenge_tls.go:88 > TLS Challenge CleanUp temp certificate for traefik.<DOMAIN 1> providerName=tlsalpn.acme
traefik  | 2023-06-18T22:35:53Z DBG github.com/traefik/traefik/v2/pkg/server/configurationwatcher.go:217 > Configuration received config={"http":{},"tcp":{},"tls":{},"udp":{}} providerName=tlsalpn.acme
traefik  | 2023-06-18T22:35:53Z DBG github.com/traefik/traefik/v2/pkg/server/aggregator.go:47 > No entryPoint defined for this router, using the default one(s) instead entryPointName=["websecure"] routerName=dashboard
traefik  | 2023-06-18T22:35:53Z DBG github.com/traefik/traefik/v2/pkg/tls/tlsmanager.go:294 > No default certificate, fallback to the internal generated certificate tlsStoreName=default
traefik  | 2023-06-18T22:35:53Z DBG github.com/traefik/traefik/v2/pkg/middlewares/tracing/forwarder.go:26 > Added outgoing tracing middleware entryPointName=web middlewareName=tracing middlewareType=TracingForwarder routerName=web-to-websecure@internal serviceName=noop@internal
traefik  | 2023-06-18T22:35:53Z DBG github.com/traefik/traefik/v2/pkg/middlewares/redirect/redirect_scheme.go:23 > Creating middleware entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme routerName=web-to-websecure@internal
traefik  | 2023-06-18T22:35:53Z DBG github.com/traefik/traefik/v2/pkg/middlewares/redirect/redirect_scheme.go:24 > Setting up redirection to https 443 entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme routerName=web-to-websecure@internal
traefik  | 2023-06-18T22:35:53Z DBG github.com/traefik/traefik/v2/pkg/middlewares/tracing/wrapper.go:32 > Adding tracing to middleware entryPointName=web middlewareName=redirect-web-to-websecure@internal routerName=web-to-websecure@internal
traefik  | 2023-06-18T22:35:53Z DBG github.com/traefik/traefik/v2/pkg/middlewares/recovery/recovery.go:22 > Creating middleware entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
traefik  | 2023-06-18T22:35:53Z DBG github.com/traefik/traefik/v2/pkg/middlewares/tracing/forwarder.go:26 > Added outgoing tracing middleware entryPointName=websecure middlewareName=tracing middlewareType=TracingForwarder routerName=dashboard@docker serviceName=api@internal
traefik  | 2023-06-18T22:35:53Z DBG github.com/traefik/traefik/v2/pkg/middlewares/auth/basic_auth.go:33 > Creating middleware entryPointName=websecure middlewareName=traefik-auth@docker middlewareType=BasicAuth routerName=dashboard@docker
traefik  | 2023-06-18T22:35:53Z DBG github.com/traefik/traefik/v2/pkg/middlewares/tracing/wrapper.go:32 > Adding tracing to middleware entryPointName=websecure middlewareName=traefik-auth@docker routerName=dashboard@docker
traefik  | 2023-06-18T22:35:53Z DBG github.com/traefik/traefik/v2/pkg/middlewares/recovery/recovery.go:22 > Creating middleware entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery
traefik  | 2023-06-18T22:35:53Z DBG github.com/traefik/traefik/v2/pkg/server/router/tcp/manager.go:235 > Adding route for traefik.<DOMAIN 1> with TLS options default entryPointName=websecure
traefik  | 2023-06-18T22:35:53Z DBG github.com/traefik/traefik/v2/pkg/provider/acme/provider.go:385 > Trying to challenge certificate for domain [traefik.<DOMAIN 1>] found in HostSNI rule acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory providerName=le.acme routerName=dashboard@docker rule=Host(`traefik.<DOMAIN 1>`)
traefik  | 2023-06-18T22:35:53Z DBG github.com/traefik/traefik/v2/pkg/provider/acme/provider.go:847 > Looking for provided certificate(s) to validate ["traefik.<DOMAIN 1>"]... acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory providerName=le.acme routerName=dashboard@docker rule=Host(`traefik.<DOMAIN 1>`)
traefik  | 2023-06-18T22:35:53Z DBG github.com/traefik/traefik/v2/pkg/provider/acme/provider.go:891 > No ACME certificate generation required for domains acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory domains=["traefik.<DOMAIN 1>"] providerName=le.acme routerName=dashboard@docker rule=Host(`traefik.<DOMAIN 1>`)
traefik  | 2023-06-18T22:35:53Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/<LE ID> lib=lego
traefik  | 2023-06-18T22:35:54Z ERR github.com/traefik/traefik/v2/pkg/provider/acme/provider.go:397 > Unable to obtain ACME certificate for domains error="unable to generate a certificate for the domains [traefik.<DOMAIN 1>]: error: one or more domains had a problem:\n[traefik.<DOMAIN 1>] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge\n" acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory domains=["traefik.<DOMAIN 1>"] providerName=le.acme routerName=dashboard@docker rule=Host(`traefik.<DOMAIN 1>`)

I'm really not sure what could be the problem anymore. I turned all other services off and even disabled the proxy sitting between Traefik and the Docker socket. I upgraded to the latest beta version and played with my Cloudflare config.

13

Ok, big update: I believe I have found the problem.
The problem was the Hetzner firewall. I more or less accidentally had to disable the firewall and at that moment I thought that I had one other service that was having a problem with the firewall so I tried to get a certificate and it worked.

But now there is another problem, why does it not work with the firewall? I have attached an image of my firewall configuration:

image
image1407×497 36.2 KB

At some point, I also had port 80 allowed, so that should not be a problem. Furthermore, all outgoing traffic is whitelisted... Is this maybe something worth opening an issue at either Hetzner or Traefik or both for? The firewall was very convenient and provided quite a bit of security so if possible I would like to continue to use it.

Another issue is that now I only get the default 404 page not found page when proxying my requests through Cloudflare. If I set it to DNS only I don't have any problems. SSL is set to Flexible if that might cause issues...

Current configuration:

services:
  traefik:
    image: traefik:v2.10
    restart: always
    container_name: traefik
    hostname: traefik
    command:
      - "--log.level=DEBUG"
      - "--api.dashboard=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
      - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
      - "--entrypoints.websecure.forwardedHeaders.trustedIPs=$CLOUDFLARE_IPS"
      - "--entrypoints.websecure.address=:443"
      - "--certificatesresolvers.le.acme.email=<LE EMAIL>"
      - "--certificatesresolvers.le.acme.storage=/letsencrypt/acme.json"
      - "--certificatesresolvers.le.acme.dnschallenge=true"
      - "--certificatesresolvers.le.acme.dnschallenge.provider=cloudflare"
      - "--certificatesresolvers.le.acme.dnschallenge.resolvers=1.1.1.1:53,1.0.0.1:53"
    networks:
      - traefik-network
    ports:
      - 443:443
      - 80:80
    volumes:
      - "./letsencrypt:/letsencrypt"
      - "./log:/log"
      - "/var/run/docker.sock:/var/run/docker.sock"
    environment:
      - CF_API_EMAIL=<CF EMAIL>
      - CF_API_KEY=<CF KEY>
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.dashboard.rule=Host(`traefik.<DOMAIN 1>`)"
      - "traefik.http.routers.dashboard.service=api@internal"
      - "traefik.http.routers.dashboard.entrypoints=websecure"
      - "traefik.http.routers.dashboard.tls.certresolver=le"
      - "traefik.http.routers.dashboard.middlewares=authentik-proxy@docker"
14

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.