Traefik unable to use cert for local connections?

Hi everyone! I have been using this setup for a while now, until something broke.

I’m trying to use the same certificate for both local only services, and external services, which have worked fine until now. As of today, it stopped working, and no matter what I do, I get the same error, namely that Traefik serves the default cert, and even if I say “Ignore and continue” it stalls and never connects to the service. If I instead set it up on the global one, it works fine.

Here’s my docker-compose for this scenario:

services:
traefik:
image: traefik:v3
container_name: traefik
restart: always
networks:
- traefik
extra_hosts:
- "host.docker.internal:host-gateway"
ports:
- 80:80
- 443:443
- 2080:2080
- 2443:2443
- 8050:8080
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /mnt/user/appdata/traefik/logs:/var/log
- /mnt/user/appdata/traefik/certs/:/var/certs/
- /mnt/user/appdata/traefik/config:/config/
- /mnt/user/appdata/traefik/plugins/:/plugins-local/
command:
# Setup docker provider
- --providers.docker=true
- --providers.docker.exposedByDefault=false
- --providers.docker.network=traefik
  # Setup ZeroSSL certificates
  - --certificatesresolvers.ZeroSSL.acme.email=HIDDEN
  - --certificatesresolvers.ZeroSSL.acme.storage=/var/certs/acme.json
  - --certificatesresolvers.ZeroSSL.acme.caserver=https://acme.zerossl.com/v2/HIDDEN
  - --certificatesresolvers.ZeroSSL.acme.eab.kid=HIDDEN
  - --certificatesresolvers.ZeroSSL.acme.eab.hmacencoded=HIDDEN
  - --certificatesresolvers.ZeroSSL.acme.dnschallenge=true
  - --certificatesresolvers.ZeroSSL.acme.dnschallenge.provider=cloudflare
  
  # Setup http, and make it redirect to https
  - --entrypoints.web.address=:2080
  - --entrypoints.web.http.redirections.entrypoint.to=websecure
  - --entrypoints.web.http.redirections.entrypoint.scheme=https
  # Setup https, with default certresolver and certificate
  - --entrypoints.websecure.address=:2443
  - --entrypoints.websecure.http.tls=true
  - --entrypoints.websecure.http.tls.certresolver=ZeroSSL
  - --entrypoints.websecure.http.tls.domains[0].main=MY_DOMAIN.EXT
  - --entrypoints.websecure.http.tls.domains[0].sans=*.MY_DOMAIN.EXT,*.local.MY_DOMAIN.EXT

  # Setup local routers
  - --entrypoints.web-int.address=:80
  - --entrypoints.web-int.http.redirections.entrypoint.to=websecure-int
  - --entrypoints.web-int.http.redirections.entrypoint.scheme=https
  # HTTPS
  - --entrypoints.websecure-int.address=:443
  - --entrypoints.websecure-int.http.tls=true
  - --entrypoints.websecure-int.http.tls.certresolver=ZeroSSL
  - --entrypoints.websecure-int.http.tls.domains[0].main=MY_DOMAIN.EXT
  - --entrypoints.websecure-int.http.tls.domains[0].sans=*.MY_DOMAIN.EXT,*.local.MY_DOMAIN.EXT

  # Setup logging
  - --log.level=INFO
  - --log.filepath=/var/log/traefik.log
  - --accesslog=true
  - --accesslog.filepath=/var/log/access.log
  - --accesslog.bufferingsize=100
  - --accesslog.fields.defaultmode=keep
  - --accesslog.fields.headers.defaultmode=keep
  - --accesslog.fields.headers.names.Authorization=drop

  # Setup dashboard
  - --api.insecure=false
  - --api.dashboard=true

  # Read config files for file providers
  - --providers.file.directory=/config/

  # Do not care about insecure SSL connections to local dockers
  - --serversTransport.insecureSkipVerify=true

environment:
  - CF_API_EMAIL=HIDDEN
  - CF_DNS_API_TOKEN=HIDDEN
labels:
   # Traffic jam identification
  - "isTraefik"
  # For Proxy
  - "traefik.enable=true"
  - "traefik.http.routers.traefik.entrypoints=websecure-int"
  - "traefik.http.routers.traefik.rule=Host(`traefik.local.MY_DOMAIN.EXT`)"
  - "traefik.http.routers.traefik.service=api@internal"
  - "traefik.http.routers.traefik.middlewares=authelia@docker"

trafficjam:
image: kaysond/trafficjam:latest
restart: always
network_mode: host
cap_add:
- NET_ADMIN
volumes:
- /var/run/docker.sock:/var/run/docker.sock
environment:
NETWORK: traefik
WHITELIST_FILTER: label=isTraefik
TZ: Europe/Stockholm

networks:
traefik:
external: true

What is interesting here is that everything works fine, unless you try to access anything that is under *.local.MY_DOMAIN.EXT, which gives the error about the SSL certificate (at least if trying to access locally), and if I instead make those public it gives errors about not having any TLS handshake possibility at all.

I know these certificates are valid for *.local.MY_DOMAIN.EXT from this acme.json:

{
  "ZeroSSL": {
    "Account": {
      "Email": "HIDDEN",
      "Registration": {
        "body": {
          "status": "valid",
          "contact": [
            "mailto:HIDDEN"
          ],
          "termsOfServiceAgreed": true,
          "orders": "https://acme.zerossl.com/v2/SOMETHING/account/HIDDEN/orders",
          "externalAccountBinding": {
            "payload": "HIDDEN",
            "protected": "HIDDEN"
          }
        },
        "uri": "https://acme.zerossl.com/v2/SOMETHING/account/HIDDEN"
      },
      "PrivateKey": "HIDDEN",
      "KeyType": "4096"
    },
    "Certificates": [
      {
        "domain": {
          "main": "MY_DOMAIN.EXT",
          "sans": [
            "*.MY_DOMAIN.EXT",
            "*.local.MY_DOMAIN.EXT"
          ]
        },
        "certificate": "HIDDEN",
        "key": "HIDDEN",
        "Store": "default"
      }
    ]
  }
}

What could be the reason for this behavior?

Did you update OS, Docker, Traefik recently?

Not that I recall. And this started becoming an issue as I was using it, and not according to any scheduled updates or similar....

I solved it! It was due to Cloudflare having pushed out ECH changes to all accounts by default. This made my local requests have an SNI to Traefik instance, only on the local connection, that used the domain cloudflare-ech.com. As such, Traefik always sent the wrong cert and everything broke down. This explains why the online connection worked, as it went through cloudflare and could effectively use the protocol correctly, and cloudflare handled the handoff.

If anyone else finds this problem, try turning on DEBUG level logs in traefik, see if you find that domain. If so, there's a guide for turning off ECH here, as it can only be done using the API (eg. curl, postman etc.)

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.