Labeled containers get correct SSL certificate, file-defined containers get self-signed

Traefik Traefik v2
,
1

Ok, that title describes the issue, but yields nothing helpful when searched in Google, not surprising.

TL;DR: Issue is that I basically had to start from scratch and I'm 95% functioning, except my wildcard certificate is only being applied to containers with labels. I have 3 other systems with containers so labels won't work (I don't think), so I've manually created those routers and services in the config.yml file. This used to work, but then I got an email about my certificates about to expire, so I went to troubleshoot that and somehow lost my entire config (hence the starting from scratch. Anything that's defined by labels, I get valid SSL hostnames for and they're entered into the acme.json. However, if it's defined within the config.yml file, I get the Traefik self-signed SSL certificate. A side note, when my setup was working, the config.yml file wasn't dynamically scraped despite being told to do so; any changes required a restart of the container to be picked up.

The not so TL;DR...
My docker-compose.yml:

---
services:
  traefik:
    image: traefik:latest
    container_name: traefik
    security_opt:
      - no-new-privileges:true
    networks:
      - traefik_default
    ports:
      - 80:80
      - 443:443
      - 8081:8080  # (optional) expose the dashboard !don't use in production!
    environment:
      - CLOUDFLARE_EMAIL=[my email address]
      - CLOUDFLARE_DNS_API_TOKEN=[my Cloudflare DNS API token (permissions are correct, just rolled it today to be sure and, again, this used to work)]
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /home/user/appdata/traefik/traefik.yml:/etc/traefik/traefik.yml:ro
      - /home/user/appdata/traefik/config.yml:/config.yml:ro
      - /home/user/appdata/traefik/letsencrypt:/letsencrypt
      - /home/user/appdata/traefik/logs:/var/log/traefik/traefik.log
      - /var/run/docker.sock:/var/run/docker.sock:ro
    restart: unless-stopped
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.rule=Host(`traefik.mydomain.com`)"
      - "traefik.http.routers.traefik.entrypoints=websecure"
      - "traefik.http.routers.traefik.tls.certresolver=cf_production"
      - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=websecure"
      - "traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=https"
      - "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
      - "traefik.http.routers.traefik-secure.entrypoints=websecure"
      - "traefik.http.routers.traefik-secure.rule=Host(`traefik.mydomain.com`)"
    #  - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
      - "traefik.http.routers.traefik-secure.tls=true"
      - "traefik.http.routers.traefik-secure.tls.domains[0].main=mydomain.com"
      - "traefik.http.routers.traefik-secure.tls.domains[0].sans=*.mydomain.com"
      - "traefik.http.routers.traefik-secure.service=api@internal"

  whoami:
    image: "traefik/whoami"
    container_name: "whoami"
    networks:
      - traefik_default
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami.rule=Host(`whoami.mydomain.com`)"
      - "traefik.http.routers.whoami.entrypoints=websecure"
      - "traefik.http.routers.whoami.tls.certresolver=cf_production"
      - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=websecure"
      - "traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=https"

networks:
  traefik_default:
    external: true

My traefik.yml:

global:
  checkNewVersion: true
  sendAnonymousUsage: false  # true by default

# (Optional) Log information
# ---
log:
  level: DEBUG  # DEBUG, INFO, WARNING, ERROR, CRITICAL
  format: common  # common, json, logfmt
  filePath: /var/log/traefik/traefik.log

# (Optional) Accesslog
# ---
#accesslog:
#  format: common  # common, json, logfmt
#  filePath: /var/log/traefik/access.log

# (Optional) Enable API and Dashboard
# ---
api:
  dashboard: true  # true by default
  insecure: true  # Don't do this in production!
  debug: true

# Entry Points configuration
# ---
entryPoints:
  web:
    address: :80
    # (Optional) Redirect to HTTPS
    # ---
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https

  websecure:
    address: :443

#serversTransport:
#  # Makes it so we don't have to specify this for any self-signed SSL HTTPS endpoints
#  insecureSkipVerify: true

# Configure your CertificateResolver here...
# ---
certificatesResolvers:
  cf_staging:
    acme:
      email: [my@email.com]
      storage: /letsencrypt/acme.json
      caServer: "https://acme-staging-v02.api.letsencrypt.org/directory"
      dnschallenge:
        provider: cloudflare
        resolvers:
          - "1.1.1.1:53"
          - "1.0.0.1:53"
      #httpChallenge:
      #  entryPoint: web

  cf_production:
    acme:
      email: [my@email.com]
      storage: /letsencrypt/acme.json
      caServer: "https://acme-v02.api.letsencrypt.org/directory"
      dnschallenge:
        provider: cloudflare
        resolvers:
          - "1.1.1.1:53"
          - "1.0.0.1:53"
      # httpChallenge:
      #   entryPoint: web

# (Optional) Overwrite Default Certificates
tls:
#  stores:
#    default:
#      defaultCertificate:
#        certFile: /letsencrypt/cert.pem
#        keyFile: /letsencrypt/cert-key.pem
# (Optional) Disable TLS version 1.0 and 1.1
  options:
    default:
      minVersion: VersionTLS12

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false  # Default is true
  file:
    # watch for dynamic configuration changes
    filename: /config.yml
    watch: true

Now with this setup, I spin it up and I get valid SSL on traefik.mydomain.com and whoami.mydomain.com.

But here are a couple router and service definitions from the config.yml that get self-siged Traefik default certificates (for the sake of shortening an already long post, I'll cut out the redundancy):

http:
# Start of routers
  routers:
    speedtesttracker:
      entryPoints:
        - "websecure"
      rule: "Host(`speedtrack.mydomain.com`)"
      middlewares:
        - default-headers
        - https-redirectscheme
      tls: {}
      service: speedtesttracker

[CUT FOR LENGTH]

  services:
    speedtesttracker:
      loadBalancer:
        servers:
          - url: "http://nasu.local:8083"
        passHostHeader: true

There are dozens of router and service definitions that are 99% identical. Then the tail end of the config.yml file if it matters:

  middlewares:
    addprefix-pihole:
      addPrefix:
        prefix: "/admin"
    https-redirectscheme:
      redirectScheme:
        scheme: https
        permanent: true

    default-headers:
      headers:
        frameDeny: true
        browserXssFilter: true
        contentTypeNosniff: true
        forceSTSHeader: true
        stsIncludeSubdomains: true
        stsPreload: true
        stsSeconds: 15552000
        customFrameOptionsValue: SAMEORIGIN
        customRequestHeaders:
          X-Forwarded-Proto: https

    default-whitelist:
      ipWhiteList:
        sourceRange:
        - "10.0.0.0/8"
        - "192.168.0.0/16"
        - "172.16.0.0/12"

    secured:
      chain:
        middlewares:
        - default-whitelist
        - default-headers

And docker-compose logs -f output (this just repeats per router definition, like it's expecting the URL to be TLS, which it's not):

traefik    | time="2023-11-27T15:24:55-06:00" level=debug msg="Serving default certificate for request: \"comm.mydomain.com\""
traefik    | time="2023-11-27T15:24:55-06:00" level=debug msg="http: TLS handshake error from 192.168.10.80:55730: remote error: tls: unknown certificate"

traefik     time="2023-11-27T15:23:44-06:00" level=debug msg="Adding route for comm.mydomain.com with TLS options default" entryPointName=websecure

traefik     time="2023-11-27T15:23:44-06:00" level=debug msg="Adding certificate for domain(s) traefik.mydomain.com"
traefik     time="2023-11-27T15:23:44-06:00" level=debug msg="Adding certificate for domain(s) whoami.mydomain.com"
traefik     time="2023-11-27T15:23:44-06:00" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default

So my primary focus is why the wildcard certificate is either A. not being obtained and applied to the routers and services defined in the config.yml file (which do show up in the web interface) or B. the wildcard certificate is being obtained, just not applied to the routers and services definitions

Then, secondary to that, why does my config.yml file not get dynamically updated without restarting the Traefik container? I've tried watching a directory, specifying a filename, nothing seemed to work.

And again, this all was working but I had to rebuild the docker-compose from scratch based on various sources including on here, my 10 week old memory, YouTube, other internet sites, etc. so I'd imagine that's where the flaw(s) is/will be.

2

Well, if you use tls: {}, you get self-signed certs (unless you loaded custom certs via dynamic config file). Add the certresolver for LE.

Or add the certresolver globally to your https entrypoint, see simple Traefik example.

1 Like
3

THANK YOU. Man this was frustrating. So it seems, when I initially set it up, all was well, and I must've been doing something weird or I didn't understand to have added the tls: {}.

So here's what I did (just in case someone else runs into this):
In the config.yml file, commented out all instances of tls: {}

I did have to do some digging around to set the default the default certresolver via the file, but then I realized I could take your example and convert it to the file (facepalm moment).
So your example is: --entrypoints.websecure.http.tls.certresolver=myresolver
And the file ended up being:

  websecure:
    address: :443
    http:
      tls:
        certresolver: cf_production
        domains:
          - main: mydomain.com
            sans:
              - "*.mydomain.com"

Restarted Traefik and now my certs are valid.

I cannot thank you enough!

4

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.