Invalid SSL certificate - even though I got a correct one

I might be on to something here... When I run this and go to the start.domain.io I get the error Invalid ssl certificate. When I check I can see that it is a valid certificate from Cloudflare has been fetched, the acme.json has been populated.
So what have I done wrong here?

version: '3.7'

services:

  traefik:
    image: traefik
    container_name: traefik
    restart: unless-stopped
    command:
      - --log.level=DEBUG
      - --log.filePath=./appdata/traefik/log/traefik.log
      - --accessLog.filePath=./appdata/traefik/log/access.log
      - --accessLog.bufferingSize=100
    
      - --providers.docker=true
      - --providers.docker.exposedbydefault=false

      - --api
      - --api.insecure=true
      
      - --entrypoints.traefik.address=:8888

      - --entrypoints.web.address=:80
      - --entrypoints.web.http.redirections.entrypoint.to=websecure
      - --entrypoints.web.http.redirections.entrypoint.scheme=https

      - --entrypoints.websecure.address=:443
      - --entrypoints.websecure.http.tls.certresolver=cloudflare

      - --certificatesresolvers.cloudflare.acme.dnschallenge=true
      - --certificatesresolvers.cloudflare.acme.email=${CF_API_EMAIL}
      - --certificatesresolvers.cloudflare.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
      - --certificatesresolvers.cloudflare.acme.storage=acme.json   
      - --certificatesresolvers.cloudflare.acme.dnschallenge.provider=cloudflare
    ports:
      - 80:80
      - 443:443
      - 8888:8888
    environment:
      - CF_API_EMAIL=$CF_API_EMAIL
      - CF_API_KEY=$CF_DNS_API_TOKEN
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./appdata/traefik/acme/acme.json:/acme.json    
      - ./appdata/traefik/log:/etc/traefik/log/
    labels:
      traefik.enable: true

      # Dashboard
      traefik.http.routers.traefik.rule: Host(`traefik.$LANDOMAIN`)
      traefik.http.routers.traefik.entrypoints: websecure
      traefik.http.routers.traefik.service: api@internal
      traefik.http.routers.traefik.middlewares: auth

      traefik.http.middlewares.auth.basicauth.users: $DASHBOARD_USER
      
  heimdall:
      image: ghcr.io/linuxserver/heimdall:development
      container_name: heimdall     
      environment:
        - PUID=1000
        - PGID=1000
        - TZ=${TZ}
      volumes:
        - ./appdata/heimdall:/config
      restart: unless-stopped
      labels:
        traefik.enable: true
        traefik.http.routers.heimdall.rule: Host(`start.$MAINDOMAIN`)
        traefik.http.routers.heimdall.entrypoints: websecure

This is what the dashboard shows:

Are you using staging LE caserver? You need to change it to production or live server or you will get errors.

So @kevdog I changed the part with cloudflare to this:

      - "--certificatesresolvers.letsencrypt.acme.email=$CF_API_EMAIL"
      - "--certificatesresolvers.letsencrypt.acme.dnschallenge.provider=cloudflare"
      - "--certificatesresolvers.letsencrypt.acme.dnschallenge.resolvers=1.1.1.1:53,8.8.8.8:53"
      - "--certificatesresolvers.letsencrypt.acme.storage=/home/admin/appdata/traefik/acme/acme.json"

First off the acme.json is empty, no errors. Still the same result in the browser, Invalid SSL certificate.

If you just changed it, remove your existing /home/admin/appdata/traefik/acme/acme.json before starting. The staging issued certificate will still be in the store and used, probably until it expires.

I did that or I opened it and emptied it. Is that enough?
I first shut the containers down, then I did redid the acme.json, cahnged the chmod.
Started the docker-compose with --force-recreate.
Still the same issue, Invalid SSL certificate on the start.domain.io url.
bild
This is how Firefox looks when visiting start.*
It says that the connection is secure, I do have a certificate from Cloudflare but Heimdall doesn't open.
Oh I'll add that part of the docker-compose too:

  heimdall:
      image: ghcr.io/linuxserver/heimdall:development
      container_name: heimdall     
      environment:
        - PUID=1000
        - PGID=1000
        - TZ=${TZ}
      volumes:
        - ./appdata/heimdall:/config
      restart: unless-stopped
      labels:
        traefik.enable: true
        traefik.http.routers.heimdall.rule: Host(`start.domain.io`)
        traefik.http.routers.heimdall.entrypoints: websecure

hello @macmattias

Basically speaking, once Traefik finds the matching certificate it will be presented even if it is obtained from the Lets Encrypt staging environment. If you change CAserver to production, the new certificate won't be obtained because the previous exists in the store (acme.json). The easiest is to just remove the certificate or the file or wait till the staging certificate expires. :slight_smile:

I have removed the acme.json file, I do that for every change I make that could use that file.
As I said in my post before yours, I said that I had removed it and still the same issue.
I have no idea what is happening here. I got a Cloudflare certificate still the browser says it is invalid.

Here is the docker-compose again, the latest updated version as of now. So with an EMPTY or NEW acme.json, when I run this all goes well. But as I have said all the time I get a "526 Invalid SSL certificate" using Firefox private mode. I should also att that the log files are empty. And a docker log traefik says that the log level is set to info.

version: '3.9'

services:

  traefik:
    image: traefik
    container_name: traefik
    restart: unless-stopped
    command:
      - --log.level=DEBUG
      - --log.filePath=traefik.log
      - --accessLog.filePath=access.log
      - --accessLog.bufferingSize=100
    
      - --providers.docker=true
      - --providers.docker.exposedbydefault=false

      - --api
      - --api.insecure=true
      
      - --entrypoints.traefik.address=:8888

      - --entrypoints.web.address=:80
      - --entrypoints.web.http.redirections.entrypoint.to=websecure
      - --entrypoints.web.http.redirections.entrypoint.scheme=https

      - --entrypoints.websecure.address=:443
      - --entrypoints.websecure.http.tls.certresolver=cloudflare

      #- --certificatesresolvers.cloudflare.acme.dnschallenge=true
      #- --certificatesresolvers.cloudflare.acme.email=${CF_API_EMAIL}
      #- --certificatesresolvers.cloudflare.acme.caserver=https://acme-v02.api.letsencrypt.org/directory
      #- --certificatesresolvers.cloudflare.acme.storage=acme.json   
      #- --certificatesresolvers.cloudflare.acme.dnschallenge.provider=cloudflare

      - "--certificatesresolvers.letsencrypt.acme.email=$CF_API_EMAIL"
      - "--certificatesresolvers.letsencrypt.acme.dnschallenge.provider=cloudflare"
      - "--certificatesresolvers.letsencrypt.acme.dnschallenge.resolvers=1.1.1.1:53,8.8.8.8:53"
      - "--certificatesresolvers.letsencrypt.acme.storage=acme.json"

    ports:
      - 80:80
      - 443:443
      - 8888:8888
    environment:
      - CF_API_EMAIL=$CF_API_EMAIL
      - CF_API_KEY=$CF_DNS_API_TOKEN
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./appdata/traefik/acme/acme.json:/acme.json    
      - ./appdata/traefik/traefik.log:/etc/traefik.log
      - ./appdata/traefik/access.log:/etc/access.log

    labels:
      #Dashboard
      traefik.enable: true
      traefik.http.routers.traefik.rule: Host(`traefik.landomain.io`)
      traefik.http.routers.traefik.entrypoints: websecure
      traefik.http.routers.traefik.service: api@internal
      traefik.http.routers.traefik.middlewares: auth
      traefik.http.middlewares.auth.basicauth.users: $DASHBOARD_USER
      
  heimdall:
      image: ghcr.io/linuxserver/heimdall:development
      container_name: heimdall     
      environment:
        - PUID=1000
        - PGID=1000
        - TZ=${TZ}
      volumes:
        - ./appdata/heimdall:/config
      restart: unless-stopped
      labels:
        traefik.enable: true
        traefik.http.routers.heimdall.rule: Host(`start.domain.io`)
        traefik.http.routers.heimdall.entrypoints: websecure

Are you going to obtain the certificate for that service? If so, you are missing TLS configuration in that section and Traefik is presenting the default built-in certificate instead of that matching your domain.

Try to add the following line:

  • "traefik.http.routers.heimdall.tls.certresolver=letsencrypt"

You should see the entire process in the log file of how the certificate is obtained.

See the full example here: DNS Challenge - Traefik

Letsencrypt? I changed it to cloudflare and still the same error. Tried with letsencrypt same error.

Ok your setup isn't technically wrong since I know other people use what you're trying to do with all the labels. I usually put all the letsencrypt stuff within the static configuration file since it probably never changes in between different hosts (or routers --- which is traefik speak).

So here is an example of how I set things up. Lets start with the compose file. I'm going to just list my traefik service:

secrets:
  CF_DNS_API_TOKEN_secret:
    file: /etc/docker/compose/CF_DNS_API_TOKEN.secret
  CF_ZONE_API_TOKEN_secret:
    file: /etc/docker/compose/CF_ZONE_API_TOKEN.secret

  traefik:
    image: traefik:latest
    container_name: traefik
    restart: always
    secrets:
      - CF_DNS_API_TOKEN_secret
      - CF_ZONE_API_TOKEN_secret
    networks:
      - net
    ports:
      - 80:80
      - 443:443
      - 3000:3000
    healthcheck:
      test: ${DOCKER_HEALTHCHECK_TEST:-traefik healthcheck --ping}
      interval: "30s"
      timeout: "3s"
      start_period: "5s"
      retries: 3
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=net"

      # Dashboard Routing
      - "traefik.http.routers.dashboard.rule=Host(`dash.domain.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
      - "traefik.http.routers.dashboard.tls=true"
      - "traefik.http.routers.dashboard.tls.options=modern@file"
      - "traefik.http.routers.dashboard.tls.certresolver=le"
      - "traefik.http.routers.dashboard.tls.domains[0].main=dash.domain.com"
      - "traefik.http.routers.dashboard.tls.domains[0].sans=dash.domain.com"
      - "traefik.http.routers.dashboard.service=api@internal"
      - "traefik.http.routers.dashboard.middlewares=auth"

      # Note: all dollar signs in the hash need to be doubled for escaping.
      # To create user:password pair, it's possible to use this command:
      # echo $(htpasswd -nb user password) | sed -e s/\\$/\\$\\$/g
      - "traefik.http.middlewares.auth.basicauth.users=admin:$$apr1$$nvrJ8Koh$$M5wohoBlxlygabJ2Entd/1"
      - "traefik.http.routers.dashboard.entrypoints=web,websecure"
    environment:
      - TZ
      - CLOUDFLARE_EMAIL
      - CF_DNS_API_TOKEN_FILE=/run/secrets/CF_DNS_API_TOKEN_secret
      - CF_ZONE_API_TOKEN_FILE=/run/secrets/CF_ZONE_API_TOKEN_secret
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /etc/traefik/traefik.yml:/etc/traefik/traefik.yml:ro
      - /etc/traefik/dynamic:/etc/traefik/dynamic
      - /etc/letsencrypt/certificates:/etc/letsencrypt
#      - /etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt:ro

Ok there is a lot to digest here but lets step through things.

Off the top some additional ENV variables are contained in a file called .env located in the same directory as the docker_compose.yml file. I usually put variables defined in the file like personal email names, and other stuff that if I'm sharing my services -- I don't want to distribute. Within my .env file I have the following:

CLOUDFLARE_EMAIL=email_user@gmail.com
TZ=America/Chicago
DOCKER_HEALTHCHECK_TEST=/bin/true

So for cloudflare -- please consult the documentation, but off the top of my head dns challenge with cloudflare either requires your email with global API key or your zone ID with a generated API token marked as Zone:Zone:Read and Zone:DNS:Edit. I usually use method #2 however it's slightly more confusing but it's more of the "modern" way to do things so I try to stay current. It doesn't matter what you use. I put the necessary information in two files located in the same directory as the docker_compose.yml file -- either CF_DNS_API_TOKEN.secret and CF_ZONE_API_TOKEN.secret. If your using the other method you could have a file called CF_API_KEY.secret. The names can be anything really it's just secrets needs to know how to find these. You don't have to use docker secrets however again it's technically the correct manner the docker allows for when dealing with very sensitive information such as passwords to your DNS records.

With ports I opened up 3000:3000 for me in order to do a healthcheck. Healthchecks are definitely not mandatory and if not wanting to use healthchecks you can do away with the 3000 port line and the healthcheck section -- when starting out I would not include healthchecks until you get your container up and running. It's one less variable to deal with.

Ok take a look at the labels -- when you skim down the labels -- notice there is a routers, middlewares, and service definition -- every host needs a routers and service defined and usually they have a middlewares (although middlewares is optional). If you remember this key point about every host needing a routers,middlewares,service defined -- you'll be like 10 times a head of the game. The dashboard.tls.options is using the modern@file provider -- translated this means look in the dynamic configuration file for the tls section and choose the modern object (we'll get to the dynamic configuration file in a minute).

The environment variable use TZ and CLOUDFLARE_EMAIL which are defined in the .env file.

The volumes perform a bind mount of 3 really important sections:
- /etc/traefik/traefik.yml:/etc/traefik/traefik.yml:ro
is the static configuration file mounted readonly.
- /etc/traefik/dynamic:/etc/traefik/dynamic
is the dynamic configuration directory with the dynamic configuration file is stored
- /etc/letsencrypt/certificates:/etc/letsencrypt
defines the directory where the acme.json file will be located that stores all the lets encrypt certificates

Ok lets move onto the static configuration file:

entryPoints:
  web:
    address: :80
    forwardedHeaders:
      insecure: true
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
  websecure:
    address: :443
    forwardedHeaders:
      insecure: true

  ping:
    address: :3000

certificatesResolvers:
  le:
    acme:
      email: email@gmail.com
      #Staging Server
      #caServer: https://acme-staging-v02.api.letsencrypt.org/directory
      #Production Server
      caServer: https://acme-v02.api.letsencrypt.org/directory
      storage: /etc/letsencrypt/acme.json
      keyType: 'EC384'
      dnsChallenge:
        provider: cloudflare
        delayBeforeCheck: 0
        resolvers:
          - "1.1.1.1:53"
          - "9.9.9.9:53"

#Section here is optional -- only needed if needing the Mozilla root certificates within the reverse proxy, make sure you bind mount the directory for this
serversTransport:
  insecureSkipVerify: false
  rootCAs:
    - /etc/ssl/certs/ca-certificates.crt

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
#    endpoint: "tcp://dockerproxy:2375"
    exposedbydefault: false
    watch: true
    network: net
  file:
    directory: /etc/traefik/dynamic
    watch: true

api:
#  insecure: true
  debug: true
  dashboard: true

log:
  #By default, the level is set to ERROR. Alternative logging levels are DEBUG, PANIC, FATAL, ERROR, WARN, and INFO.
  level: ERROR

ping:
  entryPoint: ping

The entryPoints define names for ports. I've defined web for port 80, websecure for 443 and ping for port 3000. 3000 port is for the healthchecks and isn't needed if not using healthchecks. Traefik can reverse proxy using either http headers which is Layer 7 or tcp headers which is Layer 4. All my examples are using http routing or layer 7. When you see the line under web that says http: --> this means use http routing (Layer 7) and not tcp.

The letsencrypt section pretty much defines a lot of things to get the LE certs

serversTransport is totally option -- I used it for a container using client/server certificates but you probably don't need this section

The next section is your providers section which is like super important. It's telling traefik that your getting additional dynamic configuration from two places ---> docker (via the form of labels) and a file (which is my case is a directory) containing the dynamic configuration. The neat thing about using dynamic configuration methods is that additional docker services can be added or stuff added in the dynamic configuration file while traefik is running, and traefik will pick up these changes automatically without having to restart. So it's kind of possible for any other services within the docker compose file to add themselves dynamically so traefik can route them -- traefik didn't need to know anything about these routes/services when it started. The rest of the static configuration file should be self explanatory

Lets move onto my dynamic configuration file which is contained within the directory defined in the static configuration. My dynamic_conf.yml file (within /etc/traefik/dynamic) looks like this:

http:

  middlewares:
    mw_compress_headers:
      compress: {}
    header-transform:
      plugin:
        header-transform-plugin:
          Rules:
            - Rule:
              Name: 'X-Client-Port Set'
              Header: 'X-Client-Port'
              Value: '^X-Forwarded-Port'
              HeaderPrefix: "^"
              Type: 'Set'

tls:
  options:
    default:
      minVersion: VersionTLS12
      sniStrict: true
      cipherSuites:
        - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
        - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
    intermediate:
      minVersion: VersionTLS12
      sniStrict: true
      cipherSuites:
        - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
        - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
    modern:
      minVersion: VersionTLS13
      sniStrict: true

The http section is for http layer 7 routing. Technically you could have routes, middlewares and services defined here for any docker service. For getting traefik up and running you don't really need anything from the http section. Another example of using the http section is below:

http:
  routers:
    office.domain.com:
      rule: "Host(`office.domain.com`)"
      entryPoints:
        - web
        - websecure
      middlewares:
        - mw_compress_headers
      service:
        - sv_proxy_pass_office.com
      tls:
        options: modern@file
        certResolver: le
        domains:
          - main: office.domain.com

  middlewares:
    mw_compress_headers:
      compress: {}
    mw_strip_prefix:
      stripPrefix:
        prefixes:
          - "/editors"
    mw_onlyoffice_redirectregex:
      redirectRegex:
        regex: "^http://(.*)"
        replacement: "https://$$1"
    mw_office_requestheader:
      headers:
        customRequestHeaders:
          X-Forwarded-Host: office.domain.com
          X-Forwarded-Proto: https

  services:
    sv_proxy_pass_office.com:
      loadBalancer:
        servers:
          - url: https://office.domain.com
        passHostHeader: true

If you just look briefly at this example -- it has Layer 7 routing defined by http: and then three sections routers, middlewares and services (which I know I've harped on this but every thing you want to redirect needs a routers and services section and usually a middlewares). These statements defined are equivalent and equal to those found in the docker file. You could use something like this in the dynamic configuration file if you were proxying to a non-docker (or docker) service like an nginx webserver running on the host machine (not in docker) for example.

The TLS section -- defines parameters for communication via the TLS handshake. I usually try to use TLS 1.3 (known as modern) for most of my stuff at home. Probably TLS 1.2 is used more commonly since its older and supports a wider range of clients. I always put the tls options in my dynamic configuration more as a reference. You can read more about TLS via the mozilla TLS generator here (it's in toml unfortunately but you get the idea): Mozzila SSL configuration for traefik

1 Like

Thank you @kevdog that was a bit to digest. But I will. I guess Ill find something to use.
I should say that the certificate that is created is TLS 1.3. It is not saved in acme.json but in use, can that be a cache issue? But I am using the private window for test.
Even the dashboard thinks that start.* is under TLS. It says that letsencrypt is the resolver.

Stop traefik container. Find your acme.json file and just delete it or move it to a different name. Restart traefik container in order to allow it to regenerate new certificates. Whenever I do this, sometimes when testing browers (Chrome, Firefox, Safari) the browser sometimes hangs onto the old certificate for a while which I'm not sure how to rectify. I'm not sure how often these browsers cache things.

Will the file be autogenerated? Because every guide I have read they do create that file empty.
When I was off spinning my brain told me something, the cloudflare settings.
SSL/TLS encryption mode is Full (strict)
And the domains, domain.io and start.domain.io was proxied, I have now turned that off for the time being to see if that helps.

I use a volume, I never pre-create a file.

You would have to create the file if you were mounting the file into the container. It can get messy with permissions, which is why I use a volume.

Ok I have tried to only use volume and it works. So thanks for that! :slight_smile:

Inspired @kevdog I studied my setup, and am I correct in that the command part can easily be moved to the static file?
Next question would it be as safe to let letsencrypt handle the SSL certificate as Cloudflare?
ATM I have 5 different docker-compose files trying different solutions, no luck yet.
Pretty much every guide I read does create an external docker network, why? The docker-compose network automatically created is external.

I was checking the documentation and I found a simple example under Docker-compose with let's encrypt: TLS Challenge. I copied that code and changed it to fit my stuff.

version: "3.3"

services:

  traefik:
    image: traefik
    container_name: traefik
    command:
      - "--log.level=DEBUG"
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.websecure.address=:443"
      - "--certificatesresolvers.letsencrypt.acme.tlschallenge=true"
      - "--certificatesresolvers.letsencrypt.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
      - "--certificatesresolvers.letsencrypt.acme.email=my@mail.com"
      - "--certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json"
    ports:
      - "443:443"
      - "8080:8080"
    volumes:
      - "./appdata/traefik/acme:/letsencrypt"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"

  whoami:
    image: "traefik/whoami"
    container_name: "simple-service"
    labels:
      - "traefik.enable=true"
     #mtrx is a valid subdomain, whoami is not.
      - "traefik.http.routers.whoami.rule=Host(`mtrx.$MAINDOMAIN`)"
      - "traefik.http.routers.whoami.entrypoints=websecure"
      - "traefik.http.routers.whoami.tls.certresolver=letsencrypt"
  
  heimdall:
    image: ghcr.io/linuxserver/heimdall:development
    container_name: heimdall     
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=${TZ}
    volumes:
      - ./appdata/heimdall:/config
    restart: unless-stopped
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.heimdall.rule=Host(`start.$MAINDOMAIN`)"
      - "traefik.http.routers.heimdall.entrypoints=websecure"
      - "traefik.http.routers.heimdall.tls.certresolver=letsencrypt"

Now I at least get some debug log!

Here you also will see my domain, so be nice.
But still the same result as always, 526 invalid SSL... bla bla
Will it be easier to switch to letsencrypt all together?

The domain you are having an issue with is not pointing to the same place as start.<domain> so, check your DNS.

Your kind of going off on your own on this own and trying a lot of different things..

What do you want to do -- CF DNS challenge or something else? Right now your are using TLS challenge which I've never used -- only DNS challenge.

Your still using staging servers.

You don't have any static configuration file assigned to your traefik container -- do you want this??

Are start.$MAINDOMAIN and mtrx.$MAINDOMAIN registered with cloudflare and can be resolved to a CNAME or A record?

It's DNS it is always DNS, is that what you are saying? :wink:
That is strange, since $MAINDOMAIN is a A record at Cloudflare and start.* is an CNAME record at the same place.
When I ping them, I do get an answer from the same IP. You migth have been checking just when I switched from proxied dns to none.