Cannot get custom SSL certificate to work

Hello all,

I'm trying to move from traefik v1 to v2. My setup is fairly simple one, traefik acts as reverse proxy for my backend apps, while redirecting every http to https and terminates the SSL. For that I have a wildcard certificate from Let'sEncrypt (issued and renewed on my own, not via Traefik, as I use a custom local provider whose API is not supported in Traefik).

What works:
Traefik listens on 80 and 443. It is able to redirect http to https, terminates the SSL and then forwards traffic to the correct backend.

What does not work:
In doing so, traefik is always using its own generated certificate. I tried several options how to link and use my existing LE wildcard certificates, but none of them worked. It may be that I just don't get the documentation right, but I have also seen some errors in the logs related to certificates.

Here are the relevant pieces of configuration:
Traefik docker-compose.yml

version: "3.3"
services:
  traefik:
    image: traefik:v2.0
    container_name: "traefik"
    networks:
      - net_frontend
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - ./config/static/traefik.yml:/etc/traefik/traefik.yml
      - ./config/dynamic/:/dynconf
      - ./../shared/fullchain.pem:/certs/fullchain.pem
      - ./../shared/privkey.pem:/certs/privkey.pem
#      - v_shared:/certs

volumes:
  v_shared:
    external:
      name: v_shared

networks:
  net_frontend:
    external:
      name: net_frontend

Note: I also tried mapping the cert.pem file, tried mapping the whole certs directory etc. Each time certificates are available to Traefik, but not used.

Static configuration file /etc/traefik/traefik.yml:

api:
  insecure: true
  dashboard: true
log:
  level: debug
providers:
  docker:
    watch: true
    exposedByDefault: false
    network: net_frontend
  file:
    directory: /dynconf
    watch: true
#    filename: foobar
entryPoints:
  web:
    address: :80
  websecure:
    address: :443

Content of the dynamic configuration file /dynconf/dyn1.yml

http:
  middlewares:
    https-redirect:
      redirectScheme:
        scheme: https

tls:
  certificates:
    - certFile: /certs/fullchain.pem
      keyfile: /certs/privkey.pem
      stores:
        - default
  stores:
    default:
      defaultCertificate:
        certFile: path/to/cert.crt
        keyFile: path/to/cert.key

And finally relevanr docker-compose sections of my (for the testing only 2 apps):
Who-am-I:

version: "3.3"
services:
  whoami:
    image: "containous/whoami"
    container_name: "simple-service"
    expose:
      - "80"
    networks:
      - net_frontend
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.rW.entrypoints=web"
      - "traefik.http.routers.rW.rule=Host(`b.example.com`)"
      - "traefik.http.routers.rW.middlewares=https-redirect@file"
      - "traefik.http.routers.rWs.entrypoints=websecure"
      - "traefik.http.routers.rWs.rule=Host(`b.example.com`)"
      - "traefik.http.routers.rWs.tls=true"

networks:
  net_frontend:
    external:
      name: net_frontend

And sonarr for example looks like this:

version: "3.5"
services:
  sonarr:
    image: "linuxserver/sonarr"
    container_name: "sonarr"
    volumes:
      - v_sonarr_conf:/config
      - v_sabnzbd_dl_compl:/downloads
      - v_synoTVS:/tv
      - v_shared:/shared
    expose:
      - "8989"
    restart: always
    environment:
      - PUID=${PUID}
      - PGID=${PGID}
      - TZ=${TZ}
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.rX.entrypoints=web"
      - "traefik.http.routers.rX.rule=Host(`c.example.com`)"
      - "traefik.http.routers.rX.middlewares=https-redirect@file"
      - "traefik.http.routers.rXs.entrypoints=websecure"
      - "traefik.http.routers.rXs.rule=Host(`c.example.com`)"
      - "traefik.http.routers.rXs.tls=true"
    networks:
      - net_frontend

volumes:
...(bunch of volume definitions)...

networks:
  net_frontend:
    external:
      name: net_frontend

I have a wildcard certificate issued for my domain (in above examples replaced by "example.com").
Over the course of several days I have tried different approaches - using the same routers for each app (this did not work), so I ended up having a specific router for each app. Any combination of how to force traefik to use my certificates has failed. So there must be something I am missing.

The errors I am seeing in traefik debug log are:

traefik    | time="2019-11-12T07:46:58Z" level=error msg="Error while creating certificate store: failed to load X509 key pair: tls: failed to find any PEM data in certificate input" tlsStoreName=default
traefik    | time="2019-11-12T07:46:58Z" level=error msg="Unable to append certificate MIIFbjCCBFagAwIBAgISAw2TL3TEYbnBZpCjQN2euFThMA0GCS to store: unable to generate TLS certificate : tls: failed to find any PEM data in key input" tlsStoreName=default
traefik    | time="2019-11-12T07:46:58Z" level=debug msg="No default certificate, generating one"

Finally, when I attach to container, here are the certificate files and their permissions:

/certs # ls -lah
total 16K
drwxr-xr-x    2 root     root        4.0K Nov 11 22:07 .
drwxr-xr-x    1 root     root        4.0K Nov 11 22:07 ..
-rw-r--r--    1 root     root        3.5K Nov  7 11:45 fullchain.pem
-r--------    1 root     root        1.7K Nov  7 11:45 privkey.pem

So please, if you can give me a hint what am I doing wrong or what to try next, I would be greatfull.

Thanks,
B.

There are few things you need to try out.

  1. Change the permission of certificates to 600
  2. In traefik.yml file point directly to the file provider -
file:
  filename: /dynconf/dyn1.yml
  1. In dyn1.yml make a small change by adding double quotes to the path of the certificates.
tls:
  certificates:
    - certFile: "/certs/fullchain.pem"
      keyfile: "/certs/privkey.pem"
      stores:
        - default
  stores:
    default:
      defaultCertificate:
        certFile: "path/to/cert.crt"
        keyFile: "path/to/cert.key"

Hello,

thank you for your answer.
I started applying your suggestions one at a time to see which one solves the problem. In the second step I nailed it... I was going to put the paths into quotes when I realized that my default certificate paths are copy/paste from the documentation and I forgot to actually update the paths(!) :woozy_face: What I stupid mistake and took me forever to troubleshoot.

After updating that path the certificate got picked up.

In logs I still see one error though:

traefik    | time="2019-11-12T13:05:36Z" level=error msg="Unable to append certificate MIIFbjCCBFagAwIBAgISAw2TL3TEYbnBZpCjQN2euFThMA0GCS to store: unable to generate TLS certificate : tls: failed to find any PEM data in key input" tlsStoreName=default

but since it is working now, I guess this could be perhaps ignored (?).

I will apply your suggestions as the "best practice", just to make sure.

Thanks again.
B.