Docker Compose + Custom SSL WildCard Cert = failed to find any PEM data in certificate input

Traefik Traefik v2
1

Hi all,

I am trying to figure out why I cannot seem to get my wildcard cert to pick up in my configuration. I am only using docker-compose, not swarm and I am using an existing certificate pair, not Acme. It looks like the cert isn't picked up because the certificates.yml file isn't read. If anyone could please provide some insight and suggestions I'd really appreciate it.

file structure

  • docker-compose-lite.yml
  • /configuration
    • certificates.yml
    • wildcard.pem
    • wildcard.cer

docker-compose-lite.yml

version: "3"

services:
  reverse-proxy:
    image: traefik:v2.5
    command:
      - --log.level=DEBUG
      - --log.filePath=/var/log/traefik/traefik.log
      - --log.format=json
      - --accesslog=true
      - --accesslog.filePath=/var/log/traefik/access.log
      - --api.insecure=true
      - --entrypoints.https.address=:443
      - --providers.docker
      - --providers.docker.exposedbydefault=false
      # look in a directory *** in the container *** for provider configuration
      - --providers.file.directory=/configuration
      - --providers.file.watch=true
    ports:
      - "443:443"
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /var/log:/var/log
      # Mount a directory in the root of my project into the container. This directory contains the local certificate for the project.
      - ./configuration:/configuration
      # Second, I mount a yaml file into the container, located in the directory we configured to be the provider directory from the command section
      - ./configuration/certificates.yml:/configuration/certificates.yml
    labels:
      - "traefik.enabled=true"
    networks:
      - traefik-default
  whoami:
    # A container that exposes an API to show its IP address
    image: containous/whoami
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami.rule=Host(`whoami.wildcard`)"
      - "traefik.http.routers.whoami.entrypoints=https"
      - "traefik.http.routers.whoami.tls=true"
    networks:
      - traefik-default
networks:
  traefik-default:
    external: true

certificates.yml

tls:
  certificates:
    certFile: ./configuration/wildcard_bundle.crt
    keyFile: ./configuration/wildcard.pem

/var/log/traefik/traefik.log

{"level":"info","msg":"Traefik version 2.5.3 built on 2021-09-20T15:43:56Z","time":"2021-12-10T15:00:42Z"}
{"level":"debug","msg":"Static configuration loaded {\"global\":{\"checkNewVersion\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"https\":{\"address\":\":443\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"udp\":{\"timeout\":\"3s\"}},\"traefik\":{\"address\":\":8080\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"udp\":{\"timeout\":\"3s\"}}},\"providers\":{\"providersThrottleDuration\":\"2s\",\"docker\":{\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"swarmModeRefreshSeconds\":\"15s\"},\"file\":{\"directory\":\"/configuration\",\"watch\":true}},\"api\":{\"insecure\":true,\"dashboard\":true},\"log\":{\"level\":\"DEBUG\",\"filePath\":\"/var/log/traefik/traefik.log\",\"format\":\"json\"},\"accessLog\":{\"filePath\":\"/var/log/traefik/access.log\",\"format\":\"common\",\"filters\":{},\"fields\":{\"defaultMode\":\"keep\",\"headers\":{\"defaultMode\":\"drop\"}}},\"pilot\":{\"dashboard\":true}}","time":"2021-12-10T15:00:42Z"}
{"level":"info","msg":"\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\n","time":"2021-12-10T15:00:42Z"}
{"level":"info","msg":"Starting provider aggregator.ProviderAggregator {}","time":"2021-12-10T15:00:42Z"}
{"entryPointName":"traefik","level":"debug","msg":"Start TCP Server","time":"2021-12-10T15:00:42Z"}
{"level":"info","msg":"Starting provider *file.Provider {\"directory\":\"/configuration\",\"watch\":true}","time":"2021-12-10T15:00:42Z"}
{"entryPointName":"https","level":"debug","msg":"Start TCP Server","time":"2021-12-10T15:00:42Z"}
{"level":"info","msg":"Starting provider *traefik.Provider {}","time":"2021-12-10T15:00:42Z"}
{"level":"info","msg":"Starting provider *acme.ChallengeTLSALPN {\"Timeout\":4000000000}","time":"2021-12-10T15:00:42Z"}
{"level":"debug","msg":"Configuration received from provider file: {\"http\":{},\"tcp\":{},\"udp\":{},\"tls\":{}}","providerName":"file","time":"2021-12-10T15:00:42Z"}
{"level":"debug","msg":"Configuration received from provider internal: {\"http\":{\"routers\":{\"api\":{\"entryPoints\":[\"traefik\"],\"service\":\"api@internal\",\"rule\":\"PathPrefix(`/api`)\",\"priority\":2147483646},\"dashboard\":{\"entryPoints\":[\"traefik\"],\"middlewares\":[\"dashboard_redirect@internal\",\"dashboard_stripprefix@internal\"],\"service\":\"dashboard@internal\",\"rule\":\"PathPrefix(`/`)\",\"priority\":2147483645}},\"services\":{\"api\":{},\"dashboard\":{},\"noop\":{}},\"middlewares\":{\"dashboard_redirect\":{\"redirectRegex\":{\"regex\":\"^(http:\\\\/\\\\/(\\\\[[\\\\w:.]+\\\\]|[\\\\w\\\\._-]+)(:\\\\d+)?)\\\\/$\",\"replacement\":\"${1}/dashboard/\",\"permanent\":true}},\"dashboard_stripprefix\":{\"stripPrefix\":{\"prefixes\":[\"/dashboard/\",\"/dashboard\"]}}},\"serversTransports\":{\"default\":{\"maxIdleConnsPerHost\":200}}},\"tcp\":{},\"tls\":{}}","providerName":"internal","time":"2021-12-10T15:00:42Z"}
{"level":"info","msg":"Starting provider *docker.Provider {\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"swarmModeRefreshSeconds\":\"15s\"}","time":"2021-12-10T15:00:42Z"}
{"level":"debug","msg":"No default certificate, generating one","time":"2021-12-10T15:00:42Z","tlsStoreName":"default"}
{"level":"debug","msg":"Provider connection established with docker 20.10.10 (API 1.41)","providerName":"docker","time":"2021-12-10T15:00:42Z"}
{"level":"error","msg":"Skip container reverse-proxy-docker: field not found, node: enabled","providerName":"docker","time":"2021-12-10T15:00:42Z"}
{"level":"debug","msg":"Configuration received from provider docker: {\"http\":{\"routers\":{\"whoami\":{\"entryPoints\":[\"https\"],\"service\":\"whoami-docker\",\"rule\":\"Host(`whoami.wildcard`)\",\"tls\":{}}},\"services\":{\"whoami-docker\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://192.168.64.3:80\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"udp\":{}}","providerName":"docker","time":"2021-12-10T15:00:42Z"}
{"level":"debug","msg":"No store is defined to add the certificate , it will be added to the default store.","time":"2021-12-10T15:00:43Z"}
{"level":"error","msg":"Unable to append certificate  to store: unable to generate TLS certificate : tls: failed to find any PEM data in certificate input","time":"2021-12-10T15:00:43Z","tlsStoreName":"default"}
{"level":"debug","msg":"No store is defined to add the certificate , it will be added to the default store.","time":"2021-12-10T15:00:43Z"}
{"level":"error","msg":"Unable to append certificate  to store: unable to generate TLS certificate : tls: failed to find any PEM data in certificate input","time":"2021-12-10T15:00:43Z","tlsStoreName":"default"}
{"entryPointName":"traefik","level":"debug","middlewareName":"tracing","middlewareType":"TracingForwarder","msg":"Added outgoing tracing middleware api@internal","routerName":"api@internal","time":"2021-12-10T15:00:43Z"}
{"entryPointName":"traefik","level":"debug","middlewareName":"tracing","middlewareType":"TracingForwarder","msg":"Added outgoing tracing middleware dashboard@internal","routerName":"dashboard@internal","time":"2021-12-10T15:00:43Z"}
{"entryPointName":"traefik","level":"debug","middlewareName":"dashboard_stripprefix@internal","middlewareType":"StripPrefix","msg":"Creating middleware","routerName":"dashboard@internal","time":"2021-12-10T15:00:43Z"}
{"entryPointName":"traefik","level":"debug","middlewareName":"dashboard_stripprefix@internal","msg":"Adding tracing to middleware","routerName":"dashboard@internal","time":"2021-12-10T15:00:43Z"}
{"entryPointName":"traefik","level":"debug","middlewareName":"dashboard_redirect@internal","middlewareType":"RedirectRegex","msg":"Creating middleware","routerName":"dashboard@internal","time":"2021-12-10T15:00:43Z"}
{"entryPointName":"traefik","level":"debug","middlewareName":"dashboard_redirect@internal","middlewareType":"RedirectRegex","msg":"Setting up redirection from ^(http:\\/\\/(\\[[\\w:.]+\\]|[\\w\\._-]+)(:\\d+)?)\\/$ to ${1}/dashboard/","routerName":"dashboard@internal","time":"2021-12-10T15:00:43Z"}
{"entryPointName":"traefik","level":"debug","middlewareName":"dashboard_redirect@internal","msg":"Adding tracing to middleware","routerName":"dashboard@internal","time":"2021-12-10T15:00:43Z"}
{"entryPointName":"traefik","level":"debug","middlewareName":"traefik-internal-recovery","middlewareType":"Recovery","msg":"Creating middleware","time":"2021-12-10T15:00:43Z"}
{"level":"debug","msg":"No default certificate, generating one","time":"2021-12-10T15:00:43Z","tlsStoreName":"default"}
{"level":"error","msg":"Unable to append certificate  to store: unable to generate TLS certificate : tls: failed to find any PEM data in certificate input","time":"2021-12-10T15:00:44Z","tlsStoreName":"default"}
{"level":"error","msg":"Unable to append certificate  to store: unable to generate TLS certificate : tls: failed to find any PEM data in certificate input","time":"2021-12-10T15:00:44Z","tlsStoreName":"default"}
{"entryPointName":"traefik","level":"debug","middlewareName":"tracing","middlewareType":"TracingForwarder","msg":"Added outgoing tracing middleware api@internal","routerName":"api@internal","time":"2021-12-10T15:00:44Z"}
{"entryPointName":"traefik","level":"debug","middlewareName":"tracing","middlewareType":"TracingForwarder","msg":"Added outgoing tracing middleware dashboard@internal","routerName":"dashboard@internal","time":"2021-12-10T15:00:44Z"}
{"entryPointName":"traefik","level":"debug","middlewareName":"dashboard_stripprefix@internal","middlewareType":"StripPrefix","msg":"Creating middleware","routerName":"dashboard@internal","time":"2021-12-10T15:00:44Z"}
{"entryPointName":"traefik","level":"debug","middlewareName":"dashboard_stripprefix@internal","msg":"Adding tracing to middleware","routerName":"dashboard@internal","time":"2021-12-10T15:00:44Z"}
{"entryPointName":"traefik","level":"debug","middlewareName":"dashboard_redirect@internal","middlewareType":"RedirectRegex","msg":"Creating middleware","routerName":"dashboard@internal","time":"2021-12-10T15:00:44Z"}
{"entryPointName":"traefik","level":"debug","middlewareName":"dashboard_redirect@internal","middlewareType":"RedirectRegex","msg":"Setting up redirection from ^(http:\\/\\/(\\[[\\w:.]+\\]|[\\w\\._-]+)(:\\d+)?)\\/$ to ${1}/dashboard/","routerName":"dashboard@internal","time":"2021-12-10T15:00:44Z"}
{"entryPointName":"traefik","level":"debug","middlewareName":"dashboard_redirect@internal","msg":"Adding tracing to middleware","routerName":"dashboard@internal","time":"2021-12-10T15:00:44Z"}
{"entryPointName":"traefik","level":"debug","middlewareName":"traefik-internal-recovery","middlewareType":"Recovery","msg":"Creating middleware","time":"2021-12-10T15:00:44Z"}
{"entryPointName":"https","level":"debug","middlewareName":"pipelining","middlewareType":"Pipelining","msg":"Creating middleware","routerName":"whoami@docker","serviceName":"whoami-docker","time":"2021-12-10T15:00:44Z"}
{"entryPointName":"https","level":"debug","msg":"Creating load-balancer","routerName":"whoami@docker","serviceName":"whoami-docker","time":"2021-12-10T15:00:44Z"}
{"entryPointName":"https","level":"debug","msg":"Creating server 0 http://192.168.64.3:80","routerName":"whoami@docker","serverName":0,"serviceName":"whoami-docker","time":"2021-12-10T15:00:44Z"}
{"level":"debug","msg":"child http://192.168.64.3:80 now UP","time":"2021-12-10T15:00:44Z"}
{"level":"debug","msg":"Propagating new UP status","time":"2021-12-10T15:00:44Z"}
{"entryPointName":"https","level":"debug","middlewareName":"tracing","middlewareType":"TracingForwarder","msg":"Added outgoing tracing middleware whoami-docker","routerName":"whoami@docker","time":"2021-12-10T15:00:44Z"}
{"entryPointName":"https","level":"debug","middlewareName":"traefik-internal-recovery","middlewareType":"Recovery","msg":"Creating middleware","time":"2021-12-10T15:00:44Z"}
{"entryPointName":"https","level":"debug","msg":"Adding route for whoami.wildcard with TLS options default","time":"2021-12-10T15:00:44Z"}
2

hey,

I use it in my traefik.toml and it works fine

[[tls.certificates]]
  certFile = "/var/traefik2/tls/mcert.crt"
  keyFile = "/var/traefik2/tls/mkey.key"
  stores = ["default"]

[tls.stores]
  [tls.stores.default]
    [tls.stores.default.defaultCertificate]
      certFile = "/var/traefik2/tls/mcert.crt"
      keyFile = "/var/traefik2/tls/mkey.key"
3

Having never successfully used my own certificates for traefik (for some reason people think Let's Encrypt is a good thing and say nothing about any other SSL configuration), I have no idea if this is your problem, but it is a problem. Your certificates.yml file references the crt and pem file in ./configuration while you map it into your container at /configuration. Those aren't always the same thing. Change your certificates.yml file to use /configuration instead and try it again.

Also, you shouldn't need the second volume mount for your yaml file if you mount the whole directory right above it.

4

When I initially stumbled on this post, I was trying to figure out how to handle SSLs with my own certificates. I discovered something today (completely by chance/accident) that cleared up everything and made it all start working: You need to have a certresolver defined somewhere, anywhere, for it to be able to resolve the provided certificates. Given what I know know, here is what I see wrong with your configuration.

version: "3"

services:
  reverse-proxy:
    image: traefik:v2.5
    command:
      - --log.level=DEBUG
      - --log.filePath=/var/log/traefik/traefik.log
      - --log.format=json
      - --accesslog=true
      - --accesslog.filePath=/var/log/traefik/access.log
      - --api.insecure=true
      - --entrypoints.https.address=:443
      - --providers.docker
      - --providers.docker.exposedbydefault=false
      # look in a directory *** in the container *** for provider configuration
      - --providers.file.directory=/configuration
      - --providers.file.watch=true
      --providers.file.filename=/configuration/certificates.yml  # *** this defined the "dynamic" config file... apparently.  This will get your config file read.  You might also consider "secrets" in swarm mode if you go that route.
    ports:
      - "443:443"
#      - "8080:8080"  # *** Remove this.  You don't need it if you have the dashboard proxied by traefik over https.
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /var/log:/var/log
      # Mount a directory in the root of my project into the container. This directory contains the local certificate for the project.
      - ./configuration:/configuration
      # Second, I mount a yaml file into the container, located in the directory we configured to be the provider directory from the command section
      - ./configuration/certificates.yml:/configuration/certificates.yml
    labels:
      - "traefik.enabled=true". # *** this is only needed if you want the dashboard proxied.  Going under that assumption, see the rest of this.
      - "traefik.http.routers.dashboard.rule=Host(`traefik.domain.tld`)"
      - "traefik.http.routers.dashboard.service=api@internal"
      - "traefik.http.routers.dashboard.entrypoints=https"
      - "traefik.http.routers.dashboard.tls=true"
      - "traefik.http.routers.dashboard.tls.certresolver=letsencrypt".  # *** this I figured out just today.  Left unconfigured, but defined, allows traefik to resolve to wildcard certs defined in certificates.yml but will fail if proxied containers don't match your wildcard domain name.  Also, only defining this for the dashboard will still allow all other containers to resolve your certificate without this line added to their configuration.   Again, seems odd.
      - "traefik.http.services.dashboard.loadbalancer.server.port=8080"
    networks:
      - traefik-default
  whoami:
    # A container that exposes an API to show its IP address
    image: containous/whoami
# *** Just a note, if you expose ports, it seems to break traefik for some reason.
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami.rule=Host(`whoami.wildcard`)" # *** is this actually your fully qualified host name?
      - "traefik.http.routers.whoami.entrypoints=https"
      - "traefik.http.routers.whoami.tls=true"
      - "traefik.http.routers.whoami.service=whoami". # *** add this
      - "traefik.http.services.whoami.loadbalancer.server.port=80". # *** add this (I'm assuming this is an http html response on port 80)
    networks:
      - traefik-default
networks:
  traefik-default:
    external: true

That's 9 lines added, to make sure you're not missing one. And just to emphasize, the certresolver I added to your traefik dashboard labels is only needed in one place and not needed in other container configurations. Also exposing the ports that you are also expecting to proxy causes problems.

I was working to set up rabbitmq and expose a tcp port 5672 for the app and an admin console on 15672 proxied through traefik. If the tcp port was exposed on its own, the admin port woudn't be proxied. If I didn't add the exposed port for rabbitmq, the admin console would work but no other applications could get to rabbitmq because the port wasn't exposed.

All of this seems a big failure of documentation on behalf of the Traefik team. And that certresolver and port exposure deals seem like bugs to me. And if no, really terrible user experience without explanation why we should go through the trouble.

I hope that clears things up for you.