Static conf serversTransport.rootCAs not being honored

Hey, I'm trying to configure TLS on my backends by following this guide:

I have modified the static conf /etc/traefik/traefik.yml with the following, and yes, the traefik user can read the file. Logs don't show anything interesting.

serversTransport:
  rootCAs:
    - /etc/traefik/certs/self-signed-ca.pem

I keep getting 500 internal server error on all my https backends. http ones work regularly

It seems that the rootCAs rule is not honored for some reason, because when setting:

serversTransport:
  insecureSkipVerify: true

Everything works as it should, except the above flag is not production approved.

Any thoughts? This seems like a bug...

Did you enable Traefik debug log and access log?

Are your http and https backends the same service, only with different protocol? Can you ensure the 500 error is coming from Traefik and not from your service?

Didn't enable the debug log, but the access log showed the 500 error code.

Are your http and https backends the same service, only with different protocol? Can you ensure the 500 error is coming from Traefik and not from your service?

I'm using a single service for my backend, I just changed the url: http -> https. The 500 was coming from traefik 100%, I can curl the service directly.

Update: Nothing also in the debug log. Traefik keeps returning 500.
Did you manage to get this working? I'm trying to find out if it's a bug or if it's related to my config...

I don't think you can get an error 500 from Traefik itself without something in the Traefik debug log. Have you checked the logs of your service container, maybe the 500 is coming from there? Maybe enable Traefik JSON-formatted access logs, that has more infos.

Nothing is even reaching the backend service, the request flops when it reaches traefik.

Can you share your full Traefik static and dynamic config, and docker-compose.yml if used?

config.yml:

tls:
  options:
    default:
      sniStrict: true
      minVersion: VersionTLS12
      cipherSuites:
        - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

http:
  routers:
    redacted:
      rule: Host(`sub.domain.tld`)
      entryPoints:
        - web-secure
      middlewares:
        - security-headers
      service: service1
      tls:
        certResolver: le
        domains:
          - main: "sub.domain.tld"

  services:
    service1:
      loadBalancer:
        servers:
          - url: https://backend-host:443

  middlewares:
    security-headers:
      headers:
        referrerPolicy: same-origin
        forceSTSHeader: true
        stsSeconds: 31536000
        stsIncludeSubdomains: true
        stsPreload: true
        contentTypeNosniff: true
        browserXssFilter: true
        customRequestHeaders:
          X-Forwarded-Proto: https
        frameDeny: true

traefik.yml

providers:
  file:
    filename: /etc/traefik/config.yml
    watch: true

serversTransport:
  rootCAs:
    - /etc/traefik/certs/self-signedCA.pem # also in system trust store and readable by traefik user
  insecureSkipVerify: false

entryPoints:
  web:
    address: :80
    http:
      redirections:
        entryPoint:
          to: web-secure
          scheme: https
          permanent: true

  web-secure:
    address: :443

certificatesResolvers:
  le:
...
redacted
...

Desktop->Traefik VM
image

Traefik VM->backend:

$ nc -vz backend-https-host 8443
backend-https-host [ip redacted] 8443 (?) open

To recap, when insecureSkipVerify is set to True, I can access the page normally - though as I said, that's not really a solution, more of a workaround.