ROCKET_TLS server as backend: BadCertificate

DaCHack · October 30, 2024, 9:18am

What did you do?

Set up traefik as reverse proxy to vaultwarden container (using a ROCKET_TLS server internally listening on port 80 but expecting HTTPS)
Set up vaultwarden to use the internal SSL service with a letsencrypt certificate
Configured traefik to use an own letsencrypt certificate for the frontend and forward to port 80 of vaultwarden with - "traefik.http.services.vaultwarden.loadbalancer.server.scheme=https"

Expected to see vaultwarden functional when using https://internal.domain.com.

What did you see instead?

BadCertificate error on service-side (vaultwarden logs):
[rocket_http::tls::listener][WARN] tls handshake with X.X.X.X:59756 failed: received fatal alert: BadCertificate

Seems to be an issue with how traefik handles vaultwarden's certificate as other clients work fine when directly connecting to vaultwardens port 80 via the docker network.

"--serverstransport.rootcas=/vault-ca.crt"
does not seem to help even though with this file including the rootCA and certificate of the service expectation would be that the server's certificate is accepted without further checking (which is not possible since traefik addresses the service via its IP and not its domain.

What version of Traefik are you using?

3.1.6

What is your environment & configuration?

services:

  traefik:
    image: "traefik:v3.1"
    container_name: "traefik"
    command:
      #- "--log.level=DEBUG"
      #- "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entryPoints.web.address=:80"
      - "--entryPoints.websecure.address=:443"
      - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
      - "--certificatesresolvers.letsencrypt.acme.dnschallenge=true"
      - "--certificatesresolvers.letsencrypt.acme.dnschallenge.provider=XXXX"
      - "--certificatesresolvers.letsencrypt.acme.email=XXXX"
      - "--certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json"
      - "--serverstransport.rootcas=/vault-ca.crt"
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    environment:
      - "XXXX_API_KEY=XXXX"
    volumes:
      - "/opt/appdata/traefik/letsencrypt:/letsencrypt"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "/opt/appdata/acme-lego-cron/certificates/test.test.com.crt:/vault-ca.crt"
    networks:
      - default
      - vaultwarden_default

networks:
  vaultwarden_default:
    external: true

services:

  vaultwarden:
    image: vaultwarden/server:latest
    container_name: "vaultwarden"
    restart: unless-stopped
    environment:
      ROCKET_TLS: '{certs="/ssl/XXX.pem",key="/ssl/XXX.key"}'
#TODO: Replace with credentials file
      ADMIN_TOKEN: 'XXX'
    volumes:
      - /opt/appdata/acme-lego-cron/certificates:/ssl/
      - /opt/appdata/vaultwarden:/data/
    ports:
      - 8001:80/tcp
    healthcheck:
      disable: true
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.vaultwarden.rule=Host(`test.test.com`)"
      - "traefik.http.routers.vaultwarden.entrypoints=websecure"
      - "traefik.http.routers.vaultwarden.tls=true"
      - "traefik.http.routers.vaultwarden.tls.certresolver=letsencrypt"
      - "traefik.http.services.vaultwarden.loadbalancer.server.port=80"
      - "traefik.http.services.vaultwarden.loadbalancer.server.scheme=https"

If applicable, please paste the log output in DEBUG level

2024-10-25T09:14:11Z DBG github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196 > Service selected by WRR: 7793ba43b45bb1b0
2024-10-25T09:14:11Z DBG github.com/traefik/traefik/v3/pkg/server/service/proxy.go:100 > 500 Internal Server Error error="tls: failed to verify certificate: x509: cannot validate certificate for 172.20.0.2 because it doesn't contain any IP SANs"

bluepuma77 · October 30, 2024, 2:14pm

Error message tells that it can’t validate the internal TLS cert because it’s accessing it with IP, but that is not represented in the cert.

Usually access is done via (sub-)domain and that is included in the cert, needs to match.

You probably need to add insecureSkipVerify to your serversTransport to work around this.

DaCHack · October 30, 2024, 3:44pm

Can I apply this option only to this single service (e.g. in the service's docker -compose) or does it need to be configured globally in the traefik docker-compose?
If yes, how does the label line look like in this case?

And alternatively: Why isn't it possible to just add the service's certificate to the rootCAs so that it is always accepted but still if someone tinkered with the certificate traefik would notice and not forward the request - as opposed to the insecureSkipVerify-solution.

bluepuma77 · October 30, 2024, 5:25pm

Yes, you can create a serversTransport in dynamic config and assign it to the service (doc).

The error indicates that the target is verified against the cert. Traefik will connect via IP and the IP is not in the cert. Usually you would connect via domain and it’s included in the cert.

DaCHack · November 14, 2024, 7:25am

Thanks, I tried in the vaultwarden compose:

      - "traefik.http.services.vaultwarden.loadbalancer.serverstransport=mytransport"
      - "traefik.http.serverstransports.mytransport.insecureskipverify=true"

But it gives:
2024-11-14T07:14:09Z ERR error="servers transport not found mytransport@docker" entryPointName=websecure routerName=vaultwarden@docker

Is this issue still up-to-date? If yes, how would that work in my scenario then? Never configured serversTransport nor used a file provider.
insecureSkipVerify How to apply this at a service level in docker compose - Traefik / Traefik v2 - Traefik Labs Community Forum

bluepuma77 · November 14, 2024, 8:17am

Define it according to doc:

## Dynamic configuration
http:
  serversTransports:
    mytransport:
      insecureSkipVerify: true

Assign it according to doc:

## Dynamic configuration
http:
  services:
    Service01:
      loadBalancer:
        serversTransport: mytransport

Just checked, according to the dynamic config for Docker reference (doc), it can only be assigned in labels:

  - "traefik.http.services.service02.loadbalancer.serverstransport=foobar"

But it still needs to be created in a dynamic config file, which is loaded via providers.file in static config.

Really annoying, that some simple options are missing from Docker provider.

DaCHack · November 14, 2024, 2:11pm

Thanks. This is really not that intuitive. Still does not work on my side. Traefik gives me in the log:
2024-11-14T14:01:26Z ERR error="servers transport not found SSLtransport@docker" entryPointName=websecure routerName=vaultwarden@docker

Traefik docker-compose:

services:

  traefik:
    image: "traefik:v3.1"
    container_name: "traefik"
    restart: unless-stopped
    command:
      #- "--log.level=DEBUG"
      #- "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--providers.file.directory=/config"
      - "--providers.file.watch=false"
      - "--entryPoints.web.address=:80"
      - "--entryPoints.websecure.address=:443"
      - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
      - "--certificatesresolvers.letsencrypt.acme.dnschallenge=true"
      #- "--certificatesresolvers.letsencrypt.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
      - "--certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json"
...

Vaultwarden docker-compose:

services:

  vaultwarden:
    image: vaultwarden/server:latest
    container_name: "vaultwarden"
...
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.vaultwarden.rule=Host(`test.test.com`)"
      - "traefik.http.routers.vaultwarden.entrypoints=websecure"
      - "traefik.http.routers.vaultwarden.tls=true"
      - "traefik.http.routers.vaultwarden.tls.certresolver=letsencrypt"
      - "traefik.http.services.vaultwarden.loadbalancer.server.port=80"
      - "traefik.http.services.vaultwarden.loadbalancer.server.scheme=https"
      - "traefik.http.services.vaultwarden.loadbalancer.serverstransport=SSLtransport"

/config/dynamic_conf.yml:

## Dynamic configuration
http:
  serversTransports:
    SSLtransport:
      insecureSkipVerify: true

  services:
    vaultwarden:
      loadBalancer:
        serversTransport: SSLtransport

Traefik seems to successfully read dynamic_conf.yml based on the DEBUG outputs.
What am I doing wrong?

bluepuma77 · November 14, 2024, 5:30pm

Try serverstransport=SSLtransport@file.

DaCHack · November 14, 2024, 7:42pm

Great! That works.
Thanks so much @bluepuma77 !!!

Topic		Replies	Views
[solved] Remote error: tls: bad certificate" with traefik 2.3.5 Traefik v2 docker , letsencrypt-acme	0	4453	December 15, 2020
Traefik SSL Let's Encrypt invalid certificate Traefik v2 docker , letsencrypt-acme	2	868	October 4, 2022
Issues running Nginx reverse proxy that handles SSL BEHIND Traefik Traefik v2 docker-swarm	1	1883	November 6, 2022
Proxying entire subdomain to external server: troubles with TLS Traefik v2 docker , letsencrypt-acme	5	2529	August 9, 2022
Traefik and LetsEncrypt without Internet for Local Vaultwarden Application? Traefik v2 docker	12	4020	April 17, 2024