Traefik default certificate conflicting with Let's Encrypt certificate

Traefik Traefik v3 (latest)
,
1

I'm trying to setup oidc authentication using the traefik-oidc-auth plugin. I've noticed a (in my opinion) weird behavior in which sometimes the let's encrypt certificate and sometimes the traefik default certificate is used when trying to access the identity provider (in my case a zitadel instance) for the oidc auth.

I want to secure multiple services using the oidc middleware, one of them being the traefik dashboard. When I use the following configuration for traefik (and the zitadel instance is already running) I get the following error:
http-get discovery endpoints - Err: Get "https://my.zitadel.host/.well-known/openid-configuration": tls: failed to verify certificate: x509: certificate is valid for e3a5fb4db832b2f90b90adecb99dd44d.9a559984c88710874bc7af965aabe8f4.traefik.default, not my.zitadel.host

Using the browser I can access the zitadel instance even when this error appears in the traefik logs and I was able to validate the certificate used by the browser to be the one issued by Let's Encrypt. The same was true when using curl from within the traefik container to use the zitadel api.

Weird things start happening when I remove the oidc stuff from the traefik container and add it to another http route. When I first start traefik, then zitadel and then the other service no error gets logged and the middleware works as expected for the other service.

For me it seems like the zitadel container has multiple certificates (one of them being the default traefik certificate and one of them being the let's encrypt one) and depending on the order in which the containers are started one or the other certificate is used.

This is my traefik service configuration:

services:
  proxy:
    image: traefik:v3.1.4
    restart: always
    command:
      # general
      - "--providers.docker=true"
      - "--providers.docker.network=proxy"
      - "--providers.docker.exposedbydefault=false"
      # entrypoints
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      # dashboard
      - "--api.insecure=true"
      - "--api.dashboard=true"
      # certificate resolvers
      - "--certificatesresolvers.letsencrypt.acme.storage=/acme/letsencrypt.json"
      - "--certificatesresolvers.letsencrypt.acme.tlschallenge=true"
      - "--certificatesresolvers.letsencrypt.acme.email=${LETSENCRYPT_EMAIL}"
      # plugins
      - "--experimental.plugins.traefik-oidc-auth.modulename=github.com/sevensolutions/traefik-oidc-auth"
      - "--experimental.plugins.traefik-oidc-auth.version=v0.2.0"
    labels:
      traefik.enable: true
      traefik.http.routers.traefik.entrypoints: websecure
      traefik.http.routers.traefik.rule: Host(`${DASHBOARD_HOST}`)
      traefik.http.routers.traefik.service: api@internal
      traefik.http.routers.traefik.tls.certresolver: letsencrypt
      traefik.http.routers.traefik.middlewares: oidc-auth
      traefik.http.services.traefik.loadbalancer.server.port: 8080
      traefik.http.middlewares.oidc-auth.plugin.traefik-oidc-auth.Provider.Url: ${AUTH_URL}
      traefik.http.middlewares.oidc-auth.plugin.traefik-oidc-auth.Provider.ClientId: ${AUTH_CLIENT_ID}
      traefik.http.middlewares.oidc-auth.plugin.traefik-oidc-auth.Provider.ClientSecret: ${AUTH_CLIENT_SECRET}
    ports:
      # listen on port 80
      - "80:80"
      # listen on port 443
      - "443:443"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      # location where all acme certificates are stored
      - "acme:/acme"
    networks:
      - proxy

volumes:
  acme:

networks:
  proxy:
    name: "proxy"

and this is my zitadel service configuration from which I think only the traefik labels should be of interest:

services:
  zitadel:
    restart: always
    image: ghcr.io/zitadel/zitadel:latest
    command: 'start-from-init --steps /zitadel-init-steps.yaml --masterkey "${ZITADEL_MASTERKEY}" --tlsMode external'
    environment:
      ZITADEL_EXTERNALDOMAIN: ${ZITADEL_HOST}
      ZITADEL_EXTERNALSECURE: true
      ZITADEL_EXTERNALPORT: 443
      ZITADEL_TLS_ENABLED: false
      ZITADEL_DATABASE_POSTGRES_HOST: database
      ZITADEL_DATABASE_POSTGRES_PORT: 5432
      ZITADEL_DATABASE_POSTGRES_DATABASE: zitadel-database
      ZITADEL_DATABASE_POSTGRES_USER_USERNAME: zitadel-database-user
      ZITADEL_DATABASE_POSTGRES_USER_PASSWORD: ${ZITADEL_USER_PASSWORD}
      ZITADEL_DATABASE_POSTGRES_USER_SSL_MODE: disable
      ZITADEL_DATABASE_POSTGRES_ADMIN_USERNAME: admin-username
      ZITADEL_DATABASE_POSTGRES_ADMIN_PASSWORD: ${ZITADEL_ADMIN_PASSWORD}
      ZITADEL_DATABASE_POSTGRES_ADMIN_SSL_MODE: disable
    depends_on:
      database:
        condition: service_healthy
    labels:
      traefik.enable: true
      traefik.http.routers.zitadel.entrypoints: websecure
      traefik.http.routers.zitadel.service: zitadel
      traefik.http.routers.zitadel.rule: Host(`${ZITADEL_HOST}`)
      traefik.http.routers.zitadel.tls.certresolver: letsencrypt
      traefik.http.services.zitadel.loadbalancer.server.port: 8080
    volumes:
      - ./zitadel-init.yaml:/zitadel-init-steps.yaml:ro
    networks:
      - zitadel
      - proxy

Has someone experienced similar behavior or does know the exact thing which is causing this issue? A solution would be to make a PR for the plugin to create an option to add insecureSkipVerify to the requests made from the plugin but I think from a security point of view it would be a bad move to enable this in a production setup.

Thanks for any help in advance :slight_smile: