mTLS between Traefik and an upstream service (with ServersTransport.Certificates) does not work

Starrah · December 28, 2021, 9:22am

Hello there, I tried to configure an zero-trust mTLS connection between traefik and a upstream service. Traefik v2.5.6 is installed with Helm Chart onto k8s. I tried to use ServerTransport configuration, but the upstream service (written in Node.js) didn't receive the expected TLS client certificate, which should have been sent by traefik.

My config is as the following (with personal information replaced):

apiVersion: v1
kind: Service
metadata:
  name: sso-dev
  annotations:
    traefik.ingress.kubernetes.io/service.serversscheme: "https"
    traefik.ingress.kubernetes.io/service.serverstransport: "auth-example-sso@kubernetescrd"
spec:
  clusterIP: None # headless service
  ports:
    - protocol: TCP
      port: 3000
      targetPort: 1357
---
apiVersion: v1
kind: Endpoints
metadata:
  name: sso-dev
subsets:
- addresses:
  - ip: 10.0.83.100
  ports:
  - port: 1357
---
# mTLS from traefik to backend
apiVersion: traefik.containo.us/v1alpha1
kind: ServersTransport
metadata:
  name: example-sso
spec:
  serverName: "example-sso"
  rootCAsSecrets:
  - example-sso-ca
  certificatesSecrets:
  - example-sso
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: sso-dev
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: websecure,web20080
spec:
  rules:
    - host: sso.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: sso-dev
                port:
                  number: 3000

Thank you!

Starrah · December 28, 2021, 9:23am

Part of the DEBUG log of traefik is as the following:

time="2021-12-27T18:07:19Z" level=info msg="Configuration loaded from flags."
time="2021-12-27T18:07:19Z" level=info msg="Traefik version 2.5.6 built on 2021-12-22T16:30:52Z"
time="2021-12-27T18:07:19Z" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true,\"sendAnonymousUsage\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"metrics\":{\"address\":\":9100/tcp\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"udp\":{\"timeout\":\"3s\"}},\"traefik\":{\"address\":\":9000/tcp\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"udp\":{\"timeout\":\"3s\"}},\"web\":{\"address\":\":8000/tcp\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{\"redirections\":{\"entryPoint\":{\"to\":\":443\",\"scheme\":\"https\",\"permanent\":true,\"priority\":2147483646}}},\"udp\":{\"timeout\":\"3s\"}},\"web20080\":{\"address\":\":20080/tcp\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{\"tls\":{\"certResolver\":\"example-com-resolver\"}},\"udp\":{\"timeout\":\"3s\"}},\"websecure\":{\"address\":\":8443/tcp\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{\"tls\":{\"certResolver\":\"example-com-resolver\"}},\"udp\":{\"timeout\":\"3s\"}}},\"providers\":{\"providersThrottleDuration\":\"2s\",\"kubernetesIngress\":{},\"kubernetesCRD\":{}},\"api\":{\"dashboard\":true},\"metrics\":{\"prometheus\":{\"buckets\":[0.1,0.3,1.2,5],\"addEntryPointsLabels\":true,\"addServicesLabels\":true,\"entryPoint\":\"metrics\"}},\"ping\":{\"entryPoint\":\"traefik\",\"terminatingStatusCode\":503},\"log\":{\"level\":\"DEBUG\",\"format\":\"common\"},\"accessLog\":{\"format\":\"common\",\"filters\":{},\"fields\":{\"defaultMode\":\"keep\",\"headers\":{\"defaultMode\":\"drop\"}}},\"certificatesResolvers\":{\"example-com-resolver\":{\"acme\":{\"email\":\"starrah@foxmail.com\",\"caServer\":\"https://acme-v02.api.letsencrypt.org/directory\",\"storage\":\"/acme/acme.json\",\"keyType\":\"RSA4096\",\"dnsChallenge\":{\"provider\":\"dnspod\"}}}},\"pilot\":{\"dashboard\":true}}"
......
time="2021-12-27T18:07:19Z" level=info msg="Starting provider *ingress.Provider {}"
time="2021-12-27T18:07:19Z" level=info msg="ingress label selector is: \"\"" providerName=kubernetes
time="2021-12-27T18:07:19Z" level=info msg="Creating in-cluster Provider client" providerName=kubernetes
time="2021-12-27T18:07:19Z" level=info msg="Starting provider *traefik.Provider {}"
time="2021-12-27T18:07:19Z" level=info msg="Starting provider *crd.Provider {}"
time="2021-12-27T18:07:19Z" level=info msg="Starting provider *acme.ChallengeTLSALPN {\"Timeout\":4000000000}"
time="2021-12-27T18:07:19Z" level=info msg="label selector is: \"\"" providerName=kubernetescrd
......
time="2021-12-27T18:07:19Z" level=debug msg="Configuration received from provider kubernetes: {\"http\":{\"routers\":{\"sso-dev-auth-sso-example-com\":{\"entryPoints\":[\"websecure\",\"web20080\"],\"service\":\"auth-sso-dev-3000\",\"rule\":\"Host(`sso.example.com`) \\u0026\\u0026 PathPrefix(`/`)\"},\"sso-dev-login-clienttls-auth-mtls-sso-example-com-login\":{\"entryPoints\":[\"websecure\",\"web20080\"],\"middlewares\":[\"auth-example-mtls@kubernetescrd\"],\"service\":\"auth-sso-dev-3000\",\"rule\":\"Host(`mtls.sso.example.com`) \\u0026\\u0026 Path(`/login`)\",\"tls\":{\"options\":\"auth-example-mtls@kubernetescrd\"}}},\"services\":{\"auth-sso-dev-3000\":{\"loadBalancer\":{\"servers\":[{\"url\":\"https://10.0.83.100:1357\"}],\"passHostHeader\":true,\"serversTransport\":\"auth-example-sso@kubernetescrd\"}}}},\"tcp\":{}}" providerName=kubernetes
......
time="2021-12-27T18:07:19Z" level=debug msg="No default certificate, generating one" tlsStoreName=default
time="2021-12-27T18:07:19Z" level=debug msg="Adding certificate for domain(s) test2.example.com"
time="2021-12-27T18:07:19Z" level=debug msg="Adding certificate for domain(s) sso.example.com"
time="2021-12-27T18:07:19Z" level=debug msg="Adding certificate for domain(s) mtls.sso.example.com"
time="2021-12-27T18:07:19Z" level=debug msg="Added outgoing tracing middleware ping@internal" middlewareName=tracing middlewareType=TracingForwarder routerName=ping@internal entryPointName=traefik
time="2021-12-27T18:07:19Z" level=debug msg="Added outgoing tracing middleware api@internal" middlewareName=tracing middlewareType=TracingForwarder entryPointName=traefik routerName=traefik-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd
time="2021-12-27T18:07:19Z" level=debug msg="Creating middleware" entryPointName=traefik middlewareType=Recovery middlewareName=traefik-internal-recovery
time="2021-12-27T18:07:19Z" level=debug msg="Added outgoing tracing middleware prometheus@internal" middlewareType=TracingForwarder middlewareName=tracing entryPointName=metrics routerName=prometheus@internal
time="2021-12-27T18:07:19Z" level=debug msg="Creating middleware" entryPointName=metrics middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2021-12-27T18:07:19Z" level=debug msg="Added outgoing tracing middleware noop@internal" middlewareType=TracingForwarder entryPointName=web routerName=web-to-443@internal middlewareName=tracing
time="2021-12-27T18:07:19Z" level=debug msg="Creating middleware" entryPointName=web middlewareName=redirect-web-to-443@internal middlewareType=RedirectScheme routerName=web-to-443@internal
time="2021-12-27T18:07:19Z" level=debug msg="Setting up redirection to https 443" routerName=web-to-443@internal entryPointName=web middlewareName=redirect-web-to-443@internal middlewareType=RedirectScheme
time="2021-12-27T18:07:19Z" level=debug msg="Adding tracing to middleware" entryPointName=web routerName=web-to-443@internal middlewareName=redirect-web-to-443@internal
time="2021-12-27T18:07:19Z" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=web
time="2021-12-27T18:07:19Z" level=debug msg="Creating middleware" middlewareType=Metrics entryPointName=metrics middlewareName=metrics-entrypoint
time="2021-12-27T18:07:19Z" level=debug msg="Creating middleware" entryPointName=traefik middlewareType=Metrics middlewareName=metrics-entrypoint
time="2021-12-27T18:07:19Z" level=debug msg="Creating middleware" middlewareType=Metrics entryPointName=web middlewareName=metrics-entrypoint
time="2021-12-27T18:07:19Z" level=debug msg="Creating middleware" middlewareName=metrics-entrypoint middlewareType=Metrics entryPointName=web20080
time="2021-12-27T18:07:19Z" level=debug msg="Creating middleware" middlewareName=metrics-entrypoint middlewareType=Metrics entryPointName=websecure
time="2021-12-27T18:07:19Z" level=debug msg="Creating middleware" middlewareName=metrics-entrypoint middlewareType=Metrics entryPointName=metrics
time="2021-12-27T18:07:19Z" level=debug msg="Creating middleware" entryPointName=traefik middlewareName=metrics-entrypoint middlewareType=Metrics
time="2021-12-27T18:07:19Z" level=debug msg="Creating middleware" middlewareName=metrics-entrypoint middlewareType=Metrics entryPointName=web
time="2021-12-27T18:07:19Z" level=debug msg="Creating middleware" middlewareType=Metrics entryPointName=web20080 middlewareName=metrics-entrypoint
time="2021-12-27T18:07:19Z" level=debug msg="Creating middleware" entryPointName=websecure middlewareName=metrics-entrypoint middlewareType=Metrics
time="2021-12-27T18:07:19Z" level=debug msg="No default certificate, generating one" tlsStoreName=default
time="2021-12-27T18:07:20Z" level=debug msg="Adding certificate for domain(s) test2.example.com"
time="2021-12-27T18:07:20Z" level=debug msg="Adding certificate for domain(s) sso.example.com"
time="2021-12-27T18:07:20Z" level=debug msg="Adding certificate for domain(s) mtls.sso.example.com"
time="2021-12-27T18:07:20Z" level=debug msg="Added outgoing tracing middleware api@internal" entryPointName=traefik routerName=traefik-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd middlewareName=tracing middlewareType=TracingForwarder
time="2021-12-27T18:07:20Z" level=debug msg="Added outgoing tracing middleware ping@internal" entryPointName=traefik routerName=ping@internal middlewareName=tracing middlewareType=TracingForwarder
time="2021-12-27T18:07:20Z" level=debug msg="Creating middleware" entryPointName=traefik middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2021-12-27T18:07:20Z" level=debug msg="Added outgoing tracing middleware prometheus@internal" entryPointName=metrics routerName=prometheus@internal middlewareName=tracing middlewareType=TracingForwarder
time="2021-12-27T18:07:20Z" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=metrics
time="2021-12-27T18:07:20Z" level=debug msg="Added outgoing tracing middleware noop@internal" middlewareName=tracing entryPointName=web routerName=web-to-443@internal middlewareType=TracingForwarder
time="2021-12-27T18:07:20Z" level=debug msg="Creating middleware" middlewareName=redirect-web-to-443@internal middlewareType=RedirectScheme routerName=web-to-443@internal entryPointName=web
time="2021-12-27T18:07:20Z" level=debug msg="Setting up redirection to https 443" entryPointName=web middlewareName=redirect-web-to-443@internal middlewareType=RedirectScheme routerName=web-to-443@internal
time="2021-12-27T18:07:20Z" level=debug msg="Adding tracing to middleware" middlewareName=redirect-web-to-443@internal entryPointName=web routerName=web-to-443@internal
time="2021-12-27T18:07:20Z" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=web
time="2021-12-27T18:07:20Z" level=debug msg="Creating middleware" middlewareName=metrics-entrypoint middlewareType=Metrics entryPointName=metrics
time="2021-12-27T18:07:20Z" level=debug msg="Creating middleware" entryPointName=traefik middlewareType=Metrics middlewareName=metrics-entrypoint
time="2021-12-27T18:07:20Z" level=debug msg="Creating middleware" entryPointName=web middlewareName=metrics-entrypoint middlewareType=Metrics
time="2021-12-27T18:07:20Z" level=debug msg="Creating middleware" entryPointName=web20080 middlewareName=metrics-entrypoint middlewareType=Metrics
time="2021-12-27T18:07:20Z" level=debug msg="Creating middleware" entryPointName=websecure middlewareName=metrics-entrypoint middlewareType=Metrics
time="2021-12-27T18:07:20Z" level=debug msg="Creating middleware" middlewareType=Pipelining entryPointName=web20080 routerName=web20080-sso-dev-auth-sso-example-com@kubernetes serviceName=auth-sso-dev-3000 middlewareName=pipelining
time="2021-12-27T18:07:20Z" level=debug msg="Creating middleware" entryPointName=web20080 routerName=web20080-sso-dev-auth-sso-example-com@kubernetes middlewareName=metrics-service middlewareType=Metrics serviceName=auth-sso-dev-3000
time="2021-12-27T18:07:20Z" level=debug msg="Creating load-balancer" entryPointName=web20080 routerName=web20080-sso-dev-auth-sso-example-com@kubernetes serviceName=auth-sso-dev-3000
time="2021-12-27T18:07:20Z" level=debug msg="Creating server 0 https://10.0.83.100:1357" entryPointName=web20080 routerName=web20080-sso-dev-auth-sso-example-com@kubernetes serviceName=auth-sso-dev-3000 serverName=0
time="2021-12-27T18:07:20Z" level=debug msg="child https://10.0.83.100:1357 now UP"
time="2021-12-27T18:07:20Z" level=debug msg="Propagating new UP status"
time="2021-12-27T18:07:20Z" level=debug msg="Added outgoing tracing middleware auth-sso-dev-3000" middlewareType=TracingForwarder entryPointName=web20080 routerName=web20080-sso-dev-auth-sso-example-com@kubernetes middlewareName=tracing
time="2021-12-27T18:07:20Z" level=debug msg="Creating middleware" routerName=web20080-sso-dev-login-clienttls-auth-mtls-sso-example-com-login@kubernetes serviceName=auth-sso-dev-3000 middlewareType=Pipelining middlewareName=pipelining entryPointName=web20080
time="2021-12-27T18:07:20Z" level=debug msg="Creating middleware" middlewareName=metrics-service middlewareType=Metrics routerName=web20080-sso-dev-login-clienttls-auth-mtls-sso-example-com-login@kubernetes serviceName=auth-sso-dev-3000 entryPointName=web20080
time="2021-12-27T18:07:20Z" level=debug msg="Creating load-balancer" entryPointName=web20080 routerName=web20080-sso-dev-login-clienttls-auth-mtls-sso-example-com-login@kubernetes serviceName=auth-sso-dev-3000
time="2021-12-27T18:07:20Z" level=debug msg="Creating server 0 https://10.0.83.100:1357" entryPointName=web20080 serverName=0 routerName=web20080-sso-dev-login-clienttls-auth-mtls-sso-example-com-login@kubernetes serviceName=auth-sso-dev-3000
time="2021-12-27T18:07:20Z" level=debug msg="child https://10.0.83.100:1357 now UP"
time="2021-12-27T18:07:20Z" level=debug msg="Propagating new UP status"
time="2021-12-27T18:07:20Z" level=debug msg="Added outgoing tracing middleware auth-sso-dev-3000" entryPointName=web20080 middlewareName=tracing middlewareType=TracingForwarder routerName=web20080-sso-dev-login-clienttls-auth-mtls-sso-example-com-login@kubernetes
time="2021-12-27T18:07:20Z" level=debug msg="Creating middleware" entryPointName=web20080 routerName=web20080-sso-dev-login-clienttls-auth-mtls-sso-example-com-login@kubernetes middlewareName=auth-example-mtls@kubernetescrd middlewareType=PassClientTLSCert
time="2021-12-27T18:07:20Z" level=debug msg="Adding tracing to middleware" entryPointName=web20080 routerName=web20080-sso-dev-login-clienttls-auth-mtls-sso-example-com-login@kubernetes middlewareName=auth-example-mtls@kubernetescrd
time="2021-12-27T18:07:20Z" level=debug msg="Creating middleware" entryPointName=web20080 middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2021-12-27T18:07:20Z" level=debug msg="Creating middleware" serviceName=auth-sso-dev-3000 entryPointName=websecure routerName=websecure-sso-dev-auth-sso-example-com@kubernetes middlewareName=pipelining middlewareType=Pipelining
time="2021-12-27T18:07:20Z" level=debug msg="Creating middleware" entryPointName=websecure routerName=websecure-sso-dev-auth-sso-example-com@kubernetes serviceName=auth-sso-dev-3000 middlewareName=metrics-service middlewareType=Metrics
time="2021-12-27T18:07:20Z" level=debug msg="Creating load-balancer" serviceName=auth-sso-dev-3000 entryPointName=websecure routerName=websecure-sso-dev-auth-sso-example-com@kubernetes
time="2021-12-27T18:07:20Z" level=debug msg="Creating server 0 https://10.0.83.100:1357" serviceName=auth-sso-dev-3000 serverName=0 entryPointName=websecure routerName=websecure-sso-dev-auth-sso-example-com@kubernetes
time="2021-12-27T18:07:20Z" level=debug msg="child https://10.0.83.100:1357 now UP"
time="2021-12-27T18:07:20Z" level=debug msg="Propagating new UP status"
time="2021-12-27T18:07:20Z" level=debug msg="Added outgoing tracing middleware auth-sso-dev-3000" entryPointName=websecure routerName=websecure-sso-dev-auth-sso-example-com@kubernetes middlewareName=tracing middlewareType=TracingForwarder
time="2021-12-27T18:07:20Z" level=debug msg="Creating middleware" middlewareType=Pipelining entryPointName=websecure routerName=websecure-sso-dev-login-clienttls-auth-mtls-sso-example-com-login@kubernetes serviceName=auth-sso-dev-3000 middlewareName=pipelining
time="2021-12-27T18:07:20Z" level=debug msg="Creating middleware" middlewareName=metrics-service middlewareType=Metrics entryPointName=websecure routerName=websecure-sso-dev-login-clienttls-auth-mtls-sso-example-com-login@kubernetes serviceName=auth-sso-dev-3000
time="2021-12-27T18:07:20Z" level=debug msg="Creating load-balancer" entryPointName=websecure routerName=websecure-sso-dev-login-clienttls-auth-mtls-sso-example-com-login@kubernetes serviceName=auth-sso-dev-3000
time="2021-12-27T18:07:20Z" level=debug msg="Creating server 0 https://10.0.83.100:1357" routerName=websecure-sso-dev-login-clienttls-auth-mtls-sso-example-com-login@kubernetes serviceName=auth-sso-dev-3000 serverName=0 entryPointName=websecure
time="2021-12-27T18:07:20Z" level=debug msg="child https://10.0.83.100:1357 now UP"
time="2021-12-27T18:07:20Z" level=debug msg="Propagating new UP status"
time="2021-12-27T18:07:20Z" level=debug msg="Added outgoing tracing middleware auth-sso-dev-3000" routerName=websecure-sso-dev-login-clienttls-auth-mtls-sso-example-com-login@kubernetes middlewareType=TracingForwarder middlewareName=tracing entryPointName=websecure
time="2021-12-27T18:07:20Z" level=debug msg="Creating middleware" routerName=websecure-sso-dev-login-clienttls-auth-mtls-sso-example-com-login@kubernetes middlewareName=auth-example-mtls@kubernetescrd middlewareType=PassClientTLSCert entryPointName=websecure
time="2021-12-27T18:07:20Z" level=debug msg="Adding tracing to middleware" entryPointName=websecure routerName=websecure-sso-dev-login-clienttls-auth-mtls-sso-example-com-login@kubernetes middlewareName=auth-example-mtls@kubernetescrd
time="2021-12-27T18:07:20Z" level=debug msg="Creating middleware" entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2021-12-27T18:07:20Z" level=debug msg="Creating middleware" entryPointName=metrics middlewareName=metrics-entrypoint middlewareType=Metrics
time="2021-12-27T18:07:20Z" level=debug msg="Creating middleware" middlewareType=Metrics entryPointName=traefik middlewareName=metrics-entrypoint
time="2021-12-27T18:07:20Z" level=debug msg="Creating middleware" entryPointName=web middlewareName=metrics-entrypoint middlewareType=Metrics
time="2021-12-27T18:07:20Z" level=debug msg="Creating middleware" middlewareType=Metrics entryPointName=web20080 middlewareName=metrics-entrypoint
time="2021-12-27T18:07:20Z" level=debug msg="Creating middleware" middlewareName=metrics-entrypoint middlewareType=Metrics entryPointName=websecure
time="2021-12-27T18:07:20Z" level=debug msg="Adding route for mtls.sso.example.com with TLS options auth-example-mtls@kubernetescrd" entryPointName=web20080
time="2021-12-27T18:07:20Z" level=debug msg="Adding route for sso.example.com with TLS options default" entryPointName=web20080
time="2021-12-27T18:07:20Z" level=debug msg="Adding route for mtls.sso.example.com with TLS options auth-example-mtls@kubernetescrd" entryPointName=websecure
time="2021-12-27T18:07:20Z" level=debug msg="Adding route for sso.example.com with TLS options default" entryPointName=websecure
time="2021-12-27T18:07:20Z" level=debug msg="Try to challenge certificate for domain [sso.example.com] found in HostSNI rule" providerName=example-com-resolver.acme rule="Host(`sso.example.com`) && PathPrefix(`/`)" routerName=web20080-sso-dev-auth-sso-example-com@kubernetes
time="2021-12-27T18:07:20Z" level=debug msg="Try to challenge certificate for domain [sso.example.com] found in HostSNI rule" providerName=example-com-resolver.acme routerName=websecure-sso-dev-auth-sso-example-com@kubernetes rule="Host(`sso.example.com`) && PathPrefix(`/`)"
time="2021-12-27T18:07:20Z" level=debug msg="Looking for provided certificate(s) to validate [\"sso.example.com\"]..." rule="Host(`sso.example.com`) && PathPrefix(`/`)" providerName=example-com-resolver.acme routerName=websecure-sso-dev-auth-sso-example-com@kubernetes
time="2021-12-27T18:07:20Z" level=debug msg="No ACME certificate generation required for domains [\"sso.example.com\"]." providerName=example-com-resolver.acme routerName=websecure-sso-dev-auth-sso-example-com@kubernetes rule="Host(`sso.example.com`) && PathPrefix(`/`)"
time="2021-12-27T18:07:20Z" level=debug msg="Looking for provided certificate(s) to validate [\"sso.example.com\"]..." rule="Host(`sso.example.com`) && PathPrefix(`/`)" routerName=web20080-sso-dev-auth-sso-example-com@kubernetes providerName=example-com-resolver.acme
time="2021-12-27T18:07:20Z" level=debug msg="No ACME certificate generation required for domains [\"sso.example.com\"]." routerName=web20080-sso-dev-auth-sso-example-com@kubernetes providerName=example-com-resolver.acme rule="Host(`sso.example.com`) && PathPrefix(`/`)"
time="2021-12-27T18:07:33Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetes
time="2021-12-27T18:07:33Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2021-12-27T18:07:35Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/login\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"zh-CN,zh;q=0.9\"],\"Cache-Control\":[\"max-age=0\"],\"If-None-Match\":[\"W/\\\"1604-/iwgqDSVYEIOWN61rDOCdJPepvE\\\"\"],\"Sec-Ch-Ua\":[\"\\\" Not A;Brand\\\";v=\\\"99\\\", \\\"Chromium\\\";v=\\\"96\\\", \\\"Google Chrome\\\";v=\\\"96\\\"\"],\"Sec-Ch-Ua-Mobile\":[\"?0\"],\"Sec-Ch-Ua-Platform\":[\"\\\"Linux\\\"\"],\"Sec-Fetch-Dest\":[\"document\"],\"Sec-Fetch-Mode\":[\"navigate\"],\"Sec-Fetch-Site\":[\"none\"],\"Sec-Fetch-User\":[\"?1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.93 Safari/537.36\"],\"X-Forwarded-Host\":[\"sso.example.com\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"traefik-7866b8c665-gb7zd\"],\"X-Real-Ip\":[\"10.244.0.1\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"sso.example.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.244.0.1:15970\",\"RequestURI\":\"/login\",\"TLS\":null}"
time="2021-12-27T18:07:35Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/login\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"zh-CN,zh;q=0.9\"],\"Cache-Control\":[\"max-age=0\"],\"If-None-Match\":[\"W/\\\"1604-/iwgqDSVYEIOWN61rDOCdJPepvE\\\"\"],\"Sec-Ch-Ua\":[\"\\\" Not A;Brand\\\";v=\\\"99\\\", \\\"Chromium\\\";v=\\\"96\\\", \\\"Google Chrome\\\";v=\\\"96\\\"\"],\"Sec-Ch-Ua-Mobile\":[\"?0\"],\"Sec-Ch-Ua-Platform\":[\"\\\"Linux\\\"\"],\"Sec-Fetch-Dest\":[\"document\"],\"Sec-Fetch-Mode\":[\"navigate\"],\"Sec-Fetch-Site\":[\"none\"],\"Sec-Fetch-User\":[\"?1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.93 Safari/537.36\"],\"X-Forwarded-Host\":[\"sso.example.com\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"traefik-7866b8c665-gb7zd\"],\"X-Real-Ip\":[\"10.244.0.1\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"sso.example.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.244.0.1:15970\",\"RequestURI\":\"/login\",\"TLS\":null}" ForwardURL="https://10.0.83.100:1357"
time="2021-12-27T18:07:35Z" level=debug msg="'502 Bad Gateway' caused by: remote error: tls: certificate required"
time="2021-12-27T18:07:35Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/login\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"zh-CN,zh;q=0.9\"],\"Cache-Control\":[\"max-age=0\"],\"If-None-Match\":[\"W/\\\"1604-/iwgqDSVYEIOWN61rDOCdJPepvE\\\"\"],\"Sec-Ch-Ua\":[\"\\\" Not A;Brand\\\";v=\\\"99\\\", \\\"Chromium\\\";v=\\\"96\\\", \\\"Google Chrome\\\";v=\\\"96\\\"\"],\"Sec-Ch-Ua-Mobile\":[\"?0\"],\"Sec-Ch-Ua-Platform\":[\"\\\"Linux\\\"\"],\"Sec-Fetch-Dest\":[\"document\"],\"Sec-Fetch-Mode\":[\"navigate\"],\"Sec-Fetch-Site\":[\"none\"],\"Sec-Fetch-User\":[\"?1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.93 Safari/537.36\"],\"X-Forwarded-Host\":[\"sso.example.com\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"traefik-7866b8c665-gb7zd\"],\"X-Real-Ip\":[\"10.244.0.1\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"sso.example.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.244.0.1:15970\",\"RequestURI\":\"/login\",\"TLS\":null}"
10.244.0.1 - - [27/Dec/2021:18:07:35 +0000] "GET /login HTTP/2.0" 502 11 "-" "-" 3 "websecure-sso-dev-auth-sso-example-com@kubernetes" "https://10.0.83.100:1357" 8ms
time="2021-12-27T18:17:19Z" level=info msg="Anonymous stats sent to https://collect.traefik.io/9vxmmkcdmalbdi635d4jgc5p5rx0h7h8: {\"global\":{\"checkNewVersion\":true,\"sendAnonymousUsage\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"metrics\":{\"address\":\"xxxx\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{}},\"traefik\":{\"address\":\"xxxx\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{}},\"web\":{\"address\":\"xxxx\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{\"redirections\":{\"entryPoint\":{\"to\":\":443\",\"scheme\":\"https\",\"permanent\":true,\"priority\":2147483646}}}},\"web20080\":{\"address\":\"xxxx\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{\"tls\":{\"certResolver\":\"example-com-resolver\"}}},\"websecure\":{\"address\":\"xxxx\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{\"tls\":{\"certResolver\":\"example-com-resolver\"}}}},\"providers\":{\"providersThrottleDuration\":\"2s\",\"kubernetesIngress\":{},\"kubernetesCRD\":{}},\"api\":{\"dashboard\":true},\"metrics\":{\"prometheus\":{\"buckets\":[0.1,0.3,1.2,5],\"addEntryPointsLabels\":true,\"addServicesLabels\":true,\"entryPoint\":\"metrics\"}},\"ping\":{\"entryPoint\":\"traefik\",\"terminatingStatusCode\":503},\"log\":{\"level\":\"DEBUG\",\"format\":\"common\"},\"accessLog\":{\"format\":\"common\",\"filters\":{},\"fields\":{\"defaultMode\":\"keep\",\"headers\":{\"defaultMode\":\"drop\"}}},\"certificatesResolvers\":{\"example-com-resolver\":{\"acme\":{\"email\":\"xxxx\",\"caServer\":\"xxxx\",\"storage\":\"/acme/acme.json\",\"keyType\":\"RSA4096\",\"dnsChallenge\":{\"provider\":\"dnspod\"}}}},\"pilot\":{}}"
time="2021-12-27T18:17:19Z" level=debug msg="unknown kind to hash: func"

Thank you!

rtribotte · December 28, 2021, 2:18pm

Hello @Starrah,

Thanks for your interest in Traefik!

If the manifests you have shared, the namespace field is not present, so i assume that your resources are in the default namespace.

But in the annotation traefik.ingress.kubernetes.io/service.serverstransport: "auth-example-sso@kubernetescrd", the reference to the serverstransport resource is prefixed by auth-, which should be the namespace of that resource (e.g. default).

The problem lies probably in this namespace mismatch.

Thus, an error should be visible in the debug logs, can you share the full log dump? (Especially the configuration received from the Kuberbernetes providers)

Topic		Replies	Views
mTLS Between Client and Backend Server via Traefik Traefik v2 kubernetes-ingress	1	1733	June 30, 2020
ServersTransport clarifications Traefik v2 kubernetes-ingress	1	400	October 21, 2021
How to pass-through TLS with Traefik Kubernetes Ingress? Traefik v2 kubernetes-ingress	1	2329	July 17, 2023
mTLS per Kubernetes Ingress Traefik v1 kubernetes-ingress	0	1188	November 29, 2019
TLS passthrough not working Traefik v2 kubernetes-ingress	1	369	March 29, 2024

mTLS between Traefik and an upstream service (with ServersTransport.Certificates) does not work

Related Topics