Client with no certs still has access to Traefik in RequireAndVerifyClientCert

Traefik Traefik v2
,
1

I'm trying to configure a Route that requires mTLS with Traefik v2.11.0-rc1. However, I find that Traefik does not require / verify client certs when I set 'clientAuthType' as 'RequireAndVerifyClientCert'. Here is the test results. I use wget with '--no-check-certificate' to send get request, and 4002 is the entrypoint address of Traefik.

image
image1576×202 21.7 KB

I run Traefik with docker compose and use file provider. The configure files as follows

# docker compose file
version: "3.9"
services:
  traefik:
    image: traefik:v2.11.0-rc1
    container_name: "https_traefik"
    network_mode: host
    restart: always
    command:
      - "--configFile=/etc/traefik/traefik.yaml"
    volumes:
      - "./config:/file/config:ro"
      - "./static_config.yaml:/etc/traefik/traefik.yaml"
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro

# static configuration
entryPoints:
  apiserver-withcert:
    address: ":4002"
    forwardedHeaders:
      insecure: true
      trustedIPs:
        - "127.0.0.1"

accessLog:
  bufferingSize: 100
  filePath: /log/log-access.log

log:
  filePath: /log/log-file.log
  level: DEBUG

providers:
  file:
    directory: "/file/config"
    watch: true

# dynamic configuration
http:
  # Add the router
  routers:
    router1:
      entryPoints:
        - "apiserver-withcert"
      rule: ClientIP(`127.0.0.1`, `::1`)
      service: service-apiserver-withcert
      middlewares:
        - test-replacepathregex
      tls:
        options: kube-withcert
  
  # Add the middleware
  middlewares:
    test-replacepathregex:
      replacePathRegex:
        regex: "^/(.*)"
        replacement: "/$1"

  # Add the Servers Transport
  serversTransports:
    withcert: 
      insecureSkipVerify: true
      rootCAs:
        - /file/config/ca.crt
      certificates:
        - certFile: /file/config/client.crt
          keyFile: /file/config/client.key
  
  # Add the service
  services:
    service-apiserver-withcert:
      loadBalancer:
        healthCheck:
          path: /healthz
          interval: "10s"
          timeout: "3s"
        serversTransport: withcert
        servers:
          - url: https://127.0.0.1:6443

tls:
  certificates:
    - certFile: /file/config/server.crt
      keyFile: /file/config/server.key
      stores:
        - default
  stores:
    default:
      defaultCertificate:
        certFile: /file/config/server.crt
        keyFile: /file/config/server.key
  options:
    kube-withcert:
      clientAuth:
        caFiles:
          - /file/config/ca.crt
        clientAuthType: RequireAndVerifyClientCert

debug logs as follows

bash-4.4# docker logs -f --tail 100 https_traefik
time="2024-01-08T14:19:50+08:00" level=info msg="Configuration loaded from file: /etc/traefik/traefik.yaml"
time="2024-01-08T14:19:50+08:00" level=info msg="Traefik version 2.11.0-rc1 built on 2024-01-03T10:16:22Z"
time="2024-01-08T14:19:50+08:00" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"apiserver-withcert\":{\"address\":\":4002\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}}},\"providers\":{\"providersThrottleDuration\":\"2s\",\"file\":{\"directory\":\"/file/config\",\"watch\":true}},\"log\":{\"level\":\"DEBUG\",\"format\":\"common\"},\"accessLog\":{\"filePath\":\"/log/log-access.log\",\"format\":\"common\",\"filters\":{},\"fields\":{\"defaultMode\":\"keep\",\"headers\":{\"defaultMode\":\"drop\"}},\"bufferingSize\":100}}"
time="2024-01-08T14:19:50+08:00" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\n"
time="2024-01-08T14:19:50+08:00" level=info msg="Starting provider aggregator aggregator.ProviderAggregator"
time="2024-01-08T14:19:50+08:00" level=debug msg="Starting TCP Server" entryPointName=apiserver-withcert
time="2024-01-08T14:19:50+08:00" level=info msg="Starting provider *file.Provider"
time="2024-01-08T14:19:50+08:00" level=debug msg="*file.Provider provider configuration: {\"directory\":\"/file/config\",\"watch\":true}"
time="2024-01-08T14:19:50+08:00" level=info msg="Starting provider *traefik.Provider"
time="2024-01-08T14:19:50+08:00" level=debug msg="*traefik.Provider provider configuration: {}"
time="2024-01-08T14:19:50+08:00" level=info msg="Starting provider *acme.ChallengeTLSALPN"
time="2024-01-08T14:19:50+08:00" level=debug msg="*acme.ChallengeTLSALPN provider configuration: {}"
time="2024-01-08T14:19:50+08:00" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"router1\":{\"entryPoints\":[\"apiserver-withcert\"],\"middlewares\":[\"test-replacepathregex\"],\"service\":\"service-apiserver-withcert\",\"rule\":\"ClientIP(`127.0.0.1`, `::1`)\",\"tls\":{\"options\":\"kube-withcert\"}}},\"services\":{\"service-apiserver-withcert\":{\"loadBalancer\":{\"servers\":[{\"url\":\"https://127.0.0.1:6443\"}],\"healthCheck\":{\"path\":\"/healthz\",\"interval\":\"10s\",\"timeout\":\"3s\",\"followRedirects\":true},\"passHostHeader\":true,\"serversTransport\":\"withcert\"}}},\"middlewares\":{\"test-replacepathregex\":{\"replacePathRegex\":{\"regex\":\"^/(.*)\",\"replacement\":\"/$1\"}}},\"serversTransports\":{\"withcert\":{\"insecureSkipVerify\":true}}},\"tcp\":{},\"udp\":{},\"tls\":{\"options\":{\"kube-withcert\":{\"clientAuth\":{\"clientAuthType\":\"RequireAndVerifyClientCert\"},\"alpnProtocols\":[\"h2\",\"http/1.1\",\"acme-tls/1\"]}},\"stores\":{\"default\":{}}}}" providerName=file
time="2024-01-08T14:19:50+08:00" level=debug msg="Configuration received: {\"http\":{\"services\":{\"noop\":{}},\"serversTransports\":{\"default\":{\"maxIdleConnsPerHost\":200}}},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=internal
time="2024-01-08T14:19:50+08:00" level=debug msg="Adding certificate for domain(s) 10.116.110.124,10.116.110.125,10.116.110.126,127.0.0.1,172.20.180.115,172.20.180.116,172.20.180.117,172.20.180.118,sfs"
time="2024-01-08T14:19:50+08:00" level=debug msg="Creating middleware" entryPointName=apiserver-withcert routerName=router1@file middlewareType=Pipelining middlewareName=pipelining serviceName=service-apiserver-withcert
time="2024-01-08T14:19:50+08:00" level=debug msg="Creating load-balancer" routerName=router1@file serviceName=service-apiserver-withcert entryPointName=apiserver-withcert
time="2024-01-08T14:19:50+08:00" level=debug msg="Creating server 0 https://127.0.0.1:6443" routerName=router1@file serverName=0 serviceName=service-apiserver-withcert entryPointName=apiserver-withcert
time="2024-01-08T14:19:50+08:00" level=debug msg="child https://127.0.0.1:6443 now UP"
time="2024-01-08T14:19:50+08:00" level=debug msg="Propagating new UP status"
time="2024-01-08T14:19:50+08:00" level=debug msg="Added outgoing tracing middleware service-apiserver-withcert" routerName=router1@file middlewareType=TracingForwarder middlewareName=tracing entryPointName=apiserver-withcert
time="2024-01-08T14:19:50+08:00" level=debug msg="Creating middleware" middlewareName=test-replacepathregex@file middlewareType=ReplacePathRegex entryPointName=apiserver-withcert routerName=router1@file
time="2024-01-08T14:19:50+08:00" level=debug msg="Adding tracing to middleware" entryPointName=apiserver-withcert routerName=router1@file middlewareName=test-replacepathregex@file
time="2024-01-08T14:19:50+08:00" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=apiserver-withcert middlewareName=traefik-internal-recovery
time="2024-01-08T14:19:50+08:00" level=debug msg="Setting up healthcheck for service service-apiserver-withcert@file with [Hostname:  Headers: map[] Path: /healthz Method:  Port: 0 Interval: 10s Timeout: 3s FollowRedirects: true]" serviceName=service-apiserver-withcert@file
time="2024-01-08T14:19:50+08:00" level=warning msg="No domain found in rule ClientIP(`127.0.0.1`, `::1`), the TLS options applied for this router will depend on the SNI of each request" entryPointName=apiserver-withcert routerName=router1@file
time="2024-01-08T14:19:50+08:00" level=debug msg="Initial health check for backend: \"service-apiserver-withcert@file\""
time="2024-01-08T14:20:00+08:00" level=debug msg="Routine health check refresh for backend: service-apiserver-withcert@file"
time="2024-01-08T14:20:10+08:00" level=debug msg="Routine health check refresh for backend: service-apiserver-withcert@file"

Any ideas on what could be going wrong or suggestions on how to debug?

2

From a first look at the docs [1, 2] it seems okay:

# Dynamic configuration
http:
  routers:
    Router-1:
      rule: "Host(`foo-domain`) && Path(`/foo-path/`)"
      service: service-id
      # will terminate the TLS request
      tls:
        options: foo

tls:
  options:
    foo:
      minVersion: VersionTLS12
      cipherSuites:
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
        - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
        - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

# Dynamic configuration

tls:
  options:
    default:
      clientAuth:
        # in PEM format. each file can contain multiple CAs.
        caFiles:
          - tests/clientca1.crt
          - tests/clientca2.crt
        clientAuthType: RequireAndVerifyClientCert

Can you share your full docker-compose.yml?

3

I just start Traefik with the content posted below. (command: docker compose -f ./traefik/traefik.yaml up -d)

# content of 'traefik.yaml'
version: "3.9"
services:
  traefik:
    image: traefik:v2.11.0-rc1
    container_name: "https_traefik"
    network_mode: host
    restart: always
    command:
      - "--configFile=/etc/traefik/traefik.yaml"
    volumes:
      - "./config:/file/config:ro"
      - "./static_config.yaml:/etc/traefik/traefik.yaml"
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - ./log:/log:rw

And my directory is shown as follows ( static configuration content in 'static_config.yaml' and dynamic configuration content in 'config/config.yaml' ):

image
image704×322 15.7 KB

4

It seems very unusual to use network_mode: host for Traefik. Best practice is to use ports to expose only Traefik needed ports and share a Docker network with a target service. See simple Traefik example.

But not sure if that changes anything with your issue.

5

thx for reply~
I found the root cause: if no domain found in rule configure, Taefik works with the default TLS option rather than it defined in routers.<router-name>.tls.options field

We can find the warning log:

 level=warning msg="No domain found in rule ClientIP(`172.18.0.1`, `127.0.0.1`, `::1`), the TLS options applied for this router will depend on the SNI of each request" entryPointName=server routerName=router1@file
 level=debug msg="TLS options difference: SNI:default, Header:withcert@file" req.Host="127.0.0.1:4002" req.TLS.ServerName= host=127.0.0.1