Traefik not loading client certificate for mTLS

i am trying to enable mTLS to access the traefik dashboard, but when i run https://{ip-addr}:8081/dashboard/ it is giving me the following error

{"level":"debug","msg":"http: TLS handshake error from 10.0.2.100:44550: remote error: tls: unknown certificate","time":"2023-07-25T07:36:35Z"}
{"level":"debug","msg":"Serving default certificate for request: \"\"","time":"2023-07-25T07:36:35Z"}
{"level":"debug","msg":"http: TLS handshake error from 10.0.2.100:44552: tls: client didn't provide a certificate","time":"2023-07-25T07:36:35Z"}

i have configured mtls in my dynamic config file as follows:

[http]
  [http.serversTransports]
    [http.serversTransports.traefik-mtls]
      [[http.serversTransports.traefik-mtls.certificates]]
        certFile = "/certs/client.crt"
        keyFile = "/certs/client.key"

  
  [tls.options]
    [tls.options.default]
      [tls.options.default.clientAuth]

        clientAuthType = "RequireAndVerifyClientCert"
        caFiles = [
          "/certs/ca_combined.pem"
        ]

any help would be greatly appreciated, i am not sure why my client is not presenting any certificates

Share your full Traefik static and dynamic config, and docker-compose.yml if used.

AFAIK serversTransports is used for communication between Traefik and the (external) target service.

my dynamic tls.toml file as requested here:

# Dynamic Configuration for Traefik SSL

# Enables mTLS for Traefik to Backend Service communication
# Identifies outgoing requests as from Traefik
[http]
  [http.serversTransports]
    [http.serversTransports.traefik-mtls]
      [[http.serversTransports.traefik-mtls.certificates]]
        certFile = "/certs/client.crt"
        keyFile = "/certs/client.key"

# Enables TLS for Calls to Traefik
# Ensures that domains used in incoming requests are identified
[tls]

  [[tls.certificates]]
    certFile = "/certs/webserver_cert.crt"
    keyFile = "/certs/webserver_cert.key"
    stores = ["default"]

  [tls.stores]
    [tls.stores.default]
      [tls.stores.default.defaultCertificate]
        certFile = "/certs/webserver_cert.crt"
        keyFile = "/certs/webserver_cert.key"
  
  [tls.options]
    [tls.options.default]
      [tls.options.default.clientAuth]

        clientAuthType = "RequireAndVerifyClientCert"
        caFiles = [
          "/certs/ca.pem"
        ]

my static config as shown here:

################################################################
# Global configuration
################################################################
[global]
  checkNewVersion = false
  sendAnonymousUsage = false

################################################################
# Entrypoints configuration
################################################################

# Entrypoints definition
[entryPoints]

# Dashboard and API
  [entryPoints.traefik_https]
    address = ":8443"
    [entryPoints.traefik_https.http.tls]  

  [entryPoints.ping]
    address = ":8082"

  [entryPoints.web-secure]
    address = ":8081"

    # TLS is enabled by having this empty option
    [entryPoints.web-secure.http.tls]

[serversTransport]
  insecureSkipVerify = false

################################################################
# API and dashboard configuration
################################################################

# Enable API and dashboard
[api]
  dashboard = true
  debug = true

################################################################
# Ping configuration
################################################################

# Enable ping
[ping]
  entryPoint = "ping"

################################################################
# File-Based Configuration Backend
################################################################

[providers]
  [providers.file]
    directory = "/config"
    watch =  true

Maybe this tutorial can help.