Bad gateway returned when using mutual auth

What did you do?

When attempting to use mutual auth talking to a Tomcat 8 instance setting the TLS_OPTS as:

spec: clientAuth: clientAuthType: RequireAndVerifyClientCert secretNames: - ca-cert

I'm get bad gateway returned. If I turn off mutual auth in Tomcat then I can connect but no certificate is presented. Looking in the Tomcat ssl debug logs I can see the certificate I expect being presented and no error, but still get a bad gateway response. Using the same certificate given to Traefik directly connecting to Tomcat via the Kubernetes service external IP works fine.

What did you expect to see?

I would expect for the certificate to be passed to tomcat and for the connection to be made successfully.

What did you see instead?

I'm getting the following error message:

level=debug msg="'502 Bad Gateway' caused by: remote error: tls: bad certificate"

Output of traefik version: (What version of Traefik are you using?)

Traefik version 2.1.2 built on 2020-01-07T18:23:57Z

If applicable, please paste the log output in DEBUG level (--log.level=DEBUG switch)

`2020-02-12T16:29:23.303959756Z time="2020-02-12T16:29:23Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/spine/soap_api\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Apikey\":[\"fa43ed2c-7204-4ae6-b9c8-c1d4536c661a\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"keep-alive\"],\"Content-Length\":[\"601\"],\"Content-Type\":[\"text/xml\"],\"Postman-Token\":[\"07388e12-839f-486d-a393-2e800e8fdb81\"],\"User-Agent\":[\"PostmanRuntime/7.22.0\"],\"X-Forwarded-For\":[\"81.171.156.220:54127\"],\"X-Forwarded-Host\":[\"XXX\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"traefik-7d78d45b4b-6dmmb\"],\"X-Original-Host\":[\"XXX"],\"X-Original-Url\":[\"/spine/soap_api\"],\"X-Real-Ip\":[\"192.168.20.85\"]},\"ContentLength\":601,\"TransferEncoding\":null,\"Host\":\"XXX",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"192.168.20.85:58188\",\"RequestURI\":\"/spine/soap_api\",\"TLS\":null}"
2020-02-12T16:29:23.304023956Z time="2020-02-12T16:29:23Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/spine/soap_api\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Apikey\":[\"fa43ed2c-7204-4ae6-b9c8-c1d4536c661a\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"keep-alive\"],\"Content-Length\":[\"601\"],\"Content-Type\":[\"text/xml\"],\"Postman-Token\":[\"07388e12-839f-486d-a393-2e800e8fdb81\"],\"User-Agent\":[\"PostmanRuntime/7.22.0\"],\"X-Forwarded-For\":[\"81.171.156.220:54127\"],\"X-Forwarded-Host\":[\"XXX\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"traefik-7d78d45b4b-6dmmb\"],\"X-Original-Host\":[\"XXX\"],\"X-Original-Url\":[\"/spine/soap_api\"],\"X-Real-Ip\":[\"192.168.20.85\"]},\"ContentLength\":601,\"TransferEncoding\":null,\"Host\":\"XXX",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"192.168.20.85:58188\",\"RequestURI\":\"/spine/soap_api\",\"TLS\":null}" ForwardURL="https://192.168.20.150:8443"
2020-02-12T16:29:23.315837846Z time="2020-02-12T16:29:23Z" level=debug msg="'502 Bad Gateway' caused by: remote error: tls: bad certificate"
2020-02-12T16:29:23.315864547Z time="2020-02-12T16:29:23Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/spine/soap_api\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Apikey\":[\"fa43ed2c-7204-4ae6-b9c8-c1d4536c661a\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"keep-alive\"],\"Content-Length\":[\"601\"],\"Content-Type\":[\"text/xml\"],\"Postman-Token\":[\"07388e12-839f-486d-a393-2e800e8fdb81\"],\"User-Agent\":[\"PostmanRuntime/7.22.0\"],\"X-Forwarded-For\":[\"81.171.156.220:54127\"],\"X-Forwarded-Host\":[\"XXX"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"traefik-7d78d45b4b-6dmmb\"],\"X-Original-Host\":[\"XXX\"],\"X-Original-Url\":[\"/spine/soap_api\"],\"X-Real-Ip\":[\"192.168.20.85\"]},\"ContentLength\":601,\"TransferEncoding\":null,\"Host\":\"XXX",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"192.168.20.85:58188\",\"RequestURI\":\"/spine/soap_api\",\"TLS\":null}"
2020-02-12T16:29:23.315892247Z IPADDRESS:54127 - - [12/Feb/2020:16:29:23 +0000] "POST /spine/soap_api HTTP/1.1" 502 11 "-" "-" 924 "XXX-soap-00e823cbfd49f107fd17@kubernetescrd" "https://192.168.20.150:8443" 12ms
2020-02-12T16:29:23.408789857Z time="2020-02-12T16:29:23Z" level=debug msg="Skipping Kubernetes event kind *v1.Secret" providerName=kubernetescrd`

Any help would be greatly appreciated as I've been scratching my head trying to get this fixed for a couple of days now.

Thank you!