ServersTransport clarifications

Traefik Traefik v2
1

I'm curious to know if there is a good way to debug the interactions of a ServersTransport in a k8s environment. I'm in charge of integrating a backend service that is using self-signed certs. I'm very new to using Traefik and I cannot determine from the documentation how to get more visibility into my issues. I would like to be able to see the request being made with the headers much like the view you get if you use a browser's debugging console. The setup I'm trying to configure looks like this

  1. IngressRoute (TLS termination)
  2. Middleware (for additional headers for the backend)
  3. ServersTransport (establish mTLS with certs & certificate authority cert defined from the backend service)
  4. Backend Service

I made some progress, but it seems that I was only able to get to the BackendService once an admin set --serversTransport.insecureSkipVerify=true in the traefik Deployment which I think does it globally. This isn't what I wanted, I actually want mTLS setup correctly. I've downloaded the certificates and CA certs from the k8s Secrets and did a port forward of the backend service and with curl was able to make the request with the custom headers as expected.

  1. Are there known issues withe 2.4.8 release of Traefik and the ServersTransport establishing mTLS?
  2. If you use a ServersTransport object do you lose any headers that were set in Middleware?
  3. Any thoughts on what I should try? Or am I approaching this incorrectly?
2

After a lot more investigation, I think this is my core question.