Tracing a Kubernetes (K3S) inter-namespace Traefik routing issue

Traefik Traefik v2
1

I'm running the latest K3S, MetalLB, and Traefik 2.5 on Fedora 35 Server. I installed K3S without Traefik and the built-in load balancer. Installing the MetalLB and Traefik projects was easy. I'm also running Cert-Manager and I've generated a signed SSL certificate for the Traefik dashboard. All of that works.

My problem? Routing Ingress or IngressRoute traffic from trafik to another namespace kubernetes endpoint. Traefik is able to build the route from the IngressRoute to the endpoint. I can see it in the dashboard. But when I attempt to execute a curl command using the deployed application's DNS namd i.e. apache.apps.example.com, I get "Bad Gateway." I've increased Traefik logging to Deubg, and I'm getting a 502 no route to host error message. My assumption is this: A kuberntes policy is blocking the traffic between namespaces. Either it has to do with the kubernetes service and endpoint's port, which is usually 80, or there is a missing network rule, a CRD, or something else. I shelled into the traefik pod and curl to the endpoint or service ClusterIP address generates the same error message.

My question is: What else should I be logging? I think there is something that I'm overlooking due to my newness to Kubernetes itself.

Any ideas?

2

I think that I answered this on Reddit, but I'll paste my most recent response here as well:

Fire up a test pod (monachus/sak is my tools/troubleshooting image) in the same namespace as Traefik and see if it can connect to the Service and Pod addresses with curl or nc. If it can, then it's Traefik. If it can't, then it's truly a route issue. If it's a route issue, fire up that Pod in a different namespace and see if it can connect. If it can, then it's a route issue limited to the Traefik namespace.

If it's a Traefik issue, look at your startup options (kubectl get deploy traefik -o yaml) and make sure you have the cross-namespace options in the startup flags.

--providers.kubernetescrd.allowExternalNameServices=true
--providers.kubernetesIngress.allowExternalNameServices=true

If it's a routing issue, start troubleshooting all the layers. Look for obvious things like, "did I tell Traefik to send it to the right Service name?" and "does the Service have healthy endpoints?" and "did I choose the right port?"

Here is a page with Kubernetes troubleshooting steps to make sure all those components are running.