Traefik routing to all pods even though the Rule specified the exact service to route to

Traefik Traefik v2
,
1

I'm not sure I understand how the routing works but following my configuration, I believe Traefik should only forward requests to the service I've specified. Please help.

Here's my Kubernetes IngressRoute configuration (not including the definitions and other resources)

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: ingressroutenotls
  namespace: koolaid-production
spec:
  entryPoints:
    - web
  routes:
  - match: Host(`events.new.koolaid.co`)
    kind: Rule
    services:
    - kind: Service
      name: koolaidevents
      namespace: koolaid-production
      port: 80

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: ingressroutetls
  namespace: koolaid-production
spec:
  entryPoints:
    - websecure
  routes:
  - match: Host(`events.new.koolaid.co`)
    kind: Rule
    services:
    - kind: Service
      name: koolaidevents
      namespace: koolaid-production
      port: 80
  tls:
    certResolver: myresolver
    domains:
    - main: new.koolaid.co
      sans:
      - events.new.koolaid.co

Service koolaidevents is deployed separately via Gitlab CI/CD and it detects it automatically since they are in the same namespace. I also have a different service in this same cluster with its own set of pods. The problem is it's load-balancing the requests across all pods in the cluster. I expect it to to all load-balance across pods only within koolaidevents. Whenever the request hits any of the pods in the other service, it just returns a 502 - Bad Gateway.

I'm totally new to using Traefik. Please help out.

2

Purely anecdotal: I saw a cluster with a bunch of traefik ingresses and one of the ingresses by mistake had a wild card host rule. Naturally all the other apps from time to time would hit that ingress, since it would match anything, which resulted in a wrong app being served.

The takeaway is: other apps ingresses can affect routing if your app.

3

Thanks @zespri
I have tried listing all ingresses and ingressroutes in all namespaces and I couldn't find any but the two I configured above.

> kubectl get ingress --all-namespaces
No resources found

> kubectl get ingressroute --all-namespaces
NAMESPACE             NAME                AGE
koolaid-production   ingressroutenotls   18h
koolaid-production   ingressroutetls       18h
4

They seem to match the same host events.new.koolaid.co don't they?

5

@zespri
One is for HTTP and the other is for HTTPS

I did suspect it as well and I removed it but it's still doing the same.

6

And they both backed up by the same service?

7

Yes @zespri

Shouldn't they ?

8

What is "the other service" then? The one that returns 502 - Bad Gateway?

Another thing to try is to check endpoints on this service of yours, and see if they are what you are expecting them to be.

9

@zespri Thanks a lot for your time. I really appreciate. I removed the other service while I was confused and trying to debug. I have updated the above configuration below to include the other service.

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: ingressroutenotls
  namespace: koolaid-production
spec:
  entryPoints:
    - web
  routes:
  - match: Host(`events.new.koolaid.co`)
    kind: Rule
    services:
    - kind: Service
      name: koolaidevents
      namespace: koolaid-production
      port: 80
  - match: Host(`afrione.new.koolaid.co`)
    kind: Rule
    services:
    - kind: Service
      name: afrione
      port: 80

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: ingressroutetls
  namespace: koolaid-production
spec:
  entryPoints:
    - websecure
  routes:
  - match: Host(`events.new.koolaid.co`)
    kind: Rule
    services:
    - kind: Service
      name: koolaidevents
      namespace: koolaid-production
      port: 80
  - match: Host(`afrione.new.koolaid.ng`)
    kind: Rule
    services:
    - kind: Service
      name: afrione
      port: 80
  tls:
    certResolver: myresolver
    domains:
    - main: new.koolaid.co
      sans:
      - events.new.koolaid.co
      - afrione.new.koolaid.co

Both services have been deployed to the cluster

10

What you might want to try is to check the endpoints on your services, and see if they are what you are expecting them to be.

11

Thanks so much @zespri

The issue is actually in the application. It's listening on all interfaces 0.0.0.0

I changed that to localhost still didn't work and had to use Node.js os and dns libraries to resolve the Pod's IP address and it worked.

I'm still not totally clear though. I thought just specifying the service name as I did above should correctly route all requests that match that host to the specific service's ClusterIP which should handle the routing to the appropriate pod. Am I missing something ?

12

By and large. There is an implementation detail, that for performance reasons traefik does not use service's ClusterIP, but uses the endpoints directly. The result though must be the same. It is though important that those endpoints are what you expect them to be.

Sorry, I have no idea what you are talking about. What application? What "didn't work"? What pod's IP address? How is it used in the application?

May be draw a diagram?

13

@zespri
It's a Node.js server (fastify to be precise) that was listening on 0.0.0.0 which listens on all interfaces. fastify.listen(PORT, '0.0.0.0')

What I did to resolve the issue was get the Kubernetes pod's IP with
dns.resolve(os.hostname(), (err, addr) => ...)
then listen on the addr returned.

Now the issue has resurfaced after I deployed like 5 frontends. Trying to access one of the frontend and Traefik started routing to all the pods and I can confirm that each of these pods is listening on its Pod's IP and a different port.

14

It does not sound relevant to me as in it should have no bearing on traefik routing.

Trying to access one of the frontend and Traefik started routing to all the pods and I can confirm that each of these pods is listening on its Pod's IP and a different port.

This sounds vague. Are you saying that you are hitting http://events.new.koolaid.co but getting a pod from behind afrione kubernetes service sometimes?

15

Yes @zespri

The routing is random. I have 3 pods per each service and I expected that only these 3 pods should receive the requests since the Rule says so. This is not so and pods in other services matched by a different Rule also get the request sometimes at random.

16

I have 3 pods per each service

Can you confirm the endpoints please as I asked above.

17

@zespri

I have these from kubectl get endpoints

NAME             ENDPOINTS                                                            AGE
afrione          10.244.0.118:3010,10.244.0.119:3010,10.244.0.120:3010 + 12 more...   11h
carlcare         10.244.0.118:3011,10.244.0.119:3011,10.244.0.120:3011 + 12 more...   8h
ensure           10.244.0.118:3012,10.244.0.119:3012,10.244.0.120:3012 + 12 more...   8h
hasura           10.244.0.173:8080                                                    9h
koolaidevents    10.244.0.190:3000,10.244.0.244:3000,10.244.0.35:3000                 11h
raya             10.244.0.118:3013,10.244.0.119:3013,10.244.0.120:3013 + 12 more...   8h
traefik          10.244.0.201:8000,10.244.0.201:443,10.244.0.201:8080                 11h
vida             10.244.0.118:3014,10.244.0.119:3014,10.244.0.120:3014 + 12 more...   8h
18

Cool! So you said "I have 3 pods per each service" but there are 15 endpoints on e.g. afrione? Could you please investigate a bit further to resolve this mismatch?

19

@zespri
As suggested, I investigated and found out that Kubernetes group endpoints by label selectors (https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/). I simply assigned a different label to each service and everything sorted.

Thanks so much for your time and patience.

20

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.