Using multiple Traefik Load balancer services with different IPs

Traefik Traefik v2
,
1

Hi - I am configuring Traefik v2 (installed by k3s) to run two Traefik load-balancer services, each assigned their own external IP address defined as address-pools in MetalLB.

The problem I have is similiar to the forum post:
Using multiple metallb IP address pools with Traefik - Traefik / Traefik v2 (latest) - Traefik Labs Community Forum.

I have created two traefik ingress load-balancer services and they are assigned the IP addresses intended:

$ kubectl get -n kube-system svc

NAME             TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)                                        AGE
traefik          LoadBalancer   10.43.114.151   192.168.10.30   9140:31141/TCP,9143:30527/TCP,9151:30031/TCP   14d
traefik-test     LoadBalancer   10.43.83.219    192.168.10.32   80:32533/TCP,443:31146/TCP                     94m

Note - traefik is the default ingress LB service setup by k3s, and traefik-test is the LB I created as an ingress for applications in the test namespace.

Apologies about the long-winded description - I'm hoping more details is better ...

The issue:
When I try accessing 2 CluserIP service (the apple-service and banana-service services) via the fruit-ingress IngressRoute I get the error - 'failed: No route to host':

$ wget http://test.web.local/apple

--2023-08-28 05:00:00--  http://test.web.local/apple
Resolving test.web.local (test.web.local)... 192.168.10.32
Connecting to test.web.local (test.web.local)|192.168.10.32|:80... failed: No route to host.

$ wget http://test.web.local/banana

--2023-08-28 05:00:05--  http://test.web.local/banana
Resolving test.web.local (test.web.local)... 192.168.10.32
Connecting to test.web.local (test.web.local)|192.168.10.32|:80... failed: No route to host.

The fruit-ingress IngressRoute which performs HTTP routing is:

$  kubectl get -n test ingressroute
NAME            AGE
fruit-ingress   136m

The ClusterIP services the traefik-test LB should route to are:

$ kubectl get -n test svc

NAME               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
apple-service      ClusterIP   10.43.16.73     <none>        5678/TCP   97m
banana-service     ClusterIP   10.43.215.131   <none>        5678/TCP   97m

These ClusterIP services are associated with the following pods:

$ kubectl get -n test po

NAME           READY   STATUS    RESTARTS   AGE
apple-app      1/1     Running   0          97m
banana-app     1/1     Running   0          97m

Please let feel free to provide advice on troubleshooting and resolving this issue :).

Below is some more details about my configuration and troubleshooting I have performed.

Configuration

The k8s manifest for test applications and services I am trying to route to using the traefik-test LB service are :

apiVersion: v1
kind: Namespace
metadata:
  name: test
---
kind: Pod
apiVersion: v1
metadata:
  name: apple-app
  namespace: test
  labels:
    app: apple
spec:
  containers:
    - name: apple-app
      image: hashicorp/http-echo
      args:
        - "-text=apple2"
---
kind: Service
apiVersion: v1
metadata:
  name: apple-service
  namespace: test
spec:
  selector:
    app: apple
  ports:
    - port: 5678
---
kind: Pod
apiVersion: v1
metadata:
  name: banana-app
  namespace: test
  labels:
    app: banana
spec:
  containers:
    - name: banana-app
      image: hashicorp/http-echo
      args:
        - "-text=banana2"
---
kind: Service
apiVersion: v1
metadata:
  name: banana-service
  namespace: test
spec:
  selector:
    app: banana
  ports:
    - port: 5678

The manifest used for the IngressRoute to route requests to these services is:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: fruit-ingress
  namespace: test
  labels:
    app.kubernetes.io/instance: traefik
    app.kubernetes.io/name: traefik
spec:
  entryPoints:
    - test-http
  routes:
  - match: Host(`test.web.local`) && Path(`/apple`)
    kind: Rule
    services:
    - name: apple-service
      port: 5678
  - match: Host(`test.web.local`) && Path(`/banana`)
    kind: Rule
    services:
    - name: banana-service
      port: 5678

The manifest for the traefik-test LB service used for the test.web.local domain (IP address 192.168.10.32) is:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    ports:
      test-http:
        port: 9140
        protocol: TCP
        expose: true
        exposedPort: 9140
      test-https:
        port: 9143
        protocol: TCP
        expose: true
        exposedPort: 9143

The manifest defining the MetalLB ip address pools used by the Traefik LB services are:

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: pool-default
  namespace: metallb-system
spec:
  addresses:
  - 192.168.2.30/32
---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: pool-test
  namespace: metallb-system
spec:
  addresses:
  - 192.168.2.32/32
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: default-l2advertisement
  namespace: metallb-system
spec:
  ipAddressPools:
  - pool-default
  - pool-test

Troubleshooting

Traefik appears to have recognised the HelmChartConfig file I placed in the k3s manifects folder and updated the traefik deployment with the requested settings:

$ kubectl describe -n kube-system deployments.apps traefik

Name:                   traefik
Namespace:              kube-system
...
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
...
  Service Account:  traefik
  Containers:
   traefik:
    Image:       rancher/mirrored-library-traefik:2.9.4
    Ports:       9100/TCP, 9140/TCP, 9143/TCP, 9000/TCP, 8000/TCP, 8443/TCP
    Host Ports:  0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP
    Args:
      --global.checknewversion
      --global.sendanonymoususage
      --entrypoints.metrics.address=:9100/tcp
      --entrypoints.test-http.address=:9140/tcp
      --entrypoints.test-https.address=:9143/tcp
      --entrypoints.traefik.address=:9000/tcp
      --entrypoints.web.address=:8000/tcp
      --entrypoints.websecure.address=:8443/tcp
      --api.dashboard=true
      --ping=true
      --metrics.prometheus=true
      --metrics.prometheus.entrypoint=metrics
      --providers.kubernetescrd
      --providers.kubernetesingress
      --providers.kubernetesingress.ingressendpoint.publishedservice=kube-system/traefik
      --entrypoints.websecure.http.tls=true
   ...

I inspected the logs for both the traefik and traefik-test LB services and no errors were displayed:

$ kubectl logs -n kube-system svc/traefik-test

(No log events displayed)

$ kubectl logs -n kube-system svc/traefik

time="2023-08-28T03:20:00Z" level=info msg="Configuration loaded from flags."

I created an busybox pod in the test namespace to verify connectivity between the ClusterIP services and the associated services - and the tests were successful. Tests run inside busybox pod shell were:

# Send request to `apple-service` cluser IP service.
$ wget -O- http://10.43.215.131:5678

Connecting to 10.43.215.131:5678 (10.43.215.131:5678)
writing to stdout
banana2
-                    100% |************************************************************************|     8  0:00:00 ETA

# Send request to `banana-service` cluser IP service.
$ wget -O- http://10.43.16.73:5678
Connecting to 10.43.16.73:5678 (10.43.16.73:5678)
writing to stdout
apple2
-                    100% |************************************************************************|     7  0:00:00 ETA