Connection to backend service times out when running multiple Traefik pod instances

Traefik Traefik v2 (latest)
, , ,
1

I am using Traefik v2.3.2 inside of a 1.16 Kubernetes cluster. My Kubernetes cluster uses the Traefik ingress controller as its entry point. My Traefik IC runs behind MetalLB and within a Linkerd service mesh and behind MetalLB.

When running multiple Traefik IC pods to provide HA, we see that we get timeouts on our HTTP and TCP client connections. If we only have 1 instance of the Traefik IC, then all works fine. Perhaps I'm missing a configuration setting in my deployment but I'm at a loss as to why the Traefik IC is exhibiting this behaviour. Any thoughts or suggestions would be appreciated.

Thanks

2

Here is an update from my latest latest in hopes of tracking down the issue.
I believe the issue maybe resulting from the fact that I have 2 traefik services (TCP and UDP) defined both sharing the same external IP.

If I delete the UDP service and just keep the TCP service definition, all runs smoothly with no connection drops. Both my the UDP and TCP service definitions have:

According to the metallb documentation, this should be what I need to enable my services to both share the same external IP.

Below are my service definitions:

apiVersion: v1
kind: Service
metadata:
  name: traefik-lb
  namespace: default
  annotations:
    metallb.universe.tf/allow-shared-ip: default
spec:
  externalTrafficPolicy: Local
  ports:
    - protocol: TCP
      name: web
      port: 80
    - protocol: TCP
      name: websecure
      port: 443
    - protocol: TCP
      name: admin
      port: 22 
    - protocol: TCP
      name: amqp 
      port: 5672
    - protocol: TCP
      name: amqp-tls 
      port: 5671
    - protocol: TCP
      name: es 
      port: 9300
  selector:
    app: traefik-ingress
  type: LoadBalancer
  loadBalancerIP: X.XXX.XXX.XXXX

---
apiVersion: v1
kind: Service
metadata:
  name: traefik-lb
  namespace: default
  annotations:
    metallb.universe.tf/allow-shared-ip: default
spec:
  externalTrafficPolicy: Local
  ports:
    - protocol: UDP 
      name: udp-in 
      port: 7777
  selector:
    app: traefik-ingress
  type: LoadBalancer
  loadBalancerIP: X.XXX.XXX.XXXX

---
kind: Deployment
apiVersion: apps/v1
metadata:
  namespace: default
  name: traefik-ingress
  labels:
    app: traefik-ingress
spec:
  replicas: 5
  selector:
    matchLabels:
      app: traefik-ingress
  template:
    metadata:
      labels:
        app: traefik-ingress
    spec:
      serviceAccountName: traefik-ingress-controller
      containers:
        - name: traefik
          image: traefik:v2.3.2
          args:
            - --api.dashboard=true
            - --accesslog
            - --entrypoints.traefik.address=:9000/tcp
            - --entrypoints.web.address=:8080
            - --entrypoints.web.http.redirections.entryPoint.to=websecure
            - --entrypoints.web.http.redirections.entryPoint.scheme=https
            - --entrypoints.websecure.address=:443
            - --entrypoints.websecure.http.tls
            - --entrypoints.websecure.http.tls.options=default
            - --entrypoints.amqp.address:=5672/tcp
            - --entrypoints.amqp-tls.address:=5671/tcp
            - --entrypoints.es.address:=9300/tcp
            - --entrypoints.udp-in.address:=7777/udp
            - --providers.kubernetesingress=true
            - --providers.kubernetesingress.ingressclass=traefik-default
            - --providers.kubernetesingress.ingressendpoint.publishedservice=default/traefik-in2
            - --providers.kubernetescrd=true
            - --providers.kubernetescrd.ingressclass=traefik-default
            - --ping=true
            - --log
            - --log.level=INFO
          ports:
            - name: web
              containerPort: 8080
              protocol: TCP
            - name: websecure
              containerPort: 8443
              protocol: TCP
            - name: admin
              containerPort: 8022
              protocol: TCP
            - name: amqp
              containerPort: 5672 
              protocol: TCP
            - name: amqp-tls
              containerPort: 5671
              protocol: TCP
            - name: es 
              containerPort: 9300
              protocol: TCP
            - name: traefik 
              containerPort: 9000
              protocol: TCP
            - name: udp-in 
              containerPort: 7777
              protocol: UDP

          resources:
            limits:
              cpu: 1024m
              memory: 512Mi
            requests:
              cpu: 100m
              memory: 100Mi

          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /ping
              port: 9000
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 2
          readinessProbe:
            failureThreshold: 1
            httpGet:
              path: /ping
              port: 9000
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 2