Traefik + Kubernetes + new entryPoint = hair loss

Traefik Traefik v2 (latest)
1

I'm trying to get traefik to use new entryPoints and failing miserably. I'm trying to get it to understand amqp and mqtt.

I added the options to the command and even to the container ports (in the deployment)

       - args: 
         - --global.checknewversion 
         . . .
         - --entrypoints.mqtt.address=:1833/tcp
         - --entrypoints.amqp.address=:5672/tcp
         . . .
      name: traefik
      ports:
        . . .
        - containerPort: 1833
          name: mqtt
          protocol: TCP
        - containerPort: 5672
          name: amqp
          protocol: TCP
        . . .

But I still get:

illegal rune literal" entryPointName=amqp routerName=default-rabbitmq-amqp-4fd825341ce65b095476@kubernetescrd

Defined as:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  name: rabbitmq-amqp
  namespace: default
spec:
  entryPoints:
    - amqp
  routes:
    - match: Host('rabbitmq.home.svc')
      services:
        - name: rabbitmq
          port: 5672

What trick or voodoo am I missing here to make traefik do my bidding?

2

Hi @Bill_Mair,
Thanks for your interest in Traefik!

The rule (Match) must use backtick instead of single quotes, see the docs for more info.

If you still encounter the problem, could you please post your full configuration and your DEBUG logs?

3

Hi @svx,

I went through everything again, updated to the latest version and realised my mistake.

I'm now using the latest helm chart and passing the following values:

    ports:
      mqtt:
        port: 1883
        protocol: TCP
        exposed: true
        exposedPort: 1883
      amqp:
        port: 5672
        protocol: TCP
        exposed: true
        exposedPort: 5672
    ingressRoute:
      dashboard:
        enabled: false

I have a different problem when enabling the dashboard but I'll start another thread so that the issues don't get mixed.

The message is not so obvious, I mistook the entryPointName for the issue, which was really, as you correctly identified, me using single quoutes and not backticks. Then it told me that Host is only valid for HTTP requests with the ALPN example. So nothing is complaining any more.

Current definition:

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  name: rabbitmq-ingress-mqtt
  namespace: default
spec:
  entryPoints:
    - mqtt
  routes:
    - match: ALPN(`mqtt`)
      services:
        - name: rabbitmq
          port: 1883
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  name: rabbitmq-ingress-amqp
  namespace: default
spec:
  entryPoints:
    - amqp
  routes:
    - match: ALPN(`amqp`)
      services:
        - name: rabbitmq
          port: 5672
---

I still can't access the service though. The MQTT and AMQP ports (1883 and 5672) aren't opened by traefik.

        exposed: true
        exposedPort: 5672

I thought that the exposed and exposedPort entries made the service open up the ports but they aren't.

So I'm still missing something, how do I tell Traefik to open the ports on the LoadBalancer and forward the connections as defined?

4

Hi @Bill_Mair,
could you please post your full configuration like the deployment, service, and the ingress description?

And, if you adjusted the values of the Helm chart, these, too?

5

Sure, here are the YAMLs:

I use Flux CD to deploy helm charts, so values are defined as a part of the release specification:

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
spec:
  values:
    . . .

Traefik - HelmRepository:

---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  name: traefik
  namespace: flux-system
spec:
  interval: 30m0s
  url: https://traefik.github.io/charts
...

Traefik - Certificate, HelmRelease:

As I indicated before, the dashboard at traefik.home.svc isn't working either :frowning:

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: traefik-dashboard-cert
  namespace: flux-system
spec:
  secretName: traefik-dashboard-cert
  dnsNames:
    - traefik.home.svc
  issuerRef:
    kind: ClusterIssuer
    group: cert-manager.io
    name: themairs-production-ca-2022
  usages:
    - digital signature
    - key encipherment
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: traefik
  namespace: flux-system
spec:
  targetNamespace: kube-system
  releaseName: traefik
  interval: 60m0s
  chart:
    spec:
      chart: traefik
      version: 21.1.0
      sourceRef:
        kind: HelmRepository
        name: traefik
  values:
    ports:
      mqtt:
        port: 1883
        protocol: TCP
        exposed: true
        exposedPort: 1883
      amqp:
        port: 5672
        protocol: TCP
        exposed: true
        exposedPort: 5672
    logs:
      general:
        level: INFO
        format: json
    ingressRoute:
      dashboard:
        enabled: true
        matchRule: Host(`traefik.home.svc`) && ( PathPrefix(`/#`) || PathPrefix(`/dashboard`) || PathPrefix(`/api`) )
        tls:
          secretName: traefik-dashboard-cert
...

RabbitMQ configuration (using their Operator)

Here is where I defined the IngressRouteTCP definitions.

---
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: rabbitmq
  namespace: default
spec:
  replicas: 3
  persistence:
    storageClassName: cephfs
    storage: 20Gi
  resources:
    requests:
      cpu: 25m
      memory: 120m
    limits:
      cpu: 1000m
      memory: 2Gi
  rabbitmq:
    additionalPlugins:
      - rabbitmq_mqtt
    additionalConfig: |
      mqtt.allow_anonymous  = true
      mqtt.default_user     = mqtt_default_user
      mqtt.default_pass     = mqtt_default_user
      mqtt.vhost            = /
      mqtt.exchange         = amq.topic
      mqtt.subscription_ttl = 86400000
      mqtt.prefetch         = 10
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  name: rabbitmq-ingress-mqtt
  namespace: default
spec:
  entryPoints:
    - mqtt
  routes:
    - match: ALPN(`mqtt`) || ALPN(`h2`) || ALPN(`http/1.1`)
      services:
        - name: rabbitmq
          port: 1883
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  name: rabbitmq-ingress-amqp
  namespace: default
spec:
  entryPoints:
    - amqp
  routes:
    - match: ALPN(`amqp`) || ALPN(`h2`) || ALPN(`http/1.1`)
      services:
        - name: rabbitmq
          port: 5672
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: rabbitmq-management-redirect
  namespace: default
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: web
    #                                             <namespace>-<middleware-name>@kubernetescrd
    traefik.ingress.kubernetes.io/router.middlewares: default-redirect-https@kubernetescrd
spec:
  rules:
    - host: rabbitmq.home.svc
      http:
        paths:
          - path: /
            pathType: ImplementationSpecific
            backend:
              service:
                name: rabbitmq
                port:
                  number: 15672
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: rabbitmq-management
  namespace: default
  annotations:
    cert-manager.io/cluster-issuer: themairs-production-ca-2022
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.tls: "true"
spec:
  rules:
    - host: rabbitmq.home.svc
      http:
        paths:
          - path: /
            pathType: ImplementationSpecific
            backend:
              service:
                name: rabbitmq
                port:
                  number: 15672
  tls: # < placing a host in the TLS config will determine what ends up in the cert's subjectAltNames
    - hosts:
        - rabbitmq.home.svc
      secretName: rabbitmq-cert  # < cert-manager will store the created certificate in this secret.
...

Connection from another pod in the cluster (same namespace):

root@test-service-c7bd7b55b-dhvr9:/# curl -v mqtt://rabbitmq/some/topic
*   Trying 10.43.47.82:1883...
* Connected to rabbitmq (10.43.47.82) port 1883 (#0)
* Using client id 'curl02fdffdc'
> MQTT<
       curl02fdffdcmqtt_doing: state [0]
<  mqtt_doing: state [2]
< �
> some/topicmqtt_doing: state [0]
< �mqtt_doing: state [3]
< mqtt_doing: state [0]
* mqtt_doing: state [0]

Connection from external system:

bill@home:/$ curl -v mqtt://rabbitmq.home.svc/some/topic
*   Trying 192.168.76.237:1883...
* Connected to rabbitmq.home.svc (192.168.76.237) port 1883 (#0)
* Using client id 'curl4b2e325d'
> MQTT<
       curl4b2e325dmqtt_doing: state [0]
* mqtt_doing: state [0]
* mqtt_doing: state [0]
* mqtt_doing: state [0]
6

Can you try it with

- match: HostSNI(`rabbitmq.home.svc`)

And could you post the DEBUG logs of Traefik ?

9

Here the get all:

$ kubectl get svc -n traefik 
NAME      TYPE           CLUSTER-IP     EXTERNAL-IP                                                                                 PORT(S)                      AGE
traefik   LoadBalancer   10.43.225.37   192.168.76.231,192.168.76.232,192.168.76.233,192.168.76.237,192.168.76.238,192.168.76.239   80:30295/TCP,443:30518/TCP   11m
[bm@leviathan flux-prod ]$ kubectl get all -n traefik 
NAME                           READY   STATUS    RESTARTS   AGE
pod/traefik-5c9bf6fc7d-8spnv   1/1     Running   0          11m

NAME              TYPE           CLUSTER-IP     EXTERNAL-IP                                                                                 PORT(S)                      AGE
service/traefik   LoadBalancer   10.43.225.37   192.168.76.231,192.168.76.232,192.168.76.233,192.168.76.237,192.168.76.238,192.168.76.239   80:30295/TCP,443:30518/TCP   11m

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/traefik   1/1     1            1           11m

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/traefik-5c9bf6fc7d   1         1         1       11m

I think that the reason there is nothing happening is because there service isn't even listening to the defined service ports.

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/traefik LoadBalancer 10.43.225.37 [Node IPs] 80:30295/TCP,443:30518/TCP 11m

The ports for MQTT and AMQP aren't opened on the LoadBalancer.

Curl fails on all load balancer addresses:

$ curl -v mqtt://rabbitmq.home.svc/some/topic
*   Trying 192.168.76.239:1883...
* connect to 192.168.76.239 port 1883 failed: Connection refused
*   Trying 192.168.76.231:1883...
* connect to 192.168.76.231 port 1883 failed: Connection refused
*   Trying 192.168.76.232:1883...
* connect to 192.168.76.232 port 1883 failed: Connection refused
*   Trying 192.168.76.233:1883...
* connect to 192.168.76.233 port 1883 failed: Connection refused
*   Trying 192.168.76.237:1883...
* connect to 192.168.76.237 port 1883 failed: Connection refused
*   Trying 192.168.76.238:1883...
* connect to 192.168.76.238 port 1883 failed: Connection refused
* Failed to connect to rabbitmq.home.svc port 1883 after 4 ms: Couldn't connect to server
* Closing connection 0
curl: (7) Failed to connect to rabbitmq.home.svc port 1883 after 4 ms: Couldn't connect to server

The pod appears to be correctly configured though:

$ kubectl describe pod -n traefik traefik-5c9bf6fc7d-8spnv 
Name:             traefik-5c9bf6fc7d-8spnv
Namespace:        traefik
Priority:         0
Service Account:  traefik
Node:             k3s-a3/192.168.76.233
Start Time:       Tue, 21 Feb 2023 18:54:29 +0100
Labels:           app.kubernetes.io/instance=traefik-traefik
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=traefik
                  helm.sh/chart=traefik-21.1.0
                  pod-template-hash=5c9bf6fc7d
Annotations:      prometheus.io/path: /metrics
                  prometheus.io/port: 9100
                  prometheus.io/scrape: true
Status:           Running
IP:               10.42.3.254
IPs:
  IP:           10.42.3.254
Controlled By:  ReplicaSet/traefik-5c9bf6fc7d
Containers:
  traefik:
    Container ID:  containerd://3eff296f2e6343d32fab5a9ff22bb91ce1757d28db2115a55c4b82d8bb63a86a
    Image:         traefik:v2.9.7
    Image ID:      docker.io/library/traefik@sha256:7a4fb968173b583bcf4aae0fea180cd6cd95001be686494d339da35809897cc2
    Ports:         5672/TCP, 9100/TCP, 1883/TCP, 9000/TCP, 8000/TCP, 8443/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP
    Args:
      --global.checknewversion
      --global.sendanonymoususage
      --entrypoints.amqp.address=:5672/tcp
      --entrypoints.metrics.address=:9100/tcp
      --entrypoints.mqtt.address=:1883/tcp
      --entrypoints.traefik.address=:9000/tcp
      --entrypoints.web.address=:8000/tcp
      --entrypoints.websecure.address=:8443/tcp
      --api.dashboard=true
      --ping=true
      --metrics.prometheus=true
      --metrics.prometheus.entrypoint=metrics
      --providers.kubernetescrd
      --providers.kubernetesingress
      --entrypoints.websecure.http.tls=true
      --log.format=json
      --log.level=DEBUG
    State:          Running
      Started:      Tue, 21 Feb 2023 18:54:31 +0100
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:9000/ping delay=2s timeout=2s period=10s #success=1 #failure=3
    Readiness:      http-get http://:9000/ping delay=2s timeout=2s period=10s #success=1 #failure=1
    Environment:    <none>
    Mounts:
      /data from data (rw)
      /tmp from tmp (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mrmtw (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-mrmtw:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  17m   default-scheduler  Successfully assigned traefik/traefik-5c9bf6fc7d-8spnv to k3s-a3
  Normal  Pulled     17m   kubelet            Container image "traefik:v2.9.7" already present on machine
  Normal  Created    17m   kubelet            Created container traefik
  Normal  Started    17m   kubelet            Started container traefik
10

I reactivated the redirect middlewares and the IngressRouteTCPs:

2023-02-21T19:19:21+01:00	{"entryPointName":"mqtt","level":"debug","msg":"Adding route for \"ALPN(`mqtt`) || ALPN(`h2`) || ALPN(`http/1.1`)\"","routerName":"default-rabbitmq-ingress-mqtt-15f8656c75a0efffd200@kubernetescrd","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"mqtt","level":"debug","msg":"Creating TCP server 2 at 10.42.3.249:1883","routerName":"default-rabbitmq-ingress-mqtt-15f8656c75a0efffd200@kubernetescrd","serverName":2,"serviceName":"default-rabbitmq-ingress-mqtt-15f8656c75a0efffd200","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"mqtt","level":"debug","msg":"Creating TCP server 1 at 10.42.3.251:1883","routerName":"default-rabbitmq-ingress-mqtt-15f8656c75a0efffd200@kubernetescrd","serverName":1,"serviceName":"default-rabbitmq-ingress-mqtt-15f8656c75a0efffd200","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"mqtt","level":"debug","msg":"Creating TCP server 0 at 10.42.3.248:1883","routerName":"default-rabbitmq-ingress-mqtt-15f8656c75a0efffd200@kubernetescrd","serverName":0,"serviceName":"default-rabbitmq-ingress-mqtt-15f8656c75a0efffd200","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"amqp","level":"debug","msg":"Adding route for \"ALPN(`amqp`) || ALPN(`h2`) || ALPN(`http/1.1`)\"","routerName":"default-rabbitmq-ingress-amqp-61fa68f0d81a39912e6d@kubernetescrd","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"amqp","level":"debug","msg":"Creating TCP server 2 at 10.42.3.248:5672","routerName":"default-rabbitmq-ingress-amqp-61fa68f0d81a39912e6d@kubernetescrd","serverName":2,"serviceName":"default-rabbitmq-ingress-amqp-61fa68f0d81a39912e6d","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"amqp","level":"debug","msg":"Creating TCP server 1 at 10.42.3.249:5672","routerName":"default-rabbitmq-ingress-amqp-61fa68f0d81a39912e6d@kubernetescrd","serverName":1,"serviceName":"default-rabbitmq-ingress-amqp-61fa68f0d81a39912e6d","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"amqp","level":"debug","msg":"Creating TCP server 0 at 10.42.3.251:5672","routerName":"default-rabbitmq-ingress-amqp-61fa68f0d81a39912e6d@kubernetescrd","serverName":0,"serviceName":"default-rabbitmq-ingress-amqp-61fa68f0d81a39912e6d","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"websecure","level":"debug","msg":"Adding route for prometheus.home.svc with TLS options default","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"websecure","level":"debug","msg":"Adding route for test-service.home.svc with TLS options default","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"websecure","level":"debug","msg":"Adding route for grafana.home.svc with TLS options default","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"websecure","level":"debug","msg":"Adding route for rabbitmq.home.svc with TLS options default","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"websecure","level":"debug","msg":"Adding route for test-service2.home.svc with TLS options default","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"traefik","level":"debug","msg":"Adding route for traefik.home.svc with TLS options default","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"mqtt","level":"debug","middlewareName":"metrics-entrypoint","middlewareType":"Metrics","msg":"Creating middleware","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"metrics","level":"debug","middlewareName":"metrics-entrypoint","middlewareType":"Metrics","msg":"Creating middleware","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"amqp","level":"debug","middlewareName":"metrics-entrypoint","middlewareType":"Metrics","msg":"Creating middleware","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"websecure","level":"debug","middlewareName":"metrics-entrypoint","middlewareType":"Metrics","msg":"Creating middleware","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","middlewareName":"metrics-entrypoint","middlewareType":"Metrics","msg":"Creating middleware","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"traefik","level":"debug","middlewareName":"metrics-entrypoint","middlewareType":"Metrics","msg":"Creating middleware","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"traefik","level":"debug","middlewareName":"traefik-internal-recovery","middlewareType":"Recovery","msg":"Creating middleware","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"traefik","level":"debug","middlewareName":"tracing","middlewareType":"TracingForwarder","msg":"Added outgoing tracing middleware api@internal","routerName":"traefik-traefik-dashboard-fa12567b1c53b589ec48@kubernetescrd","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"websecure","level":"debug","middlewareName":"traefik-internal-recovery","middlewareType":"Recovery","msg":"Creating middleware","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"websecure","level":"debug","middlewareName":"tracing","middlewareType":"TracingForwarder","msg":"Added outgoing tracing middleware default-test-service-http","routerName":"default-test-service-test-service-home-svc@kubernetes","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"level":"debug","msg":"Still UP, no need to propagate","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"level":"debug","msg":"child http://10.42.5.181:80 now UP","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"websecure","level":"debug","msg":"Creating server 1 http://10.42.5.181:80","routerName":"default-test-service-test-service-home-svc@kubernetes","serverName":1,"serviceName":"default-test-service-http","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"level":"debug","msg":"Propagating new UP status","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"level":"debug","msg":"child http://10.42.4.162:80 now UP","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"websecure","level":"debug","msg":"Creating server 0 http://10.42.4.162:80","routerName":"default-test-service-test-service-home-svc@kubernetes","serverName":0,"serviceName":"default-test-service-http","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"websecure","level":"debug","msg":"Creating load-balancer","routerName":"default-test-service-test-service-home-svc@kubernetes","serviceName":"default-test-service-http","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"websecure","level":"debug","middlewareName":"metrics-service","middlewareType":"Metrics","msg":"Creating middleware","routerName":"default-test-service-test-service-home-svc@kubernetes","serviceName":"default-test-service-http","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"websecure","level":"debug","middlewareName":"pipelining","middlewareType":"Pipelining","msg":"Creating middleware","routerName":"default-test-service-test-service-home-svc@kubernetes","serviceName":"default-test-service-http","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"websecure","level":"debug","middlewareName":"tracing","middlewareType":"TracingForwarder","msg":"Added outgoing tracing middleware monitoring-kube-prometheus-stack-grafana-80","routerName":"monitoring-grafana-grafana-home-svc@kubernetes","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"level":"debug","msg":"Propagating new UP status","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"level":"debug","msg":"child http://10.42.4.161:3000 now UP","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"websecure","level":"debug","msg":"Creating server 0 http://10.42.4.161:3000","routerName":"monitoring-grafana-grafana-home-svc@kubernetes","serverName":0,"serviceName":"monitoring-kube-prometheus-stack-grafana-80","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"websecure","level":"debug","msg":"Creating load-balancer","routerName":"monitoring-grafana-grafana-home-svc@kubernetes","serviceName":"monitoring-kube-prometheus-stack-grafana-80","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"websecure","level":"debug","middlewareName":"metrics-service","middlewareType":"Metrics","msg":"Creating middleware","routerName":"monitoring-grafana-grafana-home-svc@kubernetes","serviceName":"monitoring-kube-prometheus-stack-grafana-80","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"websecure","level":"debug","middlewareName":"pipelining","middlewareType":"Pipelining","msg":"Creating middleware","routerName":"monitoring-grafana-grafana-home-svc@kubernetes","serviceName":"monitoring-kube-prometheus-stack-grafana-80","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"websecure","level":"debug","middlewareName":"tracing","middlewareType":"TracingForwarder","msg":"Added outgoing tracing middleware monitoring-kube-prometheus-stack-prometheus-9090","routerName":"monitoring-prometheus-prometheus-home-svc@kubernetes","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"level":"debug","msg":"Propagating new UP status","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"level":"debug","msg":"child http://10.42.4.172:9090 now UP","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"websecure","level":"debug","msg":"Creating server 0 http://10.42.4.172:9090","routerName":"monitoring-prometheus-prometheus-home-svc@kubernetes","serverName":0,"serviceName":"monitoring-kube-prometheus-stack-prometheus-9090","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"websecure","level":"debug","msg":"Creating load-balancer","routerName":"monitoring-prometheus-prometheus-home-svc@kubernetes","serviceName":"monitoring-kube-prometheus-stack-prometheus-9090","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"websecure","level":"debug","middlewareName":"metrics-service","middlewareType":"Metrics","msg":"Creating middleware","routerName":"monitoring-prometheus-prometheus-home-svc@kubernetes","serviceName":"monitoring-kube-prometheus-stack-prometheus-9090","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"websecure","level":"debug","middlewareName":"pipelining","middlewareType":"Pipelining","msg":"Creating middleware","routerName":"monitoring-prometheus-prometheus-home-svc@kubernetes","serviceName":"monitoring-kube-prometheus-stack-prometheus-9090","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"websecure","level":"debug","middlewareName":"tracing","middlewareType":"TracingForwarder","msg":"Added outgoing tracing middleware default-rabbitmq-15672","routerName":"default-rabbitmq-management-rabbitmq-home-svc@kubernetes","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"level":"debug","msg":"Still UP, no need to propagate","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"level":"debug","msg":"child http://10.42.3.249:15672 now UP","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"websecure","level":"debug","msg":"Creating server 2 http://10.42.3.249:15672","routerName":"default-rabbitmq-management-rabbitmq-home-svc@kubernetes","serverName":2,"serviceName":"default-rabbitmq-15672","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"level":"debug","msg":"Still UP, no need to propagate","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"level":"debug","msg":"child http://10.42.3.248:15672 now UP","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"websecure","level":"debug","msg":"Creating server 1 http://10.42.3.248:15672","routerName":"default-rabbitmq-management-rabbitmq-home-svc@kubernetes","serverName":1,"serviceName":"default-rabbitmq-15672","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"level":"debug","msg":"Propagating new UP status","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"level":"debug","msg":"child http://10.42.3.251:15672 now UP","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"websecure","level":"debug","msg":"Creating server 0 http://10.42.3.251:15672","routerName":"default-rabbitmq-management-rabbitmq-home-svc@kubernetes","serverName":0,"serviceName":"default-rabbitmq-15672","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"websecure","level":"debug","msg":"Creating load-balancer","routerName":"default-rabbitmq-management-rabbitmq-home-svc@kubernetes","serviceName":"default-rabbitmq-15672","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"websecure","level":"debug","middlewareName":"metrics-service","middlewareType":"Metrics","msg":"Creating middleware","routerName":"default-rabbitmq-management-rabbitmq-home-svc@kubernetes","serviceName":"default-rabbitmq-15672","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"websecure","level":"debug","middlewareName":"pipelining","middlewareType":"Pipelining","msg":"Creating middleware","routerName":"default-rabbitmq-management-rabbitmq-home-svc@kubernetes","serviceName":"default-rabbitmq-15672","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"websecure","level":"debug","middlewareName":"tracing","middlewareType":"TracingForwarder","msg":"Added outgoing tracing middleware test-test-service-http","routerName":"test-test-service2-test-service2-home-svc@kubernetes","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"level":"debug","msg":"Still UP, no need to propagate","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"level":"debug","msg":"child http://10.42.3.252:80 now UP","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"websecure","level":"debug","msg":"Creating server 1 http://10.42.3.252:80","routerName":"test-test-service2-test-service2-home-svc@kubernetes","serverName":1,"serviceName":"test-test-service-http","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"level":"debug","msg":"Propagating new UP status","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"level":"debug","msg":"child http://10.42.5.191:80 now UP","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"websecure","level":"debug","msg":"Creating server 0 http://10.42.5.191:80","routerName":"test-test-service2-test-service2-home-svc@kubernetes","serverName":0,"serviceName":"test-test-service-http","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"websecure","level":"debug","msg":"Creating load-balancer","routerName":"test-test-service2-test-service2-home-svc@kubernetes","serviceName":"test-test-service-http","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"websecure","level":"debug","middlewareName":"metrics-service","middlewareType":"Metrics","msg":"Creating middleware","routerName":"test-test-service2-test-service2-home-svc@kubernetes","serviceName":"test-test-service-http","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"websecure","level":"debug","middlewareName":"pipelining","middlewareType":"Pipelining","msg":"Creating middleware","routerName":"test-test-service2-test-service2-home-svc@kubernetes","serviceName":"test-test-service-http","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"mqtt","level":"debug","middlewareName":"metrics-entrypoint","middlewareType":"Metrics","msg":"Creating middleware","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"metrics","level":"debug","middlewareName":"metrics-entrypoint","middlewareType":"Metrics","msg":"Creating middleware","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"amqp","level":"debug","middlewareName":"metrics-entrypoint","middlewareType":"Metrics","msg":"Creating middleware","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"websecure","level":"debug","middlewareName":"metrics-entrypoint","middlewareType":"Metrics","msg":"Creating middleware","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","middlewareName":"metrics-entrypoint","middlewareType":"Metrics","msg":"Creating middleware","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"traefik","level":"debug","middlewareName":"metrics-entrypoint","middlewareType":"Metrics","msg":"Creating middleware","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"traefik","level":"debug","middlewareName":"traefik-internal-recovery","middlewareType":"Recovery","msg":"Creating middleware","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"traefik","level":"debug","middlewareName":"tracing","middlewareType":"TracingForwarder","msg":"Added outgoing tracing middleware ping@internal","routerName":"ping@internal","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","middlewareName":"traefik-internal-recovery","middlewareType":"Recovery","msg":"Creating middleware","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","middlewareName":"default-redirect-https@kubernetescrd","middlewareType":"RedirectScheme","msg":"Setting up redirection to https ","routerName":"default-rabbitmq-management-redirect-rabbitmq-home-svc@kubernetes","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","middlewareName":"default-redirect-https@kubernetescrd","middlewareType":"RedirectScheme","msg":"Creating middleware","routerName":"default-rabbitmq-management-redirect-rabbitmq-home-svc@kubernetes","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","middlewareName":"tracing","middlewareType":"TracingForwarder","msg":"Added outgoing tracing middleware default-rabbitmq-15672","routerName":"default-rabbitmq-management-redirect-rabbitmq-home-svc@kubernetes","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"level":"debug","msg":"Still UP, no need to propagate","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"level":"debug","msg":"child http://10.42.3.248:15672 now UP","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","msg":"Creating server 2 http://10.42.3.248:15672","routerName":"default-rabbitmq-management-redirect-rabbitmq-home-svc@kubernetes","serverName":2,"serviceName":"default-rabbitmq-15672","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"level":"debug","msg":"Still UP, no need to propagate","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"level":"debug","msg":"child http://10.42.3.249:15672 now UP","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","msg":"Creating server 1 http://10.42.3.249:15672","routerName":"default-rabbitmq-management-redirect-rabbitmq-home-svc@kubernetes","serverName":1,"serviceName":"default-rabbitmq-15672","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"level":"debug","msg":"Propagating new UP status","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"level":"debug","msg":"child http://10.42.3.251:15672 now UP","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","msg":"Creating server 0 http://10.42.3.251:15672","routerName":"default-rabbitmq-management-redirect-rabbitmq-home-svc@kubernetes","serverName":0,"serviceName":"default-rabbitmq-15672","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","msg":"Creating load-balancer","routerName":"default-rabbitmq-management-redirect-rabbitmq-home-svc@kubernetes","serviceName":"default-rabbitmq-15672","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","middlewareName":"metrics-service","middlewareType":"Metrics","msg":"Creating middleware","routerName":"default-rabbitmq-management-redirect-rabbitmq-home-svc@kubernetes","serviceName":"default-rabbitmq-15672","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","middlewareName":"pipelining","middlewareType":"Pipelining","msg":"Creating middleware","routerName":"default-rabbitmq-management-redirect-rabbitmq-home-svc@kubernetes","serviceName":"default-rabbitmq-15672","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","middlewareName":"default-redirect-https@kubernetescrd","middlewareType":"RedirectScheme","msg":"Setting up redirection to https ","routerName":"test-test-service2-redirect-test-service2-home-svc@kubernetes","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","middlewareName":"default-redirect-https@kubernetescrd","middlewareType":"RedirectScheme","msg":"Creating middleware","routerName":"test-test-service2-redirect-test-service2-home-svc@kubernetes","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","middlewareName":"tracing","middlewareType":"TracingForwarder","msg":"Added outgoing tracing middleware test-test-service-http","routerName":"test-test-service2-redirect-test-service2-home-svc@kubernetes","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"level":"debug","msg":"Still UP, no need to propagate","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"level":"debug","msg":"child http://10.42.5.191:80 now UP","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","msg":"Creating server 1 http://10.42.5.191:80","routerName":"test-test-service2-redirect-test-service2-home-svc@kubernetes","serverName":1,"serviceName":"test-test-service-http","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"level":"debug","msg":"Propagating new UP status","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"level":"debug","msg":"child http://10.42.3.252:80 now UP","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","msg":"Creating server 0 http://10.42.3.252:80","routerName":"test-test-service2-redirect-test-service2-home-svc@kubernetes","serverName":0,"serviceName":"test-test-service-http","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","msg":"Creating load-balancer","routerName":"test-test-service2-redirect-test-service2-home-svc@kubernetes","serviceName":"test-test-service-http","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","middlewareName":"metrics-service","middlewareType":"Metrics","msg":"Creating middleware","routerName":"test-test-service2-redirect-test-service2-home-svc@kubernetes","serviceName":"test-test-service-http","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","middlewareName":"pipelining","middlewareType":"Pipelining","msg":"Creating middleware","routerName":"test-test-service2-redirect-test-service2-home-svc@kubernetes","serviceName":"test-test-service-http","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","middlewareName":"monitoring-redirect-https@kubernetescrd","middlewareType":"RedirectScheme","msg":"Setting up redirection to https ","routerName":"monitoring-prometheus-redirect-prometheus-home-svc@kubernetes","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","middlewareName":"monitoring-redirect-https@kubernetescrd","middlewareType":"RedirectScheme","msg":"Creating middleware","routerName":"monitoring-prometheus-redirect-prometheus-home-svc@kubernetes","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","middlewareName":"tracing","middlewareType":"TracingForwarder","msg":"Added outgoing tracing middleware monitoring-kube-prometheus-stack-prometheus-9090","routerName":"monitoring-prometheus-redirect-prometheus-home-svc@kubernetes","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"level":"debug","msg":"Propagating new UP status","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"level":"debug","msg":"child http://10.42.4.172:9090 now UP","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","msg":"Creating server 0 http://10.42.4.172:9090","routerName":"monitoring-prometheus-redirect-prometheus-home-svc@kubernetes","serverName":0,"serviceName":"monitoring-kube-prometheus-stack-prometheus-9090","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","msg":"Creating load-balancer","routerName":"monitoring-prometheus-redirect-prometheus-home-svc@kubernetes","serviceName":"monitoring-kube-prometheus-stack-prometheus-9090","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","middlewareName":"metrics-service","middlewareType":"Metrics","msg":"Creating middleware","routerName":"monitoring-prometheus-redirect-prometheus-home-svc@kubernetes","serviceName":"monitoring-kube-prometheus-stack-prometheus-9090","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","middlewareName":"pipelining","middlewareType":"Pipelining","msg":"Creating middleware","routerName":"monitoring-prometheus-redirect-prometheus-home-svc@kubernetes","serviceName":"monitoring-kube-prometheus-stack-prometheus-9090","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","middlewareName":"default-redirect-https@kubernetescrd","middlewareType":"RedirectScheme","msg":"Setting up redirection to https ","routerName":"default-test-service-redirect-test-service-home-svc@kubernetes","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","middlewareName":"default-redirect-https@kubernetescrd","middlewareType":"RedirectScheme","msg":"Creating middleware","routerName":"default-test-service-redirect-test-service-home-svc@kubernetes","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","middlewareName":"tracing","middlewareType":"TracingForwarder","msg":"Added outgoing tracing middleware default-test-service-http","routerName":"default-test-service-redirect-test-service-home-svc@kubernetes","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"level":"debug","msg":"Still UP, no need to propagate","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"level":"debug","msg":"child http://10.42.4.162:80 now UP","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","msg":"Creating server 1 http://10.42.4.162:80","routerName":"default-test-service-redirect-test-service-home-svc@kubernetes","serverName":1,"serviceName":"default-test-service-http","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"level":"debug","msg":"Propagating new UP status","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"level":"debug","msg":"child http://10.42.5.181:80 now UP","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","msg":"Creating server 0 http://10.42.5.181:80","routerName":"default-test-service-redirect-test-service-home-svc@kubernetes","serverName":0,"serviceName":"default-test-service-http","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","msg":"Creating load-balancer","routerName":"default-test-service-redirect-test-service-home-svc@kubernetes","serviceName":"default-test-service-http","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","middlewareName":"metrics-service","middlewareType":"Metrics","msg":"Creating middleware","routerName":"default-test-service-redirect-test-service-home-svc@kubernetes","serviceName":"default-test-service-http","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","middlewareName":"pipelining","middlewareType":"Pipelining","msg":"Creating middleware","routerName":"default-test-service-redirect-test-service-home-svc@kubernetes","serviceName":"default-test-service-http","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","middlewareName":"monitoring-redirect-https@kubernetescrd","middlewareType":"RedirectScheme","msg":"Setting up redirection to https ","routerName":"monitoring-grafana-redirect-grafana-home-svc@kubernetes","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","middlewareName":"monitoring-redirect-https@kubernetescrd","middlewareType":"RedirectScheme","msg":"Creating middleware","routerName":"monitoring-grafana-redirect-grafana-home-svc@kubernetes","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","middlewareName":"tracing","middlewareType":"TracingForwarder","msg":"Added outgoing tracing middleware monitoring-kube-prometheus-stack-grafana-80","routerName":"monitoring-grafana-redirect-grafana-home-svc@kubernetes","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"level":"debug","msg":"Propagating new UP status","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"level":"debug","msg":"child http://10.42.4.161:3000 now UP","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","msg":"Creating server 0 http://10.42.4.161:3000","routerName":"monitoring-grafana-redirect-grafana-home-svc@kubernetes","serverName":0,"serviceName":"monitoring-kube-prometheus-stack-grafana-80","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","msg":"Creating load-balancer","routerName":"monitoring-grafana-redirect-grafana-home-svc@kubernetes","serviceName":"monitoring-kube-prometheus-stack-grafana-80","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","middlewareName":"metrics-service","middlewareType":"Metrics","msg":"Creating middleware","routerName":"monitoring-grafana-redirect-grafana-home-svc@kubernetes","serviceName":"monitoring-kube-prometheus-stack-grafana-80","time":"2023-02-21T18:19:21Z"}
2023-02-21T19:19:21+01:00	{"entryPointName":"web","level":"debug","middlewareName":"pipelining","middlewareType":"Pipelining","msg":"Creating middleware","routerName":"monitoring-grafana-redirect-grafana-home-svc@kubernetes","serviceName":"monitoring-kube-prometheus-stack-grafana-80","time":"2023-02-21T18:19:21Z"}
11

The MQTT and AMQP ports are still closed on the load balancer:

$ kubectl get all -n traefik 
NAME                           READY   STATUS    RESTARTS   AGE
pod/traefik-5c9bf6fc7d-8spnv   1/1     Running   0          38m

NAME              TYPE           CLUSTER-IP     EXTERNAL-IP                                                                                 PORT(S)                      AGE
service/traefik   LoadBalancer   10.43.225.37   192.168.76.231,192.168.76.232,192.168.76.233,192.168.76.237,192.168.76.238,192.168.76.239   80:30295/TCP,443:30518/TCP   38m

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/traefik   1/1     1            1           38m

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/traefik-5c9bf6fc7d   1         1         1       38m
12

Fixed a typo in the values (expose not exposed):

  values:
    ports:
      mqtt:
        port: 1883
        protocol: TCP
        expose: true
        exposedPort: 1883
      amqp:
        port: 5672
        protocol: TCP
        expose: true
        exposedPort: 5672

Now the ports are open on the LB:

$ kubectl get all -n traefik 
NAME                           READY   STATUS    RESTARTS   AGE
pod/traefik-5c9bf6fc7d-8spnv   1/1     Running   0          47m

NAME              TYPE           CLUSTER-IP     EXTERNAL-IP                                                                                 PORT(S)                                                    AGE
service/traefik   LoadBalancer   10.43.225.37   192.168.76.231,192.168.76.232,192.168.76.233,192.168.76.237,192.168.76.238,192.168.76.239   5672:31501/TCP,1883:32321/TCP,80:30295/TCP,443:30518/TCP   47m

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/traefik   1/1     1            1           47m

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/traefik-5c9bf6fc7d   1         1         1       47m

Curl is back to how it was before, not getting through:

$ curl -v mqtt://rabbitmq.home.svc/some/topic
*   Trying 192.168.76.233:1883...
* Connected to rabbitmq.home.svc (192.168.76.233) port 1883 (#0)
* Using client id 'curl124dc5aa'
> MQTT<
       curl124dc5aamqtt_doing: state [0]
* mqtt_doing: state [0]
* mqtt_doing: state [0]

And I don't see anything in the logs.

13

I also tried redefining the AMQP and MQTT defintions, there is still nothing getting through.

2023-02-21T19:58:24+01:00	{"entryPointName":"mqtt","level":"debug","msg":"Adding route for \"ALPN(`mqtt`) || ALPN(`h2`) || ALPN(`http/1.1`)\"","routerName":"default-rabbitmq-ingress-mqtt-15f8656c75a0efffd200@kubernetescrd","time":"2023-02-21T18:58:24Z"}
2023-02-21T19:58:24+01:00	{"entryPointName":"mqtt","level":"debug","msg":"Creating TCP server 2 at 10.42.3.249:1883","routerName":"default-rabbitmq-ingress-mqtt-15f8656c75a0efffd200@kubernetescrd","serverName":2,"serviceName":"default-rabbitmq-ingress-mqtt-15f8656c75a0efffd200","time":"2023-02-21T18:58:24Z"}
2023-02-21T19:58:24+01:00	{"entryPointName":"mqtt","level":"debug","msg":"Creating TCP server 1 at 10.42.3.251:1883","routerName":"default-rabbitmq-ingress-mqtt-15f8656c75a0efffd200@kubernetescrd","serverName":1,"serviceName":"default-rabbitmq-ingress-mqtt-15f8656c75a0efffd200","time":"2023-02-21T18:58:24Z"}
2023-02-21T19:58:24+01:00	{"entryPointName":"mqtt","level":"debug","msg":"Creating TCP server 0 at 10.42.3.248:1883","routerName":"default-rabbitmq-ingress-mqtt-15f8656c75a0efffd200@kubernetescrd","serverName":0,"serviceName":"default-rabbitmq-ingress-mqtt-15f8656c75a0efffd200","time":"2023-02-21T18:58:24Z"}
2023-02-21T19:58:24+01:00	{"entryPointName":"amqp","level":"debug","msg":"Adding route for \"ALPN(`amqp`) || ALPN(`h2`) || ALPN(`http/1.1`)\"","routerName":"default-rabbitmq-ingress-amqp-61fa68f0d81a39912e6d@kubernetescrd","time":"2023-02-21T18:58:24Z"}
2023-02-21T19:58:24+01:00	{"entryPointName":"amqp","level":"debug","msg":"Creating TCP server 2 at 10.42.3.248:5672","routerName":"default-rabbitmq-ingress-amqp-61fa68f0d81a39912e6d@kubernetescrd","serverName":2,"serviceName":"default-rabbitmq-ingress-amqp-61fa68f0d81a39912e6d","time":"2023-02-21T18:58:24Z"}
2023-02-21T19:58:24+01:00	{"entryPointName":"amqp","level":"debug","msg":"Creating TCP server 1 at 10.42.3.251:5672","routerName":"default-rabbitmq-ingress-amqp-61fa68f0d81a39912e6d@kubernetescrd","serverName":1,"serviceName":"default-rabbitmq-ingress-amqp-61fa68f0d81a39912e6d","time":"2023-02-21T18:58:24Z"}
2023-02-21T19:58:24+01:00	{"entryPointName":"amqp","level":"debug","msg":"Creating TCP server 0 at 10.42.3.249:5672","routerName":"default-rabbitmq-ingress-amqp-61fa68f0d81a39912e6d@kubernetescrd","serverName":0,"serviceName":"default-rabbitmq-ingress-amqp-61fa68f0d81a39912e6d","time":"2023-02-21T18:58:24Z"}

And I forgot to say, the dashboard still isn't working either.

14

Success! I stopped matching the APLN and switched to ClientIP, now it works.

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  name: rabbitmq-ingress-mqtt
  namespace: default
spec:
  entryPoints:
    - mqtt
  routes:
    - match: ClientIP(`192.168.0.0/16`,`10.0.0.0/8`)
      services:
        - name: rabbitmq
          port: 1883
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  name: rabbitmq-ingress-amqp
  namespace: default
spec:
  entryPoints:
    - amqp
  routes:
    - match: ClientIP(`192.168.0.0/16`,`10.0.0.0/8`)
      services:
        - name: rabbitmq
          port: 5672
15

The dashboard is working now too.
The default chart entry point is "traefik", I changed it to "websecure".
I also disabled the path matching and only looked at the host name.

    ingressRoute:
      dashboard:
        enabled: true
        entryPoints:
          - websecure
        matchRule: Host(`traefik.home.svc`)
        tls:
          secretName: traefik-dashboard-cert

Thanks @svx for your input. Without it I wouldn't have gone deeper. :metal:t2:

16

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.