Published services not working

Traefik Hub
1

I updated traefik to 2.9.4 and added:

+    additionalArguments:
+      - --experimental.hub
+      - --hub
     providers:
       kubernetesIngress:
+        allowExternalNameServices: true
         publishedService:
           enabled: true
+    hub:
+      enabled: true
+      tunnelPort: 9901

to my values.yaml

I also installed the hub agent to the same traefik ns with this:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: traefik

helmCharts:
- name: hub-agent
  includeCRDs: true
  namespace: traefik
  releaseName: traefik-hub
  version: 1.1.2
  repo: https://helm.traefik.io/hub
  valuesInline:
    tokenSecretRef:
       name: hub-token
       key: token
    tunnelDeployment:
      traefik:
        tunnelHost: traefik-hub.traefik.svc.cluster.local

and it seems fine, heres a snippet of logs from the tunnel pod

{"level":"debug","component":"tunnel-client","method":"GET","url":"https://platform.hub.traefik.io/agent/tunnel-endpoints","time":1669141841,"message":"Performing request"}
{"level":"error","error":"dial: websocket: bad handshake","time":1669141841,"message":"Launch tunnel"}
{"level":"error","error":"dial: websocket: bad handshake","time":1669141841,"message":"Launch tunnel"}
{"level":"debug","component":"tunnel-client","method":"GET","url":"https://platform.hub.traefik.io/agent/tunnel-endpoints","time":1669141901,"message":"Performing request"}

The agent shows green and all good in the dashboard

I then ran:

kubectl create deployment whoami --image=traefik/whoami
kubectl create service loadbalancer whoami --tcp=8080:80

from the docs here

found it in the service list and hit publish, got https://marked-donkey-f0x1p9.uzs7qbm5.traefikhub.io/

but that never loads anything and Im unsure how to go forward from here

2

Hello @myoung34,

Thank you for reporting this.
We found that the tunnels are indeed well established, and https://marked-donkey-f0x1p9.uzs7qbm5.traefikhub.io/ allows reaching Traefik that is serving this page (which is a placeholder page served by a "catchall" router when no other router can be matched).
In order to help you better, and understand what the problem is, we are missing a few pieces of information. Can you share with us the full values files for both the Hub agent and Traefik, and also, the hub agent controller's full log trace?

3

Here's my traefik values.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: traefik

helmCharts:
- name: traefik
  includeCRDs: true
  namespace: traefik
  releaseName: traefik
  version: 20.4.1
  repo: https://helm.traefik.io/traefik
  valuesInline:
    ingressRoute:
      dashboard:
        matchRule: "Host(`traefik.service.kube`) && (PathPrefix(`/dashboard`) || PathPrefix(`/api`))"
        entryPoints:
          - web
    logs:
      general:
        level: INFO
      access:
        enabled: true
    additionalArguments:
      - --experimental.hub
      - --hub
    globalArguments: []
    providers:
      kubernetesIngress:
        allowExternalNameServices: true
        publishedService:
          enabled: true
    hub:
      enabled: true
      tunnelPort: 9901

Here's my traefik-hub values.yaml (kustomize)

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: traefik

helmCharts:
- name: hub-agent
  includeCRDs: true
  namespace: traefik
  releaseName: traefik-hub
  version: 1.1.2
  repo: https://helm.traefik.io/hub
  valuesInline:
    tokenSecretRef:
       name: hub-token
       key: token
    tunnelDeployment:
      traefik:
        tunnelHost: traefik-hub.traefik.svc.cluster.local

I cant upload logs here, its large, so its here: www.astronaut.ninja/logs.txt

Biggest thing that stands out is

{"level":"debug","component":"kubernetes_client","method":"GET","url":"https://10.43.0.1:443/api/v1/namespaces/traefik/secrets/hub-certificate","time":1669296637,"message":"Performing request"}
{"level":"debug","component":"kubernetes_client","method":"POST","url":"https://10.43.0.1:443/apis/traefik.containo.us/v1alpha1/namespaces/traefik/middlewares","time":1669296637,"message":"Performing request"}
{"level":"debug","component":"kubernetes_client","method":"POST","url":"https://10.43.0.1:443/apis/traefik.containo.us/v1alpha1/namespaces/traefik/middlewares","time":1669296637,"message":"Performing request"}
{"level":"debug","component":"kubernetes_client","method":"POST","url":"https://10.43.0.1:443/apis/networking.k8s.io/v1/namespaces/traefik/ingresses","time":1669296637,"message":"Performing request"}
{"time":1669296637,"message":"http: TLS handshake error from 10.42.1.0:44080: remote error: tls: bad certificate"}
{"level":"debug","component":"kubernetes_client","request":"POST https://10.43.0.1:443/apis/networking.k8s.io/v1/namespaces/traefik/ingresses (status: 500)","timeout":"1s","remaining":2,"time":1669296637,"message":"Retrying request"}
{"level":"debug","component":"kubernetes_client","method":"GET","url":"https://10.43.0.1:443/apis/networking.k8s.io/v1/ingresses?allowWatchBookmarks=true&resourceVersion=76014383&timeout=7m0s&timeoutSeconds=420&watch=true","time":1669296638,"message":"Performing request"}
{"time":1669296639,"message":"http: TLS handshake error from 10.42.1.0:44094: remote error: tls: bad certificate"}
{"level":"debug","component":"kubernetes_client","request":"POST https://10.43.0.1:443/apis/networking.k8s.io/v1/namespaces/traefik/ingresses (status: 500)","timeout":"2s","remaining":1,"time":1669296639,"message":"Retrying request"}
{"level":"debug","component":"platform_client","method":"GET","url":"https://platform.hub.traefik.io/agent/commands","time":1669296640,"message":"Performing request"}
{"level":"error","error":"create ingress: Internal error occurred: failed calling webhook \"hub-agent.traefik.svc\": failed to call webhook: Post \"https://hub-agent-controller.traefik.svc:443/ingress?timeout=10s\": x509: certificate signed by unknown authority (possibly because of \"x509: invalid signature: parent certificate cannot sign this kind of certificate\" while trying to verify candidate authority certificate \"hub-agent-controller.traefik.svc\")","time":1669296641,"message":"Unable to synchronize certificate with platform"}
{"time":1669296641,"message":"http: TLS handshake error from 10.42.1.0:45130: remote error: tls: bad certificate"}
4

Hey all. Any ideas or updates?
I'm not super impressed overall yet, it doesnt seem to be able to work at all despite a simplistic set up on my side

5

I was able to get it working.

Not sure which of these did the trick:

  • Deleted everything and recreated it with a new agent on 1.1.0 (was 1.0.0)
  • Changed both kustomize files (see below for final)
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: traefik

resources:
  - prereq.yaml

helmCharts:
- name: traefik
  includeCRDs: true
  namespace: traefik
  releaseName: traefik
  version: 20.4.1
  repo: https://helm.traefik.io/traefik
  valuesInline:
    logs:
      general:
        level: INFO
      access:
        enabled: true
    globalArguments: []
    providers:
      kubernetesIngress:
        allowExternalNameServices: true
        publishedService:
          enabled: true
    hub:
      enabled: true

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: traefik

resources:
  - vault.yaml

helmCharts:
- name: hub-agent
  includeCRDs: true
  namespace: traefik
  releaseName: traefik-hub
  version: 1.2.1
  repo: https://helm.traefik.io/hub
  valuesInline:
    tokenSecretRef:
       name: hub-token
       key: token
    tunnelDeployment:
      traefik:
        tunnelHost: traefik-hub.traefik.svc.cluster.local
6

Hey @myoung34 ,

I am glad you got it working. I wanted to let you know that @rtribotte did not forget you, unfortunately, something urgent came up on our side that took his attention.

I know that it is working for you now, but someone will be looking more deeply into this early next week and they will be able to verify what happened and if your actions fixed it or if it is incidentally working.

If you don't mind, I would love to follow up with you after.

7

Hey @myoung34 , We have tried, but were unable to reproduce whatever it was that went wrong for you, it worked for us. Glad you managed to get it working, we are not sure that the changes you made are what fixed it, but let us know if you have further problems.