Traefik Hub & k3s

johnford2002 · April 17, 2023, 2:08am

I've been trying to get Traefik Hub to work with an existing installation of Traefik Proxy running in k3s.

Some version info first:

k3s v1.25.8+k3s1
traefik v2.9.4
hub-agent-kubernetes v1.3.0

Additional configuration provided to traefik via k3s HelmChartConfig:

additionalArguments:
      - "--api.insecure=true"
      - "--entrypoints.websecure.http.tls"
      - "--providers.kubernetesingress.ingressclass=traefik-cert-manager"
      - "--experimental.hub"
      - "--hub"
      - "--hub.tls.insecure=true"
      - "--entrypoints.traefikhub-api.address=:9900"
      - "--entrypoints.traefikhub-tunl.address=:9901"
      - "--metrics.prometheus.addrouterslabels"

I noticed that even though I specified an alternate namespace during helm install, the tunnel and controller still tried to point to a service in the default instructions namespace of hub-agent.

I created the following service and updated both deploys to point to it:

apiVersion: v1
kind: Service
metadata:
  name: traefik-hub
  namespace: kube-system
  labels:
    app: traefik
    app.kubernetes.io/name: traefik
spec:
  type: ClusterIP
  selector:
    app.kubernetes.io/name: traefik
  ports:
  - port: 9100
    name: "metrics"
    targetPort: 9100
    protocol: TCP
  - port: 9901
    name: "traefikhub-tunl"
    targetPort: 9901
    protocol: TCP

With that much done, the agent can connect and see all services. I see that there are registered routers in my traefik dashboard for proxy.traefik. However, no matter which service I try to expose, the result is a 404 page.

The one possibly related error that I see is a TLS handshake error on the proxy coming from the tunnel, even after setting the arg for insecure:

time="2023-04-17T02:07:13Z" level=debug msg="http: TLS handshake error from 10.42.3.95:57854: remote error: tls: unknown certificate"

I'm a bit lost as to what I'm missing at this stage. Can anyone provide some guidance?

svx · April 19, 2023, 12:59pm

Hi @johnford2002, thanks for your interest in Traefik Hub!

This might be related to this PR.

project0 · April 19, 2023, 2:08pm

@johnford2002 can you provide more information how you installed the hub agent? Did you installed via helm in the same namespace.

johnford2002 · April 19, 2023, 4:04pm

@svx - Thanks for the PR link. I think I was facing issues similar to what was going on there initially, but I found the docs located here which at least helped me figure out what was going on with the missing service. I manually edited the deploys, for now, to point to my manually created service, although I realize I'll need to update the helm later if I don't want to lose my changes on a future update.

I suspect that using the additionalArguments of the k3s HelmChartConfig doesn't trigger this portion of the traefik proxy helm chart. I certainly may have missed it, but if not, might want to include some guidance to use this setting under valuesContent to get the service to create automatically rather than manually:

hub:
      enabled: true

@project0 - I used the following helm command to install the hub agent:

helm upgrade --install hub-agent traefik/hub-agent --set token="REDACTED" --namespace kube-system

To clarify, I don't think I'm having a service communication issue at this stage. I'm not getting timeouts. The agent connects, services are displayed in Traefik Hub web UI, and I can see at least some routers registered by components of the hub agent. I just can't seem to figure out why published services are all returning a 404, and the previously mentioned TLS error is the only thing I've found so far in the logs.

Happy to uninstall, reinstall, or try any various settings that would be helpful in debugging. Thanks for your assistance!

johnford2002 · April 19, 2023, 5:19pm

An update here, I deleted all previously published services, disconnected the agent, and uninstalled the hub release.

I updated the k3s HelmChartConfig to contain the following under valuesContent:

    hub:
      enabled: true
    logs:
      access: 
        enabled: true
      general:
        level: DEBUG
    additionalArguments:
      - "--api.insecure=true"
      - "--entrypoints.websecure.http.tls"
      - "--providers.kubernetesingress.ingressclass=traefik-cert-manager"
      - "--hub.tls.insecure=true"

Looking at the deploy of Traefik Proxy afterward, this seems to have set a number of additional arguments that weren't in my original list, and it automatically created the traefik-hub service as expected.

I launched the agent setup instructions again from Traefik Hub, and I saved the resulting token into a k8s secret for the next step.

I then re-installed the hub-agent using the following:

export METRICS_URL=http://traefik-hub.kube-system.svc.cluster.local:9100/metrics
export TUNNEL_HOST=traefik-hub.kube-system.svc.cluster.local
export TUNNEL_PORT=9901
export INGRESS_CLASS_NAME=traefik

helm upgrade --install hub-agent traefik/hub-agent --namespace kube-system \
  --set tokenSecretRef.name=hub-agent --set tokenSecretRef.key=token \
  --set controllerDeployment.traefik.metricsURL="${METRICS_URL}" \
  --set controllerDeployment.args="{--ingress-class-name=${INGRESS_CLASS_NAME}}" \
  --set tunnelDeployment.traefik.tunnelHost=${TUNNEL_HOST} \
  --set tunnelDeployment.traefik.tunnelPort=${TUNNEL_PORT}

That brought me back to agent connected and service discovery working. Trying to publish a service, I end up getting the same behavior as before, a 404 and a TLS handshake error in the logs:

time="2023-04-19T17:06:19Z" level=debug msg="Serving default certificate for request: \"casual-flyingfish-k9c6e5.jq4pduby.traefikhub.io\""
time="2023-04-19T17:06:19Z" level=debug msg="http: TLS handshake error from 10.42.0.54:40396: remote error: tls: unknown certificate"

I do see something that is either new or missed from earlier logs on the hub-agent-controller:

Reviewing EdgeIngress resource

I do not have a corresponding log entry for "Creating EdgeIngress resource" or "Updating EdgeIngress resource".

However, I DO have that CRD available, and an EdgeIngress resource was created in my cluster in the namespace for the service that was published, a different namespace from where traefik and the hub agent are installed. The EdgeIngress doesn't show up under Ingresses within the Traefik Hub UI.

johnford2002 · April 19, 2023, 8:32pm

More details...

NOT including any hub TLS settings, as suggested in the minimal static configuration for kubernetes, results in no routers being added for proxy.traefik.

Adding at least the insecure option will create those routers:

hub:
    tls:
        insecure: true

This can be seen in a bunch of the debug logs that show up once this argument gets passed:

time="2023-04-19T20:25:58Z" level=info msg="The entryPoint \"traefikhub-tunl\" is created on port 9901 to allow exposition of services."
time="2023-04-19T20:25:58Z" level=info msg="The entryPoint \"traefikhub-api\" is created on port 9900 to allow Traefik to communicate with the Hub Agent for Traefik."
time="2023-04-19T20:25:58Z" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true,\"sendAnonymousUsage\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"metrics\":{\"address\":\":9100/tcp\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}},\"traefik\":{\"address\":\":9000/tcp\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}},\"traefikhub-api\":{\"address\":\":9900\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{\"tls\":{\"options\":\"traefik-hub\"}},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}},\"traefikhub-tunl\":{\"address\":\":9901\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}},\"web\":{\"address\":\":8000/tcp\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}},\"websecure\":{\"address\":\":8443/tcp\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{\"tls\":{}},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}}},\"providers\":{\"providersThrottleDuration\":\"2s\",\"kubernetesIngress\":{\"ingressClass\":\"traefik-cert-manager\",\"ingressEndpoint\":{\"publishedService\":\"kube-system/traefik\"}},\"kubernetesCRD\":{\"allowCrossNamespace\":true,\"allowExternalNameServices\":true}},\"api\":{\"insecure\":true,\"dashboard\":true},\"metrics\":{\"prometheus\":{\"buckets\":[0.1,0.3,1.2,5],\"addEntryPointsLabels\":true,\"addRoutersLabels\":true,\"addServicesLabels\":true,\"entryPoint\":\"metrics\"}},\"ping\":{\"entryPoint\":\"traefik\",\"terminatingStatusCode\":503},\"log\":{\"level\":\"DEBUG\",\"format\":\"common\"},\"accessLog\":{\"format\":\"common\",\"filters\":{},\"fields\":{\"defaultMode\":\"keep\",\"headers\":{\"defaultMode\":\"drop\"}}},\"hub\":{\"tls\":{\"insecure\":true}},\"experimental\":{\"hub\":true}}"
time="2023-04-19T20:25:58Z" level=debug msg="Starting TCP Server" entryPointName=traefikhub-tunl
time="2023-04-19T20:25:58Z" level=debug msg="Starting TCP Server" entryPointName=traefikhub-api
time="2023-04-19T20:25:58Z" level=info msg="Starting provider *hub.Provider"
time="2023-04-19T20:25:58Z" level=debug msg="*hub.Provider provider configuration: {\"tls\":{\"insecure\":true}}"
time="2023-04-19T20:25:58Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"api\":{\"entryPoints\":[\"traefik\"],\"service\":\"api@internal\",\"rule\":\"PathPrefix(`/api`)\",\"priority\":2147483646},\"dashboard\":{\"entryPoints\":[\"traefik\"],\"middlewares\":[\"dashboard_redirect@internal\",\"dashboard_stripprefix@internal\"],\"service\":\"dashboard@internal\",\"rule\":\"PathPrefix(`/`)\",\"priority\":2147483645},\"ping\":{\"entryPoints\":[\"traefik\"],\"service\":\"ping@internal\",\"rule\":\"PathPrefix(`/ping`)\",\"priority\":2147483647},\"prometheus\":{\"entryPoints\":[\"metrics\"],\"service\":\"prometheus@internal\",\"rule\":\"PathPrefix(`/metrics`)\",\"priority\":2147483647}},\"services\":{\"api\":{},\"dashboard\":{},\"noop\":{},\"ping\":{},\"prometheus\":{}},\"middlewares\":{\"dashboard_redirect\":{\"redirectRegex\":{\"regex\":\"^(http:\\\\/\\\\/(\\\\[[\\\\w:.]+\\\\]|[\\\\w\\\\._-]+)(:\\\\d+)?)\\\\/$\",\"replacement\":\"${1}/dashboard/\",\"permanent\":true}},\"dashboard_stripprefix\":{\"stripPrefix\":{\"prefixes\":[\"/dashboard/\",\"/dashboard\"]}}},\"models\":{\"traefikhub-api\":{\"tls\":{\"options\":\"traefik-hub\"}},\"websecure\":{\"tls\":{}}},\"serversTransports\":{\"default\":{\"maxIdleConnsPerHost\":200}}},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=internal
time="2023-04-19T20:25:58Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"traefik-hub-agent-api\":{\"entryPoints\":[\"traefikhub-api\"],\"service\":\"api@internal\",\"rule\":\"Host(`proxy.traefik`) \\u0026\\u0026 PathPrefix(`/api`)\"},\"traefik-hub-agent-metrics\":{\"entryPoints\":[\"traefikhub-api\"],\"service\":\"prometheus@internal\",\"rule\":\"Host(`proxy.traefik`) \\u0026\\u0026 PathPrefix(`/metrics`)\"},\"traefik-hub-agent-service\":{\"entryPoints\":[\"traefikhub-api\"],\"service\":\"traefik-hub-agent-service\",\"rule\":\"Host(`proxy.traefik`) \\u0026\\u0026 PathPrefix(`/config`, `/discover-ip`, `/state`)\"}},\"services\":{\"traefik-hub-agent-service\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://127.0.0.1:34253\"}],\"passHostHeader\":null}}}},\"tcp\":{},\"udp\":{},\"tls\":{\"options\":{\"traefik-hub\":{\"minVersion\":\"VersionTLS13\",\"clientAuth\":{}}}}}" providerName=hub
time="2023-04-19T20:25:59Z" level=debug msg="Creating middleware" middlewareName=metrics-entrypoint middlewareType=Metrics entryPointName=traefikhub-api
time="2023-04-19T20:25:59Z" level=debug msg="Creating middleware" entryPointName=traefikhub-tunl middlewareName=metrics-entrypoint middlewareType=Metrics
time="2023-04-19T20:25:59Z" level=debug msg="Creating middleware" entryPointName=traefikhub-api middlewareName=metrics-entrypoint middlewareType=Metrics
time="2023-04-19T20:25:59Z" level=debug msg="Creating middleware" entryPointName=traefikhub-tunl middlewareName=metrics-entrypoint middlewareType=Metrics
time="2023-04-19T20:26:00Z" level=debug msg="Creating middleware" middlewareName=metrics-entrypoint middlewareType=Metrics entryPointName=traefikhub-api
time="2023-04-19T20:26:00Z" level=debug msg="Creating middleware" entryPointName=traefikhub-tunl middlewareName=metrics-entrypoint middlewareType=Metrics
time="2023-04-19T20:26:00Z" level=debug msg="Added outgoing tracing middleware api@internal" middlewareType=TracingForwarder entryPointName=traefikhub-api routerName=traefik-hub-agent-api
hub middlewareName=tracing
time="2023-04-19T20:26:00Z" level=debug msg="Creating middleware" entryPointName=traefikhub-api routerName=traefik-hub-agent-api@hub middlewareName=metrics-router middlewareType=Metrics
time="2023-04-19T20:26:00Z" level=debug msg="Added outgoing tracing middleware prometheus@internal" entryPointName=traefikhub-api routerName=traefik-hub-agent-metrics@hub middlewareType=TracingForwarder middlewareName=tracing
time="2023-04-19T20:26:00Z" level=debug msg="Creating middleware" entryPointName=traefikhub-api routerName=traefik-hub-agent-metrics@hub middlewareType=Metrics middlewareName=metrics-router
time="2023-04-19T20:26:00Z" level=debug msg="Creating middleware" entryPointName=traefikhub-api middlewareName=pipelining middlewareType=Pipelining routerName=traefik-hub-agent-service@hu
 serviceName=traefik-hub-agent-service
time="2023-04-19T20:26:00Z" level=debug msg="Creating middleware" middlewareType=Metrics middlewareName=metrics-service entryPointName=traefikhub-api routerName=traefik-hub-agent-service@hub serviceName=traefik-hub-agent-service
time="2023-04-19T20:26:00Z" level=debug msg="Creating load-balancer" routerName=traefik-hub-agent-service@hub serviceName=traefik-hub-agent-service entryPointName=traefikhub-api
time="2023-04-19T20:26:00Z" level=debug msg="Creating server 0 http://127.0.0.1:34253" serverName=0 entryPointName=traefikhub-api routerName=traefik-hub-agent-service@hub serviceName=traefik-hub-agent-service
time="2023-04-19T20:26:00Z" level=debug msg="Added outgoing tracing middleware traefik-hub-agent-service" middlewareName=tracing middlewareType=TracingForwarder entryPointName=traefikhub-api routerName=traefik-hub-agent-service@hub
time="2023-04-19T20:26:00Z" level=debug msg="Creating middleware" entryPointName=traefikhub-api routerName=traefik-hub-agent-service@hub middlewareName=metrics-router middlewareType=Metrics
time="2023-04-19T20:26:00Z" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=traefikhub-api middlewareName=traefik-internal-recovery
time="2023-04-19T20:26:00Z" level=debug msg="Creating middleware" middlewareType=Metrics entryPointName=traefikhub-api middlewareName=metrics-entrypoint
time="2023-04-19T20:26:00Z" level=debug msg="Creating middleware" entryPointName=traefikhub-tunl middlewareName=metrics-entrypoint middlewareType=Metrics
time="2023-04-19T20:26:00Z" level=debug msg="Adding route for proxy.traefik with TLS options traefik-hub@hub" entryPointName=traefikhub-api
time="2023-04-19T20:26:02Z" level=debug msg="Creating middleware" entryPointName=traefikhub-api middlewareName=metrics-entrypoint middlewareType=Metrics
time="2023-04-19T20:26:02Z" level=debug msg="Creating middleware" entryPointName=traefikhub-tunl middlewareName=metrics-entrypoint middlewareType=Metrics
time="2023-04-19T20:26:02Z" level=debug msg="Creating middleware" middlewareName=pipelining middlewareType=Pipelining routerName=traefik-hub-agent-service@hub serviceName=traefik-hub-agent-service entryPointName=traefikhub-api
time="2023-04-19T20:26:02Z" level=debug msg="Creating middleware" entryPointName=traefikhub-api routerName=traefik-hub-agent-service@hub serviceName=traefik-hub-agent-service middlewareName=metrics-service middlewareType=Metrics
time="2023-04-19T20:26:02Z" level=debug msg="Creating load-balancer" routerName=traefik-hub-agent-service@hub serviceName=traefik-hub-agent-service entryPointName=traefikhub-api
time="2023-04-19T20:26:02Z" level=debug msg="Creating server 0 http://127.0.0.1:34253" serverName=0 entryPointName=traefikhub-api routerName=traefik-hub-agent-service@hub serviceName=traefik-hub-agent-service
time="2023-04-19T20:26:02Z" level=debug msg="Added outgoing tracing middleware traefik-hub-agent-service" entryPointName=traefikhub-api routerName=traefik-hub-agent-service@hub middlewareName=tracing middlewareType=TracingForwarder
time="2023-04-19T20:26:02Z" level=debug msg="Creating middleware" routerName=traefik-hub-agent-service@hub middlewareName=metrics-router middlewareType=Metrics entryPointName=traefikhub-api
time="2023-04-19T20:26:02Z" level=debug msg="Added outgoing tracing middleware api@internal" middlewareName=tracing middlewareType=TracingForwarder routerName=traefik-hub-agent-api@hub entryPointName=traefikhub-api
time="2023-04-19T20:26:02Z" level=debug msg="Creating middleware" entryPointName=traefikhub-api routerName=traefik-hub-agent-api@hub middlewareName=metrics-router middlewareType=Metrics
time="2023-04-19T20:26:02Z" level=debug msg="Added outgoing tracing middleware prometheus@internal" entryPointName=traefikhub-api routerName=traefik-hub-agent-metrics@hub middlewareName=tracing middlewareType=TracingForwarder
time="2023-04-19T20:26:02Z" level=debug msg="Creating middleware" entryPointName=traefikhub-api routerName=traefik-hub-agent-metrics@hub middlewareName=metrics-router middlewareType=Metrics
time="2023-04-19T20:26:02Z" level=debug msg="Creating middleware" entryPointName=traefikhub-api middlewareType=Recovery middlewareName=traefik-internal-recovery
time="2023-04-19T20:26:02Z" level=debug msg="Creating middleware" entryPointName=traefikhub-api middlewareName=metrics-entrypoint middlewareType=Metrics
time="2023-04-19T20:26:02Z" level=debug msg="Creating middleware" entryPointName=traefikhub-tunl middlewareName=metrics-entrypoint middlewareType=Metrics
time="2023-04-19T20:26:02Z" level=debug msg="Adding route for proxy.traefik with TLS options traefik-hub@hub" entryPointName=traefikhub-api

I'm starting to suspect that there's an issue on the hub-agent-kubernetes side of things. Comparing it to the hub-agent-traefik (what's used for Docker I believe?), it doesn't seem to have an argument to set TLS to insecure, so I'm wondering if the client that is created defaults to trying to verify TLS?

johnford2002 · April 21, 2023, 1:55pm

@svx / @project0 - Any chance either of you could provide some more detailed instructions on generating an appropriate TLS ca, cert, and key to set on the Traefik Proxy side? I've tried the insecure setting and using the default certs that are mentioned in the docs, and neither approach seems to correct the TLS handshake error.

svx · April 26, 2023, 9:12am

Hi @johnford2002,

You should not have to generate a TLS cert.

I just scrolled up to the beginning of the tread and saw that you're using traefik v2.9.4.

Did you install Traefik via Helm, or do you use the version which comes with k3s out of the box?

If possible for you, I suggest updating Traefik to 2.10, it seems the image got already added to the rancher image mirror.

johnford2002 · April 27, 2023, 8:49pm

I'll give that a shot this weekend and report back.

Topic		Replies	Views
Published services not working Traefik Hub	6	593	December 6, 2022
Dashboard not reachable - helm install on k3s Traefik v2 kubernetes-ingress	3	1395	June 2, 2024
Error lookup traefik-hub.hub-agent.svc.cluster.local Traefik Hub kubernetes	4	307	December 1, 2022
Traefik Hub: Publish & Secure Applications \| Traefik Labs Blog	0	280	June 29, 2022
Kubernetes TLS unsupported versions Traefik v2 kubernetes-crd , kubernetes-ingress , tcp	11	3262	July 22, 2020

Traefik Hub & k3s

Related topics