Unable to create working HTTPRoute with Traefik 3.1.0 on K3s

Traefik Traefik v3 (latest)
1

Hi, I'm starting a separate post to the discussion of the recent blog post on how to set up Traefik 3.1.0 on Kubernetes and create a simple HTTPRoute.

I have a 3-node K3s cluster on my local network installed with --disable=traefik. The node IPs on my local network are:

  • 10.0.0.192 (control plane)
  • 10.0.0.35
  • 10.0.0.80

I have copied ~/.kube/config from the control plane to my local machine and commands like kubectl get pods -A show 3 pods running:

$ kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS    RESTARTS   AGE
kube-system   coredns-576bfc4dc7-frvvd                  1/1     Running   0          23m
kube-system   local-path-provisioner-86f46b7bf7-rtjcx   1/1     Running   0          23m
kube-system   metrics-server-557ff575fb-8sztf           1/1     Running   0          23m

So far so good. I now attempt to install Traefik 3.1.0 using the Helm chart. First we need to install experimental gateway CRDs to avoid the error Error: UPGRADE FAILED: failed to create resource: Gateway.gateway.networking.k8s.io "traefik-gateway" is invalid: spec.listeners: Invalid value: "array": tls must be specified for protocols ['HTTPS', 'TLS'] as described here.

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.1.0/experimental-install.yaml

I then create values.yaml as described in the blog post:

providers:
  kubernetesIngress:
    enabled: false
  kubernetesGateway:
    enabled: true
gateway:
  listeners:
    web:
      namespacePolicy: All
    websecure:
      namespacePolicy: All

Next we install Traefik 3.1.0 using the Helm chart, currently v30.0.2:

$ helm upgrade --install --namespace traefik traefik traefik/traefik -f values.yaml
Release "traefik" has been upgraded. Happy Helming!
NAME: traefik
LAST DEPLOYED: Wed Aug 14 15:49:04 2024
NAMESPACE: traefik
STATUS: deployed
REVISION: 3
TEST SUITE: None
NOTES:
traefik with docker.io/traefik:v3.1.0 has been deployed successfully on traefik namespace !

Let's inspect the resources it created:

$ kubectl describe deployments.apps traefik --namespace traefik
Name:                   traefik
Namespace:              traefik
CreationTimestamp:      Wed, 14 Aug 2024 15:42:31 +0200
Labels:                 app.kubernetes.io/instance=traefik-traefik
                        app.kubernetes.io/managed-by=Helm
                        app.kubernetes.io/name=traefik
                        helm.sh/chart=traefik-30.0.2
Annotations:            deployment.kubernetes.io/revision: 1
                        meta.helm.sh/release-name: traefik
                        meta.helm.sh/release-namespace: traefik
Selector:               app.kubernetes.io/instance=traefik-traefik,app.kubernetes.io/name=traefik
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  0 max unavailable, 1 max surge
Pod Template:
  Labels:           app.kubernetes.io/instance=traefik-traefik
                    app.kubernetes.io/managed-by=Helm
                    app.kubernetes.io/name=traefik
                    helm.sh/chart=traefik-30.0.2
  Annotations:      prometheus.io/path: /metrics
                    prometheus.io/port: 9100
                    prometheus.io/scrape: true
  Service Account:  traefik
  Containers:
   traefik:
    Image:       docker.io/traefik:v3.1.0
    Ports:       9100/TCP, 9000/TCP, 8000/TCP, 8443/TCP
    Host Ports:  0/TCP, 0/TCP, 0/TCP, 0/TCP
    Args:
      --global.checknewversion
      --global.sendanonymoususage
      --entryPoints.metrics.address=:9100/tcp
      --entryPoints.traefik.address=:9000/tcp
      --entryPoints.web.address=:8000/tcp
      --entryPoints.websecure.address=:8443/tcp
      --api.dashboard=true
      --ping=true
      --metrics.prometheus=true
      --metrics.prometheus.entrypoint=metrics
      --providers.kubernetescrd
      --providers.kubernetesgateway
      --entryPoints.websecure.http.tls=true
      --log.level=INFO
    Liveness:   http-get http://:9000/ping delay=2s timeout=2s period=10s #success=1 #failure=3
    Readiness:  http-get http://:9000/ping delay=2s timeout=2s period=10s #success=1 #failure=1
    Environment:
      POD_NAME:        (v1:metadata.name)
      POD_NAMESPACE:   (v1:metadata.namespace)
    Mounts:
      /data from data (rw)
      /tmp from tmp (rw)
  Volumes:
   data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
   tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   traefik-7c7587b647 (1/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  7m3s  deployment-controller  Scaled up replica set traefik-7c7587b647 to 1
$ kubectl describe GatewayClass traefik
Name:         traefik
Namespace:    
Labels:       app.kubernetes.io/instance=traefik-traefik
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=traefik
              helm.sh/chart=traefik-30.0.2
Annotations:  meta.helm.sh/release-name: traefik
              meta.helm.sh/release-namespace: traefik
API Version:  gateway.networking.k8s.io/v1
Kind:         GatewayClass
Metadata:
  Creation Timestamp:  2024-08-14T13:42:31Z
  Generation:          1
  Resource Version:    2120
  UID:                 3f0269e6-f9fa-4870-b1da-23ff17baf08d
Spec:
  Controller Name:  traefik.io/gateway-controller
Status:
  Conditions:
    Last Transition Time:  2024-08-14T13:42:49Z
    Message:               Handled by Traefik controller
    Observed Generation:   1
    Reason:                Handled
    Status:                True
    Type:                  Accepted
Events:                    <none>
$  kubectl describe Gateway traefik --namespace traefik
Name:         traefik-gateway
Namespace:    traefik
Labels:       app.kubernetes.io/instance=traefik-traefik
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=traefik
              helm.sh/chart=traefik-30.0.2
Annotations:  meta.helm.sh/release-name: traefik
              meta.helm.sh/release-namespace: traefik
API Version:  gateway.networking.k8s.io/v1
Kind:         Gateway
Metadata:
  Creation Timestamp:  2024-08-14T13:49:05Z
  Generation:          1
  Resource Version:    2119
  UID:                 f287a5ae-21fe-4776-864d-142a2030895d
Spec:
  Gateway Class Name:  traefik
  Listeners:
    Allowed Routes:
      Namespaces:
        From:  All
    Name:      web
    Port:      8000
    Protocol:  HTTP
    Allowed Routes:
      Namespaces:
        From:  All
    Name:      websecure
    Port:      8443
    Protocol:  HTTPS
Status:
  Conditions:
    Last Transition Time:  2024-08-14T13:49:05Z
    Message:               All Listeners must be valid
    Observed Generation:   1
    Reason:                ListenersNotValid
    Status:                False
    Type:                  Accepted
  Listeners:
    Attached Routes:  0
    Conditions:
      Last Transition Time:  2024-08-14T13:49:05Z
      Message:               No error found
      Observed Generation:   1
      Reason:                Accepted
      Status:                True
      Type:                  Accepted
      Last Transition Time:  2024-08-14T13:49:05Z
      Message:               No error found
      Observed Generation:   1
      Reason:                ResolvedRefs
      Status:                True
      Type:                  ResolvedRefs
      Last Transition Time:  2024-08-14T13:49:05Z
      Message:               No error found
      Observed Generation:   1
      Reason:                Programmed
      Status:                True
      Type:                  Programmed
    Name:                    web
    Supported Kinds:
      Group:          gateway.networking.k8s.io
      Kind:           HTTPRoute
    Attached Routes:  0
    Conditions:
      Last Transition Time:  2024-08-14T13:49:05Z
      Message:               No TLS configuration for Gateway Listener websecure:8443 and protocol "HTTPS"
      Observed Generation:   1
      Reason:                InvalidTLSConfiguration
      Status:                False
      Type:                  Accepted
    Name:                    websecure
    Supported Kinds:
      Group:  gateway.networking.k8s.io
      Kind:   HTTPRoute
Events:       <none>

It looks like it is listening on ports 8000 (HTTP), 8443 (HTTPS), 9000 (Traefik) and 9100 (Metrics). I'm a bit concerned about InvalidTLSConfiguration, ListenersNotValid and No TLS configuration for Gateway Listener websecure:8443 and protocol "HTTPS" but I think as long as we connect on port 8000 the routing should still work? My test cluster now looks like this:

$ kubectl get deployments --all-namespaces
NAMESPACE     NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   coredns                  1/1     1            1           51m
kube-system   local-path-provisioner   1/1     1            1           51m
kube-system   metrics-server           1/1     1            1           51m
traefik       traefik                  1/1     1            1           12m

Now I'm going to try and create a namespace for the whoami deployment:

$ kubectl create namespace whoami
namespace/whoami created

I create a file called whoami.yaml with a Deployment, Service and HTTPRoute

$ cat whoami.yaml 
# Application to expose
kind: Deployment
apiVersion: apps/v1
metadata:
  name: whoami
  namespace: whoami
spec:
  replicas: 3
  selector:
    matchLabels:
      app: whoami
  template:
    metadata:
      labels:
        app: whoami
    spec:
      containers:
        - name: whoami
          image: traefik/whoami
---
# Service to reach the application on the cluster
apiVersion: v1
kind: Service
metadata:
  name: whoami
  namespace: whoami
  labels:
    app: whoami
spec:
  type: ClusterIP
  ports:
    - port: 80
      name: whoami
  selector:
    app: whoami
---
# HTTPRoute
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: whoami-httproute
  namespace: whoami
spec:
  parentRefs:
    - name: traefik-gateway
      namespace: traefik
  hostnames:
    - whoami.myexample.io
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: whoami
          namespace: whoami
          port: 80

And deploy it:

$ kubectl apply -f whoami.yaml 
deployment.apps/whoami created
service/whoami created
httproute.gateway.networking.k8s.io/whoami-httproute created

My system now looks like this:

$ kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS    RESTARTS   AGE
kube-system   coredns-576bfc4dc7-frvvd                  1/1     Running   0          113m
kube-system   local-path-provisioner-86f46b7bf7-rtjcx   1/1     Running   0          113m
kube-system   metrics-server-557ff575fb-8sztf           1/1     Running   0          113m
kube-system   svclb-traefik-769c0052-sf9gq              2/2     Running   0          73m
kube-system   svclb-traefik-769c0052-wdnkg              2/2     Running   0          73m
kube-system   svclb-traefik-769c0052-zhf7p              2/2     Running   0          73m
traefik       traefik-7c7587b647-ms624                  1/1     Running   0          73m
whoami        whoami-98d7579fb-264vv                    1/1     Running   0          86s
whoami        whoami-98d7579fb-4txnv                    1/1     Running   0          86s
whoami        whoami-98d7579fb-59jx2                    1/1     Running   0          86s
$ kubectl get deployments -A
NAMESPACE     NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   coredns                  1/1     1            1           113m
kube-system   local-path-provisioner   1/1     1            1           113m
kube-system   metrics-server           1/1     1            1           113m
traefik       traefik                  1/1     1            1           73m
whoami        whoami                   3/3     3            3           94s
$ kubectl get httproute --all-namespaces
NAMESPACE   NAME               HOSTNAMES                 AGE
whoami      whoami-httproute   ["whoami.myexample.io"]   116s
$ kubectl describe httproute whoami-httproute -n whoami
Name:         whoami-httproute
Namespace:    whoami
Labels:       <none>
Annotations:  <none>
API Version:  gateway.networking.k8s.io/v1
Kind:         HTTPRoute
Metadata:
  Creation Timestamp:  2024-08-14T14:54:42Z
  Generation:          1
  Resource Version:    4273
  UID:                 021689ab-7e07-482d-8f43-8d854bea0009
Spec:
  Hostnames:
    whoami.myexample.io
  Parent Refs:
    Group:      gateway.networking.k8s.io
    Kind:       Gateway
    Name:       traefik-gateway
    Namespace:  traefik
  Rules:
    Backend Refs:
      Group:      
      Kind:       Service
      Name:       whoami
      Namespace:  whoami
      Port:       80
      Weight:     1
    Matches:
      Path:
        Type:   PathPrefix
        Value:  /
Status:
  Parents:
    Conditions:
      Last Transition Time:  2024-08-14T14:54:48Z
      Message:               
      Observed Generation:   1
      Reason:                Accepted
      Status:                True
      Type:                  Accepted
      Last Transition Time:  2024-08-14T14:54:48Z
      Message:               
      Observed Generation:   1
      Reason:                ResolvedRefs
      Status:                True
      Type:                  ResolvedRefs
    Controller Name:         traefik.io/gateway-controller
    Parent Ref:
      Group:      gateway.networking.k8s.io
      Kind:       Gateway
      Name:       traefik-gateway
      Namespace:  traefik
Events:           <none>

This should complete the setup, but I don't yet understand how to access the service through the Traefik gateway. Based on these docs, it should be possible to use curl like this:

$ curl -H Host:whoami.myexample.io http://10.0.0.192:8000
curl: (7) Failed to connect to 10.0.0.192 port 8000 after 4 ms: Couldn't connect to server

It seems that Traefik isn't actually listening on the entrypoint. I tried editing my local /etc/hosts file like this:

127.0.0.1       localhost
127.0.1.1       XPS-HNCG1T3
10.0.0.192      whoami.myexample.io

Curl still fails:

$ curl http://whoami.myexample.io:8000
curl: (7) Failed to connect to whoami.myexample.io port 8000 after 4 ms: Couldn't connect to server
$ dig whoami.myexample.io

; <<>> DiG 9.18.28-0ubuntu0.24.04.1-Ubuntu <<>> whoami.myexample.io
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 8838
;; flags: qr aa rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;whoami.myexample.io.		IN	A

;; ANSWER SECTION:
whoami.myexample.io.	0	IN	A	10.0.0.192

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Wed Aug 14 17:31:48 CEST 2024
;; MSG SIZE  rcvd: 64

I've also tried logging in to one of the cluster nodes to see if it works from localhost:

$ ssh pi@10.0.0.192
pi@turing-one:~ $ curl -H Host:whoami.myexample.io http://localhost:8000
curl: (7) Failed to connect to localhost port 8000 after 0 ms: Couldn't connect to server
pi@turing-one:~ $ curl -H Host:whoami.myexample.io http://localhost:8000/
curl: (7) Failed to connect to localhost port 8000 after 0 ms: Couldn't connect to server

I've been stuck on this for around a week, and would greatly appreciate any help! Is there some step I am missing? Should I remove/replace the example name? How can I get this simple test scenario working?

2

An interesting read. Just not to waste your time and so that you know what to expect: I have not done the traefik 3 gateway API configuration yet. This is in my to do list and I will definitely have to do it for work, but I did not get to it yet. At the same time, I'm experienced with traefik in general and have been using it in multiple configurations for years.

From what you wrote it appears to me that you are not exposing the traefik ports and thus cannot connect to it. The blog post probably assumes some sort of cloud provider where you set up the port mappings between your kubernetes pod and external world in a cloud management panel, but since you are running your local copy it's up to you to expose the ports.

The blog post says

The entrypoints web and websecure will be used to expose your applications.

And this is true, but it is up to you to make sure that you can connect to them. As it's not possible to tell what cloud platform you are running on it also is not possible to give generic instructions in the blog post - that's my guess what happened here.

So you just need to expose your traefik ports in the usual way (whichever you prefer).

Once you've done that you should be good to go. (Once again, have not done it myself with v3, so there may be wrinkles I'm unaware of).

3

@strophy One needs to locate the service from the proper namespace by using the following:

kubectl get svc --namespace traefik traefik

The result should look something like the following:

NAME      TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE
traefik   LoadBalancer   10.98.135.173   192.1.2.101   80:31087/TCP,443:30107/TCP   29s

Now, we have an external IP address, 192.1.2.101. Thus, we can update /etc/hosts by appending the following to the bottom of the file:

192.1.2.101  whoami.myexample.io

Now, you can curl this domain from the command line:

➜ curl whoami.myexample.io
404 page not found

Note: This is the expected message because we haven't applied our HTTPRoute resource manifest to the K8s cluster to communicate with our underlying whoami service.

After applying the HTTPRoute resource manifest, one should see the following message:

➜ curl whoami.myexample.io                 
Hostname: whoami-778d7d4bdf-mkgng
IP: 127.0.0.1
IP: ::1
IP: 10.244.0.3
IP: fe80::505c:f6ff:fed6:5e53
RemoteAddr: 10.244.2.2:46346
GET / HTTP/1.1
Host: whoami.myexample.io
User-Agent: curl/8.9.1
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 192.1.2.3
X-Forwarded-Host: whoami.myexample.io
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Server: traefik-697c646c8b-5wwv7
X-Real-Ip: 192.1.2.3

I hope that the above helps.

4

@conradwt it works, thank you so much for sticking with a noob internet stranger through so many posts as I learn this! I actually had it working the whole time (I mentioned modifying the /etc/hosts file in my post) but incorrectly assumed that I needed to reach the service through the Traefik gateway listener port, which is configured like this:

Spec:
  Gateway Class Name:  traefik
  Listeners:
    Allowed Routes:
      Namespaces:
        From:  All
    Name:      web
    Port:      8000
    Protocol:  HTTP
    Allowed Routes:
      Namespaces:
        From:  All
    Name:      websecure
    Port:      8443
    Protocol:  HTTPS

Which doesn't work:

$ curl http://whoami.myexample.io:8000
curl: (7) Failed to connect to whoami.myexample.io port 8000 after 3 ms: Couldn't connect to server

When in fact it was listening on port 80 this whole time:

$ curl http://whoami.myexample.io
Hostname: whoami-98d7579fb-59jx2
IP: 127.0.0.1
IP: ::1
IP: 10.42.1.5
IP: fe80::bc2a:5aff:fe88:dadc
RemoteAddr: 10.42.3.6:40918
GET / HTTP/1.1
Host: whoami.myexample.io
User-Agent: curl/8.5.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 10.42.0.0
X-Forwarded-Host: whoami.myexample.io
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Server: traefik-7c7587b647-ms624
X-Real-Ip: 10.42.0.0

It also works when logged in to any of the k3s hosts with curl -H Host:whoami.myexample.io http://localhost.

Just to help develop my conceptual understanding, what is purpose of the port 8000 listener, and why does it appear to not actually be listening?

Thanks again mate, huge help!!! :beers:

5

It's great to hear that things are working now. Anyway, Traefik configures port 8000 to expose an HTTP entry point. Thus, all incoming HTTP requests are routed through port 8000 to the underlying service resource.

Next, the following failed because the underlying whoami service is listening on port 80 and not 8000:

curl http://whoami.myexample.io:8000

Please see your K8s Gayway, HTTPRoute, and Service manifests for the details. For production environments, you should generally avoid using port 8000 directly and instead ensure that traffic is routed through HTTPS on port 443.

1 Like
6

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.