I am learning Traefik after hearing about it at Kubecon 2024.
I have a vanilla minikube cluster that i created using the following
The objective for me is to setup
- Traefik as ingress proxy to my minikube
- Use K8s GW api as the definitive way to author routes
- Expose all my workloads on the Tailscale over HTTPS ( even though HTTP is secured over TS overlay network )
My minikube cluster is setup as
minikube start --memory=16384 --cpus=4 --kubernetes-version=v1.31.0
Then i deployed traefik using the following values.yaml via helm
service:
type: ClusterIP
annotations:
tailscale.com/expose: "true"
tailscale.com/hostname: "proxy"
providers:
kubernetesIngress:
enabled: false
kubernetesGateway:
enabled: true
certificatesResolvers:
myresolver:
tailscale: {}
logs:
general:
level: "INFO"
access:
enabled: true
addInternals: false
ingressRoute:
dashboard:
enabled: true
gateway:
listeners:
web:
namespacePolicy: All
websecure:
namespacePolicy: All
port: 8443
protocol: HTTPS
certificateRefs:
- name: myresolver
Tailscale operator was setup and confirmed working prior independent of the traefik.
Then i deployed whoami to validate sample request over http and https
The manifest is
kind: Namespace
apiVersion: v1
metadata:
name: whoami
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: whoami
labels:
app: whoami
namespace: whoami
spec:
replicas: 1
selector:
matchLabels:
app: whoami
template:
metadata:
labels:
app: whoami
spec:
containers:
- name: whoami
image: traefik/whoami
ports:
- name: web
containerPort: 80
resources:
requests:
cpu: "50m"
memory: "100Mi"
limits:
cpu: "100m"
memory: "100Mi"
---
kind: Service
apiVersion: v1
metadata:
name: whoami
namespace: whoami
spec:
type: ClusterIP
ports:
- name: web
port: 80
targetPort: web
selector:
app: whoami
---
# HTTPRoute
kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1
metadata:
name: whoami-httproute
namespace: whoami
spec:
parentRefs:
- name: traefik-gateway
namespace: traefik-v2
hostnames:
- proxy.penguin-ghost.ts.net
rules:
- matches:
- path:
type: PathPrefix
value: /whoami
backendRefs:
- name: whoami
namespace: whoami
port: 80
---
kind: IngressRoute
apiVersion: traefik.io/v1alpha1
metadata:
name: whoami
namespace: whoami
spec:
entryPoints:
- websecure
routes:
- match: Path(`/whoami`)
kind: Rule
services:
- name: whoami
port: 80
tls:
certResolver: myresolver
domains:
- main: proxy.penguin-ghost.ts.net
I do see the tailscale creating a machine hostname for entry into my minikube via Traefik Gateway however i do not see a certificate being issue.
HTTP endpoint works
http http://proxy.penguin-ghost.ts.net/whoami minikube
HTTP/1.1 200 OK
Content-Length: 453
Content-Type: text/plain; charset=utf-8
Date: Sun, 17 Nov 2024 23:27:20 GMT
Hostname: whoami-d4f874664-nmgss
IP: 127.0.0.1
IP: ::1
IP: 10.244.0.47
IP: fe80::28f6:96ff:fe21:201e
RemoteAddr: 10.244.0.60:34936
GET /whoami HTTP/1.1
Host: proxy.penguin-ghost.ts.net
User-Agent: HTTPie/2.6.0
Accept: */*
Accept-Encoding: gzip, deflate
X-Forwarded-For: 10.244.0.62
X-Forwarded-Host: proxy.penguin-ghost.ts.net
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Server: traefik-589c6bf49c-7hhhf
X-Real-Ip: 10.244.0.62
However the HTTPS endpoint returns a default certificate
curl -v -k https://proxy.penguin-ghost.ts.net/whoami minikube
* Host proxy.penguin-ghost.ts.net:443 was resolved.
* IPv6: (none)
* IPv4: 100.109.154.55
* Trying 100.109.154.55:443...
* Connected to proxy.penguin-ghost.ts.net (100.109.154.55) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / X25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
* subject: CN=TRAEFIK DEFAULT CERT
* start date: Nov 17 23:11:34 2024 GMT
* expire date: Nov 17 23:11:34 2025 GMT
* issuer: CN=TRAEFIK DEFAULT CERT
* SSL certificate verify result: self-signed certificate (18), continuing anyway.
* Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://proxy.penguin-ghost.ts.net/whoami
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: proxy.penguin-ghost.ts.net]
* [HTTP/2] [1] [:path: /whoami]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
> GET /whoami HTTP/2
> Host: proxy.penguin-ghost.ts.net
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/2 200
< content-type: text/plain; charset=utf-8
< date: Sun, 17 Nov 2024 23:30:05 GMT
< content-length: 444
<
Hostname: whoami-d4f874664-nmgss
IP: 127.0.0.1
IP: ::1
IP: 10.244.0.47
IP: fe80::28f6:96ff:fe21:201e
RemoteAddr: 10.244.0.60:55506
GET /whoami HTTP/1.1
Host: proxy.penguin-ghost.ts.net
User-Agent: curl/8.7.1
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 10.244.0.62
X-Forwarded-Host: proxy.penguin-ghost.ts.net
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: traefik-589c6bf49c-7hhhf
X-Real-Ip: 10.244.0.62
* Connection #0 to host proxy.penguin-ghost.ts.net left intact
As seen above a self signed certificate is being used.
The following logs indicate an issue with the integration but i am not equipped with right information to make further progress in my diagnosis.
2024-11-17T23:11:34Z ERR Router uses a nonexistent certificate resolver certificateResolver=myresolver routerName=whoami-whoami-759a94f3e4eeb1b8554d@kubernetescrd
10.244.0.62 - - [17/Nov/2024:23:11:37 +0000] "GET /whoami HTTP/1.1" 200 763 "-" "-" 30 "httproute-whoami-whoami-httproute-gw-traefik-v2-traefik-gateway-ep-web-0-6602de0a6a9971408ffb@kubernetesgateway" "http://10.244.0.47:80" 0ms
10.244.0.62 - - [17/Nov/2024:23:11:39 +0000] "GET /whoami HTTP/1.1" 200 763 "-" "-" 31 "httproute-whoami-whoami-httproute-gw-traefik-v2-traefik-gateway-ep-web-0-6602de0a6a9971408ffb@kubernetesgateway" "http://10.244.0.47:80" 0ms
10.244.0.62 - - [17/Nov/2024:23:11:43 +0000] "GET /whoami HTTP/2.0" 200 990 "-" "-" 32 "whoami-whoami-759a94f3e4eeb1b8554d@kubernetescrd" "http://10.244.0.47:80" 0ms
10.244.0.62 - - [17/Nov/2024:23:11:45 +0000] "GET /whoami HTTP/2.0" 200 1016 "-" "-" 33 "whoami-whoami-759a94f3e4eeb1b8554d@kubernetescrd" "http://10.244.0.47:80" 0ms
10.244.0.62 - - [17/Nov/2024:23:11:46 +0000] "GET /whoami HTTP/2.0" 200 1016 "-" "-" 34 "whoami-whoami-759a94f3e4eeb1b8554d@kubernetescrd" "http://10.244.0.47:80" 0ms
10.244.0.62 - - [17/Nov/2024:23:11:47 +0000] "GET /whoami HTTP/2.0" 200 1016 "-" "-" 35 "whoami-whoami-759a94f3e4eeb1b8554d@kubernetescrd" "http://10.244.0.47:80" 0ms
2024-11-17T23:14:13Z ERR Gateway Not Accepted error="1 error occurred:\n\t* Error while retrieving certificate: secret traefik-v2/myresolver does not exist\n\n" gateway=traefik-gateway namespace=traefik-v2 providerName=kubernetesgateway
2024-11-17T23:14:13Z ERR Gateway Not Accepted error="1 error occurred:\n\t* Error while retrieving certificate: secret traefik-v2/myresolver does not exist\n\n" gateway=traefik-gateway namespace=traefik-v2 providerName=kubernetesgateway
2024-11-17T23:14:13Z ERR Gateway Not Accepted error="1 error occurred:\n\t* Error while retrieving certificate: secret traefik-v2/myresolver does not exist\n\n" gateway=traefik-gateway namespace=traefik-v2 providerName=kubernetesgateway
2024-11-17T23:14:13Z ERR Gateway Not Accepted error="1 error occurred:\n\t* Error while retrieving certificate: secret traefik-v2/myresolver does not exist\n\n" gateway=traefik-gateway namespace=traefik-v2 providerName=kubernetesgateway
2024-11-17T23:14:13Z ERR Gateway Not Accepted error="1 error occurred:\n\t* Error while retrieving certificate: secret traefik-v2/myresolver does not exist\n\n" gateway=traefik-gateway namespace=traefik-v2 providerName=kubernetesgateway
2024-11-17T23:14:13Z ERR Gateway Not Accepted error="1 error occurred:\n\t* Error while retrieving certificate: secret traefik-v2/myresolver does not exist\n\n" gateway=traefik-gateway namespace=traefik-v2 providerName=kubernetesgateway
2024-11-17T23:14:13Z ERR Gateway Not Accepted error="1 error occurred:\n\t* Error while retrieving certificate: secret traefik-v2/myresolver does not exist\n\n" gateway=traefik-gateway namespace=traefik-v2 providerName=kubernetesgateway
2024-11-17T23:14:13Z ERR Gateway Not Accepted error="1 error occurred:\n\t* Error while retrieving certificate: secret traefik-v2/myresolver does not exist\n\n" gateway=traefik-gateway namespace=traefik-v2 providerName=kubernetesgateway
2024-11-17T23:24:13Z ERR Gateway Not Accepted error="1 error occurred:\n\t* Error while retrieving certificate: secret traefik-v2/myresolver does not exist\n\n" gateway=traefik-gateway namespace=traefik-v2 providerName=kubernetesgateway
2024-11-17T23:24:13Z ERR Gateway Not Accepted error="1 error occurred:\n\t* Error while retrieving certificate: secret traefik-v2/myresolver does not exist\n\n" gateway=traefik-gateway namespace=traefik-v2 providerName=kubernetesgateway
2024-11-17T23:24:13Z ERR Gateway Not Accepted error="1 error occurred:\n\t* Error while retrieving certificate: secret traefik-v2/myresolver does not exist\n\n" gateway=traefik-gateway namespace=traefik-v2 providerName=kubernetesgateway
2024-11-17T23:24:13Z ERR Gateway Not Accepted error="1 error occurred:\n\t* Error while retrieving certificate: secret traefik-v2/myresolver does not exist\n\n" gateway=traefik-gateway namespace=traefik-v2 providerName=kubernetesgateway
2024-11-17T23:24:13Z ERR Gateway Not Accepted error="1 error occurred:\n\t* Error while retrieving certificate: secret traefik-v2/myresolver does not exist\n\n" gateway=traefik-gateway namespace=traefik-v2 providerName=kubernetesgateway
2024-11-17T23:24:13Z ERR Gateway Not Accepted error="1 error occurred:\n\t* Error while retrieving certificate: secret traefik-v2/myresolver does not exist\n\n" gateway=traefik-gateway namespace=traefik-v2 providerName=kubernetesgateway
2024-11-17T23:24:13Z ERR Gateway Not Accepted error="1 error occurred:\n\t* Error while retrieving certificate: secret traefik-v2/myresolver does not exist\n\n" gateway=traefik-gateway namespace=traefik-v2 providerName=kubernetesgateway
2024-11-17T23:24:13Z ERR Gateway Not Accepted error="1 error occurred:\n\t* Error while retrieving certificate: secret traefik-v2/myresolver does not exist\n\n" gateway=traefik-gateway namespace=traefik-v2 providerName=kubernetesgateway
10.244.0.62 - - [17/Nov/2024:23:27:20 +0000] "GET /whoami HTTP/1.1" 200 453 "-" "-" 36 "httproute-whoami-whoami-httproute-gw-traefik-v2-traefik-gateway-ep-web-0-6602de0a6a9971408ffb@kubernetesgateway" "http://10.244.0.47:80" 1ms
10.244.0.62 - - [17/Nov/2024:23:29:59 +0000] "GET /whoami HTTP/1.1" 200 455 "-" "-" 37 "whoami-whoami-759a94f3e4eeb1b8554d@kubernetescrd" "http://10.244.0.47:80" 0ms
10.244.0.62 - - [17/Nov/2024:23:30:05 +0000] "GET /whoami HTTP/2.0" 200 444 "-" "-" 38 "whoami-whoami-759a94f3e4eeb1b8554d@kubernetescrd" "http://10.244.0.47:80" 0ms
I do not see any secret in the traefik-v2 namespace or in the whole cluster
helm k get secret -A minikube
NAMESPACE NAME TYPE DATA AGE
argocd argocd-initial-admin-secret Opaque 1 41h
argocd argocd-notifications-secret Opaque 0 41h
argocd argocd-redis Opaque 1 41h
argocd argocd-secret Opaque 5 41h
tailscale operator Opaque 4 23h
tailscale operator-oauth Opaque 2 23h
tailscale sh.helm.release.v1.tailscale-operator.v1 helm.sh/release.v1 1 23h
tailscale ts-traefik-dpj9c-0 Opaque 9 38m
traefik-v2 sh.helm.release.v1.traefik.v1 helm.sh/release.v1 1 38m
The full helm template yaml does not contain the secret being referenced in the logs
---
# Source: traefik/templates/rbac/serviceaccount.yaml
kind: ServiceAccount
apiVersion: v1
metadata:
name: traefik
namespace: tailscale
labels:
app.kubernetes.io/name: traefik
app.kubernetes.io/instance: traefik-tailscale
helm.sh/chart: traefik-33.0.0
app.kubernetes.io/managed-by: Helm
annotations:
automountServiceAccountToken: false
---
# Source: traefik/templates/rbac/clusterrole.yaml
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: traefik-tailscale
labels:
app.kubernetes.io/name: traefik
app.kubernetes.io/instance: traefik-tailscale
helm.sh/chart: traefik-33.0.0
app.kubernetes.io/managed-by: Helm
rules:
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- discovery.k8s.io
resources:
- endpointslices
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- secrets
verbs:
- get
- list
- watch
- apiGroups:
- traefik.io
resources:
- ingressroutes
- ingressroutetcps
- ingressrouteudps
- middlewares
- middlewaretcps
- serverstransports
- serverstransporttcps
- tlsoptions
- tlsstores
- traefikservices
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- namespaces
- secrets
- services
- configmaps
verbs:
- get
- list
- watch
- apiGroups:
- discovery.k8s.io
resources:
- endpointslices
verbs:
- list
- watch
- apiGroups:
- gateway.networking.k8s.io
resources:
- backendtlspolicies
- gatewayclasses
- gateways
- grpcroutes
- httproutes
- referencegrants
- tcproutes
- tlsroutes
verbs:
- get
- list
- watch
- apiGroups:
- gateway.networking.k8s.io
resources:
- backendtlspolicies/status
- gatewayclasses/status
- gateways/status
- grpcroutes/status
- httproutes/status
- tcproutes/status
- tlsroutes/status
verbs:
- update
---
# Source: traefik/templates/rbac/clusterrolebinding.yaml
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: traefik-tailscale
labels:
app.kubernetes.io/name: traefik
app.kubernetes.io/instance: traefik-tailscale
helm.sh/chart: traefik-33.0.0
app.kubernetes.io/managed-by: Helm
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: traefik-tailscale
subjects:
- kind: ServiceAccount
name: traefik
namespace: tailscale
---
# Source: traefik/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
name: traefik
namespace: tailscale
labels:
app.kubernetes.io/name: traefik
app.kubernetes.io/instance: traefik-tailscale
helm.sh/chart: traefik-33.0.0
app.kubernetes.io/managed-by: Helm
annotations:
tailscale.com/expose: "true"
tailscale.com/hostname: proxy
spec:
type: ClusterIP
selector:
app.kubernetes.io/name: traefik
app.kubernetes.io/instance: traefik-tailscale
ports:
- port: 80
name: "web"
targetPort: web
protocol: TCP
- port: 443
name: "websecure"
targetPort: websecure
protocol: TCP
---
# Source: traefik/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: traefik
namespace: tailscale
labels:
app.kubernetes.io/name: traefik
app.kubernetes.io/instance: traefik-tailscale
helm.sh/chart: traefik-33.0.0
app.kubernetes.io/managed-by: Helm
annotations:
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: traefik
app.kubernetes.io/instance: traefik-tailscale
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 1
minReadySeconds: 0
template:
metadata:
annotations:
prometheus.io/scrape: "true"
prometheus.io/path: "/metrics"
prometheus.io/port: "9100"
labels:
app.kubernetes.io/name: traefik
app.kubernetes.io/instance: traefik-tailscale
helm.sh/chart: traefik-33.0.0
app.kubernetes.io/managed-by: Helm
spec:
serviceAccountName: traefik
automountServiceAccountToken: true
terminationGracePeriodSeconds: 60
hostNetwork: false
containers:
- image: docker.io/traefik:v3.2.0
imagePullPolicy: IfNotPresent
name: traefik
resources:
readinessProbe:
httpGet:
path: /ping
port: 8080
scheme: HTTP
failureThreshold: 1
initialDelaySeconds: 2
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 2
livenessProbe:
httpGet:
path: /ping
port: 8080
scheme: HTTP
failureThreshold: 3
initialDelaySeconds: 2
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 2
lifecycle:
ports:
- name: "metrics"
containerPort: 9100
protocol: "TCP"
- name: "traefik"
containerPort: 8080
protocol: "TCP"
- name: "web"
containerPort: 8000
protocol: "TCP"
- name: "websecure"
containerPort: 8443
protocol: "TCP"
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
volumeMounts:
- name: data
mountPath: /data
- name: tmp
mountPath: /tmp
args:
- "--global.checknewversion"
- "--global.sendanonymoususage"
- "--entryPoints.metrics.address=:9100/tcp"
- "--entryPoints.traefik.address=:8080/tcp"
- "--entryPoints.web.address=:8000/tcp"
- "--entryPoints.websecure.address=:8443/tcp"
- "--api.dashboard=true"
- "--ping=true"
- "--metrics.prometheus=true"
- "--metrics.prometheus.entrypoint=metrics"
- "--providers.kubernetescrd"
- "--providers.kubernetescrd.allowEmptyServices=true"
- "--providers.kubernetesgateway"
- "--providers.kubernetesgateway.statusaddress.service.name=traefik"
- "--providers.kubernetesgateway.statusaddress.service.namespace=tailscale"
- "--entryPoints.websecure.http.tls=true"
- "--log.level=INFO"
- "--accesslog=true"
- "--accesslog.fields.defaultmode=keep"
- "--accesslog.fields.headers.defaultmode=drop"
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumes:
- name: data
emptyDir: {}
- name: tmp
emptyDir: {}
securityContext:
runAsGroup: 65532
runAsNonRoot: true
runAsUser: 65532
---
# Source: traefik/templates/ingressclass.yaml
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
annotations:
ingressclass.kubernetes.io/is-default-class: "true"
labels:
app.kubernetes.io/name: traefik
app.kubernetes.io/instance: traefik-tailscale
helm.sh/chart: traefik-33.0.0
app.kubernetes.io/managed-by: Helm
name: traefik
spec:
controller: traefik.io/ingress-controller
---
# Source: traefik/templates/gateway.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: traefik-gateway
namespace: tailscale
labels:
app.kubernetes.io/name: traefik
app.kubernetes.io/instance: traefik-tailscale
helm.sh/chart: traefik-33.0.0
app.kubernetes.io/managed-by: Helm
spec:
gatewayClassName: traefik
listeners:
- name: web
port: 8000
protocol: HTTP
allowedRoutes:
namespaces:
from: All
- name: websecure
port: 8443
protocol: HTTPS
allowedRoutes:
namespaces:
from: All
tls:
certificateRefs:
- name: myresolver
---
# Source: traefik/templates/gatewayclass.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: traefik
labels:
app.kubernetes.io/name: traefik
app.kubernetes.io/instance: traefik-tailscale
helm.sh/chart: traefik-33.0.0
app.kubernetes.io/managed-by: Helm
spec:
controllerName: traefik.io/gateway-controller
---
# Source: traefik/templates/ingressroute.yaml
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: traefik-dashboard
namespace: tailscale
annotations:
labels:
app.kubernetes.io/name: traefik
app.kubernetes.io/instance: traefik-tailscale
helm.sh/chart: traefik-33.0.0
app.kubernetes.io/managed-by: Helm
spec:
entryPoints:
- traefik
routes:
- match: PathPrefix(`/dashboard`) || PathPrefix(`/api`)
kind: Rule
services:
- kind: TraefikService
name: api@internal
References: