traefik version
Version: 2.11.10
Codename: mimolette
Go version: go1.23.1
Built: 2024-09-19T10:11:01Z
OS/Arch: linux/amd64
k3s version v1.31.4+k3s1 (a562d090)
go version go1.22.9
I struggle hard getting my k3s/traefik v2 setup to work for my seemingly very simple usecase. This is on a Debian 12 bare metal server.
What works:
- port 80 and 443 answer on desired domain and the requests are correctly routed to my django pod
- port 50051 for a grpc server answers on pod and service level
- the traefik pod shows 50051 as open
- traffic on 80/443 vs on 50051 are dealt with two different ingress routes and host matching (2 subdomains)
What doesn't work:
Port 50051 for a grpc server answers on pod and service level but cannot be reached from the outside world. I tried with nc and grpcurl which work from inside but they cannot connect from outside.
In general I struggle hard to have a consistent grip over the k3s config. Sometimes it seems like it ignores or forgets my config changes. E.g. trying to have debug log by adding these lines to valuesContent of traefik has no effect (checking description of the pod).
Even when I force a rollout with: k3s kubectl rollout restart deployment traefik -n kube-system
log:
level: "DEBUG"
general:
level: "DEBUG"
additionalArguments:
- "--log.level=DEBUG"
When I kill the pod and it is recreated I see only: time="2025-01-23T13:20:23Z" level=info msg="Configuration loaded from flags."
My endpoints:
k3s kubectl get endpoints -n kube-system
NAME ENDPOINTS AGE
kube-dns 10.42.0.3:53,10.42.0.3:53,10.42.0.3:9153 26h
metrics-server 10.42.0.2:10250 26h
traefik 10.42.0.33:80,10.42.0.33:50051,10.42.0.33:443 14h
My traefik config:
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
name: traefik
namespace: kube-system
spec:
valuesContent: |-
image:
repository: "rancher/mirrored-library-traefik"
tag: "2.11.10"
deployment:
replicas: 1
ports:
web:
port: 80
expose:
default: true
websecure:
port: 443
expose:
default: true
grpc:
port: 50051
expose:
default: true
My ingress routes:
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: http-ingressroute
namespace: myapp-staging
spec:
entryPoints:
- web
- websecure
routes:
- match: Host(`staging.myapp.domain.com`)
kind: Rule
services:
- name: django
port: {{ .Values.django.ports.http }} # = 8000
scheme: http
namespace: myapp-staging
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: grpc-ingressroute
namespace: myapp-staging
spec:
entryPoints:
- grpc
routes:
- match: Host(`grpc.myapp.domain.com`)
kind: Rule
services:
- name: grpc
port: {{ .Values.django.ports.grpc }} # = 50051
scheme: h2c
namespace: myapp-staging
My grpc service:
---
apiVersion: v1
kind: Service
metadata:
name: grpc
labels:
app: django
spec:
selector:
app: django
ports:
- port: {{ .Values.django.ports.grpc }}
targetPort: {{ .Values.django.ports.grpc }}
My grpc container:
- name: grpc-server
image: {{ .Values.django.image.main }}
command: ["python", "-u", "grpc_app/server.py"]
workingDir: /app/myapp.git
ports:
- containerPort: {{ .Values.django.ports.grpc }}
volumeMounts:
- name: app-code
mountPath: /app
envFrom:
- configMapRef:
name: django-env-variables
env:
- name: CELERY_BROKER_URL
value: "redis://redis:{{ .Values.redis.port }}/0"
- name: CELERY_RESULT_BACKEND
value: "redis://redis:{{ .Values.redis.port }}/1"
- name: NATS_URL
value: "nats://nats:{{ .Values.nats.port }}"
volumes:
- name: app-code
emptyDir: {}
My traefik pod:
k3s kubectl describe pod traefik -n kube-system
Name: traefik-6c4d764f45-zw2tf
Namespace: kube-system
Priority: 0
Service Account: traefik
Node: myserver/10.5.50.10
Start Time: Thu, 23 Jan 2025 13:27:33 +0100
Labels: app.kubernetes.io/instance=traefik-kube-system
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=traefik
helm.sh/chart=traefik-27.0.201_up27.0.2
pod-template-hash=6c4d764f45
Annotations: prometheus.io/path: /metrics
prometheus.io/port: 9100
prometheus.io/scrape: true
Status: Running
IP: 10.42.0.33
IPs:
IP: 10.42.0.33
Controlled By: ReplicaSet/traefik-6c4d764f45
Containers:
traefik:
Container ID: containerd://408ecb22b243e52e36c3aa2b2c7c3edbe27c1c20bf4ac22f65e8bda771d691e8
Image: rancher/mirrored-library-traefik:2.11.10
Image ID: docker.io/rancher/mirrored-library-traefik@sha256:b19934b3f8e03e213e258e69d57376fd0c4441ea04a518313cab2835fe9b92b3
Ports: 50051/TCP, 9100/TCP, 9000/TCP, 80/TCP, 443/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP
Args:
--global.checknewversion
--global.sendanonymoususage
--entrypoints.grpc.address=:50051/tcp
--entrypoints.metrics.address=:9100/tcp
--entrypoints.traefik.address=:9000/tcp
--entrypoints.web.address=:80/tcp
--entrypoints.websecure.address=:443/tcp
--api.dashboard=true
--ping=true
--metrics.prometheus=true
--metrics.prometheus.entrypoint=metrics
--providers.kubernetescrd
--providers.kubernetesingress
--entrypoints.websecure.http.tls=true
State: Running
Started: Thu, 23 Jan 2025 13:27:33 +0100
Ready: True
Restart Count: 0
Liveness: http-get http://:9000/ping delay=2s timeout=2s period=10s #success=1 #failure=3
Readiness: http-get http://:9000/ping delay=2s timeout=2s period=10s #success=1 #failure=1
Environment:
POD_NAME: traefik-6c4d764f45-zw2tf (v1:metadata.name)
POD_NAMESPACE: kube-system (v1:metadata.namespace)
Mounts:
/data from data (rw)
/tmp from tmp (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5mvk6 (ro)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
data:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
tmp:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
kube-api-access-5mvk6:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 18m default-scheduler Successfully assigned kube-system/traefik-6c4d764f45-zw2tf to myserver
Normal Pulled 18m kubelet Container image "rancher/mirrored-library-traefik:2.11.10" already present on machine
Normal Created 18m kubelet Created container traefik
Normal Started 18m kubelet Started container traefik