Hi everyone,
We’ve been experimenting with the new OpenTelemetry logging support in Traefik (enabled via the experimental.otlpLogs flag), specifically using the gRPC exporter to send logs to our OpenTelemetry Collector.
Despite configuring everything according to the official docs, we’re not seeing any logs arrive at the collector. Here’s what we’ve tried:
- Enabled
experimental.otlpLogs: true
- Configured the gRPC endpoint under
log.otlp.grpc.endpoint
- Verified that the collector is listening on port
4317 with the OTLP gRPC receiver enabled
- Confirmed network connectivity between Traefik and the collector
Still, no logs are being received.
i'm using the latest version of traefik in a kubernetes cluster.
Has anyone successfully gotten this working? Are there known limitations or additional steps required for gRPC-based OTLP logging in Traefik?
We’d love to hear your experiences or suggestions—and happy to collaborate on debugging this further!
Thanks in advance!
Maybe share your full static config, not just a single line.
here is the deployment file of traefik :
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
meta.helm.sh/release-name: traefik
meta.helm.sh/release-namespace: traefik
labels:
app.kubernetes.io/instance: traefik-traefik
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: traefik
helm.sh/chart: traefik-35.3.0
name: traefik
namespace: traefik
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app.kubernetes.io/instance: traefik-traefik
app.kubernetes.io/name: traefik
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
type: RollingUpdate
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "9100"
prometheus.io/scrape: "true"
creationTimestamp: null
labels:
app.kubernetes.io/instance: traefik-traefik
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: traefik
helm.sh/chart: traefik-35.3.0
spec:
automountServiceAccountToken: true
containers:
- args:
- --global.checknewversion
- --global.sendanonymoususage
- --entryPoints.metrics.address=:9100/tcp
- --entryPoints.traefik.address=:8080/tcp
- --entryPoints.web.address=:8000/tcp
- --entryPoints.websecure.address=:8443/tcp
- --api.dashboard=true
- --ping=true
- --metrics.addinternals
- --metrics.prometheus=true
- --metrics.prometheus.entrypoint=metrics
- --metrics.otlp=true
- --metrics.otlp.addEntryPointsLabels=true
- --metrics.otlp.addRoutersLabels=true
- --metrics.otlp.addServicesLabels=true
- --metrics.otlp.grpc=true
- --metrics.otlp.grpc.endpoint=otel-collector.default.svc.cluster.local:4317
- --metrics.otlp.grpc.insecure=true
- --tracing.addinternals
- --tracing.serviceName=traefik
- --tracing.resourceAttributes.fromProxy=
- --tracing.otlp=true
- --tracing.otlp.grpc=true
- --tracing.otlp.grpc.endpoint=otel-collector.default.svc.cluster.local:4317
- --tracing.otlp.grpc.insecure=true
- --experimental.fastProxy
- --providers.kubernetescrd
- --providers.kubernetescrd.allowEmptyServices=true
- --experimental.kubernetesgateway
- --providers.kubernetesgateway
- --providers.kubernetesgateway.statusaddress.service.name=traefik
- --providers.kubernetesgateway.statusaddress.service.namespace=traefik
- --providers.kubernetesgateway.experimentalchannel=true
- --entryPoints.websecure.http.tls=true
- --log.format=json
- --log.level=INFO
- --log.otlp.grpc.tls.insecureSkipVerify=true
- --experimental.otlpLogs=true
- --log.otlp.grpc=true
- --log.otlp.grpc.endpoint=otel-collector.default.svc.cluster.local:4317
- --accesslog=true
- --accesslog.format=json
- --accesslog.fields.defaultmode=keep
- --accesslog.fields.headers.defaultmode=keep
env:
- name: TRAEFIK_EXPERIMENTAL_OTLPLOGS
value: "true"
- name: TRAEFIK_LOG_OTLP_GRPC_ENDPOINT
value: "otel-collector.default.svc.cluster.local:4317"
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: docker.io/traefik:v3.4.0
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /ping
port: 8080
scheme: HTTP
initialDelaySeconds: 2
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 2
name: traefik
ports:
- containerPort: 9100
name: metrics
protocol: TCP
- containerPort: 8080
name: traefik
protocol: TCP
- containerPort: 8000
name: web
protocol: TCP
- containerPort: 8443
name: websecure
protocol: TCP
readinessProbe:
failureThreshold: 1
httpGet:
path: /ping
port: 8080
scheme: HTTP
initialDelaySeconds: 2
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 2
resources: {}
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /data
name: data
- mountPath: /tmp
name: tmp
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
runAsGroup: 65532
runAsNonRoot: true
runAsUser: 65532
serviceAccount: traefik
serviceAccountName: traefik
terminationGracePeriodSeconds: 60
volumes:
- emptyDir: {}
name: data
- emptyDir: {}
name: tmp
When you think it’s a bug, then you can report it to the developers at Traefik Github.
So i remove log.format json and use otlp.http instead of grpc then i'm getting some logs but i don't have the same level of details compare to stdout.
So i have tried various settings and when otlp is enabled i'm not getting the accesslog. if i add the accesslog settings then i'm not getting otlp logs anymore.