Opentelemetry & traceID

Traefik Traefik v3 (latest)
1

Hello!

I don't see a traceID or traceparent in the log line generated by traefik. How can I generate a traceID if it doesn't exist yet? Is there an option?
Also, it would be amazing if there was a way to return this traceID to the front user as a header, so they can report to our systems the trace ID in case of errors or weird behaviour.

Here is an example of LOG request I get on my traefik instance.

{
  "ClientAddr": "10.x.0.x:x",
  "ClientHost": "10.x.0.x",
  "ClientPort": "x",
  "ClientUsername": "-",
  "DownstreamContentSize": 19,
  "DownstreamStatus": 404,
  "Duration": 162044,
  "GzipRatio": 0,
  "OriginContentSize": 0,
  "OriginDuration": 0,
  "OriginStatus": 0,
  "Overhead": 162044,
  "RequestAddr": "<redacted>",
  "RequestContentSize": 0,
  "RequestCount": 195,
  "RequestHost": "<redacted>",
  "RequestMethod": "GET",
  "RequestPath": "/",
  "RequestPort": "-",
  "RequestProtocol": "HTTP/2.0",
  "RequestScheme": "https",
  "RetryAttempts": 0,
  "StartLocal": "2025-01-17T18:45:57.701837007Z",
  "StartUTC": "2025-01-17T18:45:57.701837007Z",
  "TLSCipher": "TLS_AES_128_GCM_SHA256",
  "TLSVersion": "1.3",
  "entryPointName": "websecure",
  "level": "info",
  "msg": "",
  "time": "2025-01-17T18:45:57Z"
}

Config:

  • Kubernetes
  • Traefik helm chart 32.1.1

Helm values:

...
        logs:
          general:
            level: INFO
          access:
            # -- To enable access logs
            enabled: true
            # -- Set [access log format](https://doc.traefik.io/traefik/observability/access-logs/#format)
            format: json
            # -- Enables accessLogs for internal resources. Default: false.
            addInternals: true
            # -- Set [filtering](https://docs.traefik.io/observability/access-logs/#filtering)
            filters:  # @schema additionalProperties: false
              # -- Set statusCodes, to limit the access logs to requests with a status codes in the specified range
              statuscodes: "400-499,500-599"
              # -- Set retryAttempts, to keep the access logs when at least one retry has happened
              retryattempts: true
              # -- Set minDuration, to keep access logs when requests take longer than the specified duration
              minduration: "2000ms"

Best regards,

2

Did you enable Traefik tracing?

3

Yes! I forgot to include it above, here is my tracing config (helm chart values):

...
        tracing:
          otlp:
            enabled: true
            http:
              enabled: true
              endpoint: "http://grafana-k8s-monitoring-grafana-agent.monitoring.svc:4318/v1/traces"
...
4

I just tried locally to see if it could be an issue with my Helm/Kubernetes configuration, but I couldn't make it work locally either:

docker run --rm -it \
  --name traefik \
  -p 80:80 \
  -p 443:443 \
  -p 8080:8080 \
  -v /var/run/docker.sock:/var/run/docker.sock \
  traefik:latest \
  --api.insecure=true \
  --providers.docker \
  --entrypoints.web.address=:80 \
  --entrypoints.websecure.address=:443 \
  --log.level=DEBUG \
  --accesslog=true --accesslog.format=json \
  --tracing.otlp=true --tracing.otlp.http=true --api.dashboard=true  --ping=true  --accesslog.addinternals

and curl -v localhost:80 shows:

// logs in traefik
{
  "ClientAddr": "192.168.215.1:40570",
  "ClientHost": "192.168.215.1",
  "ClientPort": "40570",
  "ClientUsername": "-",
  "DownstreamContentSize": 19,
  "DownstreamStatus": 404,
  "Duration": 144833,
  "GzipRatio": 0,
  "OriginContentSize": 0,
  "OriginDuration": 0,
  "OriginStatus": 0,
  "Overhead": 144833,
  "RequestAddr": "localhost",
  "RequestContentSize": 0,
  "RequestCount": 1,
  "RequestHost": "localhost",
  "RequestMethod": "GET",
  "RequestPath": "/",
  "RequestPort": "-",
  "RequestProtocol": "HTTP/1.1",
  "RequestScheme": "http",
  "RetryAttempts": 0,
  "StartLocal": "2025-01-17T22:30:39.819427032Z",
  "StartUTC": "2025-01-17T22:30:39.819427032Z",
  "entryPointName": "web",
  "level": "info",
  "msg": "",
  "time": "2025-01-17T22:30:39Z"
}

// curl logs
* Host localhost:80 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:80...
* Connected to localhost (::1) port 80
> GET / HTTP/1.1
> Host: localhost
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 404 Not Found
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Fri, 17 Jan 2025 22:32:47 GMT
< Content-Length: 19
<
404 page not found
* Connection #0 to host localhost left intact

So I am not sure how to enable trace ID in the logs, and respond it to the client.