Can you help me getting traefik metrics ? (version 2.2.8)

jbtruffault · April 9, 2021, 8:54am

Dears,
I am really new to Kubernetes / Traefik / Prometheus. I achieved to visualize a lot of metrics about my cluster with Prometheus / Grafana, and now I'd like to get metrics from Traefik.

Here are some details about my configuration:

I installed Traefik through helm, each time I want to upgrade the configuration, I use the following command:

$ helm upgrade -n kube-system traefik traefik/traefik --values values.yml

my file values.yml:

additionalArguments:
  - "--providers.kubernetesIngress.ingressClass=traefik-cert-manager"
  - "--ping"
  - "--log.level=DEBUG"
  - "--log.format=json"
  - "--metrics.prometheus"
  - "--metrics.prometheus.buckets=0.100000, 0.300000, 1.200000, 5.000000"
  - "--metrics.prometheus.addEntryPointsLabels=true"
  - "--metrics.prometheus.addServicesLabels=true"

My traefik dashboard shows that it seems ready to share metrics:

For checking:

$ kubectl describe pod/traefik-64686bd987-9ws5r -n kube-system

Name:         traefik-64686bd987-9ws5r
Namespace:    kube-system
Priority:     0
Node:         ip-172-31-45-205.us-east-2.compute.internal/172.31.45.205
Start Time:   Fri, 09 Apr 2021 08:43:35 +0000
Labels:       app.kubernetes.io/instance=traefik
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=traefik
              helm.sh/chart=traefik-9.1.1
              pod-template-hash=64686bd987
Annotations:  kubernetes.io/psp: eks.privileged
Status:       Running
IP:           172.31.41.59
IPs:
  IP:           172.31.41.59
Controlled By:  ReplicaSet/traefik-64686bd987
Containers:
  traefik:
    Container ID:  docker://19f366b0441c40d67b227747ae93ddc849c1bc392b22e84005f7db4d5f9b5c49
    Image:         traefik:2.2.8
    Image ID:      docker-pullable://traefik@sha256:f5af5a5ce17fc3e353b507e8acce65d7f28126408a8c92dc3cac246d023dc9e8
    Ports:         9000/TCP, 8000/TCP, 8443/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Args:
      --global.checknewversion
      --global.sendanonymoususage
      --entryPoints.traefik.address=:9000/tcp
      --entryPoints.web.address=:8000/tcp
      --entryPoints.websecure.address=:8443/tcp
      --api.dashboard=true
      --ping=true
      --providers.kubernetescrd
      --providers.kubernetesingress
      --providers.kubernetesIngress.ingressClass=traefik-cert-manager
      --ping
      --log.level=DEBUG
      --log.format=json
      --metrics.prometheus=true
      --metrics.prometheus.buckets=0.100000, 0.300000, 1.200000, 5.000000
      --metrics.prometheus.addEntryPointsLabels=true
      --metrics.prometheus.addServicesLabels=true
    State:          Running
      Started:      Fri, 09 Apr 2021 08:43:37 +0000
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:9000/ping delay=10s timeout=2s period=10s #success=1 #failure=3
    Readiness:      http-get http://:9000/ping delay=10s timeout=2s period=10s #success=1 #failure=1
    Environment:    <none>
    Mounts:
      /data from data (rw)
      /tmp from tmp (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from traefik-token-gktx2 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  traefik-token-gktx2:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  traefik-token-gktx2
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  3m10s  default-scheduler  Successfully assigned kube-system/traefik-64686bd987-9ws5r to ip-172-31-45-205.us-east-2.compute.internal
  Normal  Pulled     3m9s   kubelet            Container image "traefik:2.2.8" already present on machine
  Normal  Created    3m9s   kubelet            Created container traefik
  Normal  Started    3m9s   kubelet            Started container traefik

Nevertheless, prometheus cannot reach the metrics:

Here is an extract of my prometheus.yml file:

 42     scrape_configs:
 43       - job_name: 'traefik'
 44         static_configs:
 45           - targets:
 46             - traefik.kube-system.svc.cluster.local:9000

At this point, I tried to search deeper on the internet for solution, I found this post: Capture Traefik Metrics for Apps on Kubernetes with Prometheus. It recommands to create a file traefik-service-monitor.yaml, so I created it and applied it with kubectl

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name:  traefik
  namespace: default
  labels:
    app: traefik
    release: prometheus-stack
spec:
  jobLabel: traefik-metrics
  selector:
    matchLabels:
      app.kubernetes.io/instance: traefik
      app.kubernetes.io/name: traefik-dashboard
  namespaceSelector:
    matchNames:
    - kube-system
  endpoints:
  - port: traefik
    path: /metrics

To stick the tutorial, I also created this file: traefik-dashboard-service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: traefik-dashboard
  namespace: kube-system
  labels:
    app.kubernetes.io/instance: traefik
    app.kubernetes.io/name: traefik-dashboard
spec:
  type: ClusterIP
  ports:
  - name: traefik
    port: 9000
    targetPort: traefik
    protocol: TCP
  selector:
    app.kubernetes.io/instance: traefik
    app.kubernetes.io/name: traefik

After all that, I still have the same problem: prometheus cannot reach traefik metrics.

Can someone help me with all that ? Any idea would be really appretiated !

Regards

mariusrugan · April 10, 2021, 11:11am

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name:  traefik
  namespace: observability
  labels:
    app: traefik
    release: kube-prometheus-stack
spec:
  jobLabel: traefik-metrics
  selector:
    matchLabels:
      app.kubernetes.io/instance: traefik
      app.kubernetes.io/name: traefik
  namespaceSelector:
    matchNames:
    - network
  endpoints:
    - port: traefik
      targetPort: 9000
      path: /metrics
      interval: 10s
      honorLabels: true

i run traefik in network namespace and the service monitor is in the observability namespace. matchLabels / namespace selector is the key to get it running.

jbtruffault · April 10, 2021, 10:02pm

Thank you for your answer !!
I don't really understand what is the purpose of ServiceMonitor I think that my error comes from my Traefik configuration: I can't even get the metrics by using curl:

curl http://traefik.kube-system.svc.cluster.local:9000/metrics

it just run for a moment in void then stops...

mariusrugan · April 11, 2021, 1:59pm

you should expose port 9000
how does the traefik service look like?

network         traefik                                          LoadBalancer   A.B.C.D   X.Y.Z.W   9000:30547/TCP,80:31624/TCP,443:32083/TCP   13d

mariusrugan · April 11, 2021, 2:02pm

the purpose of the ServiceMonitor is to enable / and / signal Prometheus that a port is ready to be "visited" to collect metrics

jbtruffault · April 11, 2021, 3:45pm

The problem might be here: I probably misunderstood something about the exposed ports in kubernetes.

With the following command:
kubectl describe pod/traefik-64686bd987-9ws5r -n kube-system

I get port 9000, 8000 and 8443 opened:

Containers:
  traefik:
    Container ID:  docker://19f366b0441c40d67b227747ae93ddc849c1bc392b22e84005f7db4d5f9b5c49
    Image:         traefik:2.2.8
    Image ID:      docker-pullable://traefik@sha256:f5af5a5ce17fc3e353b507e8acce65d7f28126408a8c92dc3cac246d023dc9e8
    Ports:         9000/TCP, 8000/TCP, 8443/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    ...

But with this one I only get 80 and 443 opened:

$ kubectl get svc -n kube-system
NAME                 TYPE           CLUSTER-IP       EXTERNAL-IP                                                              PORT(S)                      AGE
traefik              LoadBalancer   10.100.111.101   .....amazonaws.com   80:32707/TCP,443:31982/TCP   26d

How can I change this by using helm ?

mariusrugan · April 11, 2021, 8:19pm

in the helm chart you have ports: section,

github.com

traefik/traefik-helm-chart/blob/master/traefik/values.yaml

# Default values for Traefik
image:
  name: traefik
  # defaults to appVersion
  tag: ""
  pullPolicy: IfNotPresent

#
# Configure the deployment
#
deployment:
  enabled: true
  # Can be either Deployment or DaemonSet
  kind: Deployment
  # Number of pods of the deployment (only applies when kind == Deployment)
  replicas: 1
  # Additional deployment annotations (e.g. for jaeger-operator sidecar injection)
  annotations: {}
  # Additional deployment labels (e.g. for filtering deployment by custom labels)
  labels: {}

This file has been truncated. show original

    image:
      name: traefik
      tag: v2.4.8
      pullPolicy: IfNotPresent
      repository: traefik/traefik
    ports:
      traefik:
        port: 9000
        expose: true

ucturbo105 · April 12, 2021, 3:57pm

This complementary process helped project failing scenarios and the systems involved. ... For the next step, we download a version of the httpbin service from release pages ... The aim is to compile the code and get a binary artifact from it . ... You can configure Traefik logs by adding log configuration.

jbtruffault · April 12, 2021, 10:15pm

Thank you, I understand this solution.
Now my point is: after this configuration, the port 9000 will be open to internet. And for security reasons, I don't want that.
Is there a way to open the port only for my cluster ?
Regards

system · April 21, 2021, 8:06pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Monitor traefik with prometheus all on k8s Traefik v2 kubernetes-ingress , metrics , tracing	4	1702	October 31, 2021
Monitor Traefik v2 Ingress controller with Prometheus by host Traefik v2 kubernetes-ingress , metrics	2	698	January 16, 2024
Prometheus metrics routing Traefik v2 kubernetes-ingress , metrics	2	981	April 8, 2021
Missing backend metrics Traefik v2 metrics	6	3286	April 28, 2020
Metrics in Traefik V2 with Prometheus Traefik v2 docker , docker-swarm , metrics	1	3672	November 5, 2019

Can you help me getting traefik metrics ? (version 2.2.8)

Related topics