Prometheus Authentication in File

Traefik Traefik v2
,
1

Hi all,

First of all i'd like to thank the creators for building Traefik 2.0. So far I really love it!

So i'm running into an issue which i'm unable to find information about. I am using the file configuration (traefik.yml). I'm trying to get a prometheus endpoint up and running with auth on it. I can't seem to find what the service of prometheus is called (like api@internal for the api).

Does anyone know how to add authentication to prometheus and/or how the prometheus service is called?

Config attached:

global:
  checkNewVersion: true
  sendAnonymousUsage: true
entryPoints:
  traefik:
    address: :8082
  https:
    address: :443
  prometheus:
    address: :8083

providers:
  providersThrottleDuration: 2s
  file:
    filename: "/etc/traefik/traefik.yml"
api:
  dashboard: true
metrics:
  prometheus:
    entryPoint: prometheus
log:
  level: DEBUG
  format: json
  filePath: "/logs/traefik.log.json"
accessLog:
  format: json
  filePath: "/logs/accessLogs.log.json"

certificatesResolvers:
  letsencrypt:
    acme:
      email: some-email@something.com
      storage: "/etc/traefik/acme.json"
      dnsChallenge:
        provider: cloudflare
        delayBeforeCheck: 0

http:
  routers:
    api:
      rule: PathPrefix(`/`)
      service: api@internal
      entryPoints:
      - traefik
      middlewares:
      - apiAuth
      - compress
    prometheus:
      rule: PathPrefix(`/`)
      service: prometheus@internal
      entryPoints:
      - prometheus
      middlewares:
      - prometheusAuth
      - compress
    home:
      rule: "Host(`home.somedomain.com`)"
      tls:
        certResolver: letsencrypt
      service: home@file
      entryPoints:
      - https
      middlewares:
      - referrerPolicy
      - compress
      - ipWhitelisting
    firstservice:
      rule: "Host(`firstservice.somedomain.com`)"
      tls:
        certResolver: letsencrypt
      service: firstservice
      entryPoints:
      - https
      middlewares:
      - referrerPolicy
      - compress
      - ipWhitelisting
    otherservice:
      rule: "Host(`otherservice.somedomain.com`)"
      tls:
        certResolver: letsencrypt
      service: otherservice
      entryPoints:
      - https
      middlewares:
      - referrerPolicy
      - compress
      - ipWhitelisting

  middlewares:
    referrerPolicy:
      headers:
        referrerPolicy: "no-referrer"
    apiAuth:
      basicAuth:
        users:
          - "username:$apr1$<removed>"
    prometheusAuth:
      basicAuth:
        users:
          - "anotherusername:$apr1$<removed>"
    compress:
      compress: {}
    ipWhitelisting:
      ipWhiteList:
        sourceRange:
        - 173.245.48.0/20
        - 103.21.244.0/22
        - 103.22.200.0/22
        - 103.31.4.0/22
        - 141.101.64.0/18
        - 108.162.192.0/18
        - 190.93.240.0/20
        - 188.114.96.0/20
        - 197.234.240.0/22
        - 198.41.128.0/17
        - 162.158.0.0/15
        - 104.16.0.0/12
        - 172.64.0.0/13
        - 131.0.72.0/22
        - 10.0.0.0/8

  services:
    home:
      loadBalancer:
        passHostHeader: true
        servers:
          - url: http://10.0.7.28:1234
    firstservice:
      loadBalancer:
        passHostHeader: true
        servers:
          - url: "http://10.0.7.9:5678/"
    otherservice:
      loadBalancer:
        passHostHeader: true
        servers:
          - url: http://10.0.7.31

The "prometheus@internal" part is where it goes wrong. Somehow thats incorrect.

Thank you in advance!

2

Hi

I think you missed this part :

how to enable prometheus

metrics:
  prometheus: {}
  buckets:
      - 0.1
      - 0.3
      - 1.2
      - 5.0
 addEntryPointsLabels: true
 addServicesLabels: true 
 entryPoint: prometheus
3

Thanks for your reply!

I changed my prometheus config to this:

metrics:
  prometheus:
    entryPoint: prometheus
    buckets:
      - 0.1
      - 0.3
      - 1.2
      - 5.0
    addEntryPointsLabels: true
    addServicesLabels: true

Also the entrypoint is in place:

entryPoints:
  traefik:
    address: :8082
  https:
    address: :443
  prometheus:
    address: :8083

Then i added the prometheusAuth middleware:

  middlewares:
    prometheusAuth:
      basicAuth:
        users:
          - "prometheus:$apr1$<securestring>"

Last but not least the router is in place:

    prometheus:
      rule: PathPrefix(`/`)
      service: prometheus@internal
      entryPoints:
      - prometheus
      middlewares:
      - prometheusAuth
      - compress

I dont get any errors in the config, but the prometheus endpoint doesnt have any authentication on it (i expect it to show a basic auth popup, but it doesnt).

Am i doing something wrong?

EDIT: In the dashboard i do see this:

the service "prometheus@internal" does not exist
4

The scope @internal can only be used for api and rest.

Prometheus is not a service, you need to create manually one.

5

Alright let me try to rephrase the underlying question then: How do I put authentication on the prometheus metrics page from Traefik like I was able to in Traefik 1.x? (i'm talking about the /metrics page)

6

I have not tried it, but from the conversation above, it looks, like what you've done might have worked if you just created Prometheus manually, instead of trying to use prometheus@internal. It sounds like this is the changes that is needed to get you there.

7

How does that work? Im trying to expose the metrics page (the /metrics endpoint) in Traefik 2.0.

8

I would image that the service would target traefik itself and you'll be using the /metrics endpoint from it.

9

Alright, but how do i expose the /metrics endpoint on Traefik? And how do i add authentication to that endpoint? In Traefik 1.x this was possible, but i can't figure out how to do it on 2.x.

10

Exactly the same way as you expose any other end point on any other web application in traefik. What seems to be the problem?

11

There is a built in /metrics page as described here:
https://docs.traefik.io/observability/metrics/prometheus/

The problem i'm having is that i'm trying to enable basic auth on this /metrics endpoint, but I can't figure out how to do this. This used to be possible in Traefik 1.x, but it appears this isn't supported in Traefik 2.x.

12

I understand that, I'm just not sure which part you need help with. You simply write your config the way your wrote but instead of using prometheus@internal you define a service for metrics and point to that service from the router. You do this exactly the same way as you would do with any other application you route a traefik request to. I think we are going in circles here, but I don't know how to progress since I don't understand what's not clear.

13

Here is a minimal example for exposing Promehteus with auth. No swarm, no TLS:

docker-compose.yaml 
version: "3.3"

services:

  traefik:
    image: "traefik:v2.0.4"
    container_name: "traefik"
    ports:
      - "80:80"
      - "8080:8080"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "./traefik.toml:/traefik.toml"
    labels:
      - "traefik.enable=true"       
      - "traefik.http.routers.metrics.entrypoints=http"
      - "traefik.http.routers.metrics.rule=Path(`/metrics`)"
      - "traefik.http.routers.metrics.service=metrics"
      - "traefik.http.routers.metrics.middlewares=auth"
      - "traefik.http.middlewares.auth.basicauth.users=test:$$apr1$$H6uskkkW$$IgXLP6ewTrSuBkTrqE8wj/,test2:$$apr1$$d9hr9HBB$$4HxwgUir3HP4EsggP/QNo0"     
      - "traefik.http.services.metrics.loadbalancer.server.port=5443"
traefik.toml 
[entryPoints.http]
address = ":80"
[entryPoints.metrics]
address = ":5443"

[api]
insecure = true

[log]
level = "debug"

[accessLog]

[metrics.prometheus]
addEntryPointsLabels = true
addServicesLabels = true
entryPoint = "metrics"

[providers.docker]
exposedByDefault = false

Here you expose /metrics on 5443, then you create a service to point to that port, a router to point to that service, and then you can access it at http://yourhost/metrics. You will have provide credentials such as test:test.

Hope this helps.

14

Hi! I tried to reproduce this by translating this to a YAML structure, but i dont get a basic auth popup. So i can enter the /metrics anonymously. Please see my current config attached:

global:
  checkNewVersion: true

serversTransport:
  insecureSkipVerify: true

entryPoints:
  https:
    address: :443
  metrics:
    address: :8083

providers:
  providersThrottleDuration: 2s
  file:
    filename: "/etc/traefik/traefik.yml"

metrics:
  prometheus:
    entryPoint: metrics
    buckets:
      - 0.1
      - 0.3
      - 1.2
      - 5.0
    addEntryPointsLabels: true
    addServicesLabels: true

log:
  level: WARN
  format: json
  filePath: "/logs/traefik.log.json"
accessLog:
  format: json
  filePath: "/logs/accessLogs.log.json"

http:
  routers:
    metrics:
      rule: PathPrefix(`/metrics`)
      service: metrics
      entryPoints:
      - metrics
      middlewares:
      - metricsAuth

  middlewares:
    metricsAuth:
      basicAuth:
        users:
          - "prometheus:<redacted>"

  services:
    metrics:
      loadbalancer:
        server:
          port: 8083

Do you see any errors I made? Or know why this isn't working as expected?

15

You should not mix dynamic and static configuration

16

I'm sorry, what do you mean by static and dynamic configuration? Above configuration is the only configuration i'm using (I dont use a docker backend or something in that liking). Also i don't add labels to containers.

17

I'm using those in the same sense the documentation uses those terms. Read here: https://docs.traefik.io/getting-started/configuration-overview/

18

So I can't use entrypoints (static) AND routers (dynamic) for example? I have to choose between the two? This really confuses me. How does traefik work without entrypoints?

19

No, of course you have to use both, just do not mix them together. Perhaps this link: https://docs.traefik.io/providers/file/ will help?

20

I'm totally lost now. Can you provide an example of what i'm trying to achieve (a metrics page with auth) using the YAML filestructure (without docker compose or labels on dockers etc)?

From the page you referenced, I can't find out what I seem to be doing wrong.

PS. Sorry for me not understanding this, i'm relatively new to Traefik.