Routing issues with prometheus + docker + traefik

Traefik Traefik v2
1

Hi!

I recently moved all my configuration into my docker-compose file but I'm unable to route to my prometheus service (can't access prometheus webui @ :9090 /graphs, etc etc). I'm still learning all this, but are swarms even needed for this or can i simply hook it up to traefik somehow and let it do its thing? thanks!

docker-compose.yml

version: "3.7"

services:
  traefik:
    image: "traefik:v2.1.1"
    container_name: "traefik"
    restart: "always"

    ports: # used ports
      - "80:80"
      - "443:443"
      - "8080:8080"
      - "8082:8082"
      - "9090:9090"

    volumes: # mounts
      - "/var/run/docker.sock:/var/run/docker.sock:ro" # listen for docker events (read only)
      - "./traefik/acme.json:/acme.json"               # save traefik SSL cert data here
      - "./traefik/logs:/logs"                         # save log files here

    command:
      # traefik options
      - "--log.level=ERROR"
      - "--log.format=json"
      - "--log.filePath=logs/traefik.log"
      - "--api=true"
      - "--api.dashboard=true"
      - "--api.debug=false"
      - "--ping=true"
      # - "--metrics.prometheus=true" # NOTE: isn't this redundent if we're running prometheus in a container? it places two different routes and doesn't work anyway.
      # - "--metrics.prometheus.buckets=0.1,0.3,1.2,5.0"
      - "--ping.entryPoint=ping"

      # docker provider
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--providers.docker.watch=true"

      # entry points
      - "--entrypoints.web.address=:80"
      - "--entrypoints.web-secure.address=:443"
      - "--entryPoints.ping.address=:8082"
      - "--entryPoints.prometheus.address=:9090"

      # ssl options
      - "--certificatesresolvers.ssl-resolver.acme.email=admin@site.net"
      - "--certificatesresolvers.ssl-resolver.acme.storage=acme.json"
      - "--certificatesresolvers.ssl-resolver.acme.tlschallenge=true"

    labels:
      # enable service
      - "traefik.enable=true"

      # redirect all HTTP requests to HTTPS
      - "traefik.http.routers.http-catchall.rule=hostregexp( `{host:.+}` )"
      - "traefik.http.routers.http-catchall.entrypoints=web"
      - "traefik.http.routers.http-catchall.middlewares=https-redirect@docker"

      # dashboard
      - "traefik.http.routers.api.rule=Host( `traefik.site.net` )"
      - "traefik.http.routers.api.service=api@internal"
      - "traefik.http.routers.api.tls.certresolver=ssl-resolver"
      - "traefik.http.routers.api.entrypoints=web-secure"
      - "traefik.http.routers.api.middlewares=auth"
      
      # prometheus
      - "traefik.http.routers.prometheus.rule=Host( `prometheus.site.net` )"
      - "traefik.http.routers.prometheus.service=prometheus@docker"
      - "traefik.http.routers.prometheus.tls.certresolver=ssl-resolver"
      - "traefik.http.routers.prometheus.entrypoints=web-secure"
      - "traefik.http.routers.prometheus.middlewares=auth"

      # # main website
      # - "traefik.http.routers.main.rule=Host( `site.net` )"
      # - "traefik.http.routers.main.tls.certresolver=ssl-resolver"
      # - "traefik.http.routers.main.entrypoints=web-secure"

      # auth middleware
      - "traefik.http.middlewares.auth.basicauth.users=admin:$$apr1$$.8iKBYyx$$QMe2EbLlYtjCNdjgGHyua/"

      # HTTPS redirect middleware
      - "traefik.http.middlewares.https-redirect.redirectscheme.scheme=https"

  prometheus:
    image: "prom/prometheus"
    restart: "always"
    user: root

    volumes: # mounts
      - "./prometheus:/etc/prometheus"
      - "./prometheus/data:/prometheus"
      - "./prometheus/prometheus.yml:/prometheus.yml"

    command:
      - "--config.file=/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--web.console.libraries=/etc/prometheus/console_libraries"
      - "--web.console.templates=/etc/prometheus/consoles"
      
    labels:
      # enable service
      - "prometheus.enable=true"
      
      # route internally
      - "traefik.http.routers.prometheus.rule=Host(`prometheus.localhost`)"
      - "traefik.http.routers.prometheus.service=prometheus"
      - "traefik.http.services.prometheus.loadbalancer.server.port=9090"

1553×770

Any tips on how i can expose prometheus webui and data while being protected by my middleware? Thanks again!

2

Hello,

Traefik serves the data for Prometheus and the Prometheus container consume those data.
It's 2 different things.

You have to enable Prometheus on Traefik to expose the data.

services:
  traefik:
    image: "traefik:v2.1.1"
    # ...
    command:
    # ...
    - --metrics.prometheus=true

Then the Prometheus data are exposed on the port 8080.

If you want to add a basic auth and expose the endpoint on 443 (instead of 8080), you have to enable the manualRouting and add a router for the service prometheus@internal:

services:
  traefik:
    image: "traefik:v2.1.1"
    # ...
    command:
    # ...
    - --metrics.prometheus=true
    - --metrics.prometheus.manualrouting=true
    labels:
      - "traefik.enable=true"
      # ...

      # prometheus (data)
      - "traefik.http.routers.prometheus.rule=Host(`prometheus.site.net`)"
      - "traefik.http.routers.prometheus.service=prometheus@internal"
      - "traefik.http.routers.prometheus.tls.certresolver=ssl-resolver"
      - "traefik.http.routers.prometheus.entrypoints=web-secure"
      - "traefik.http.routers.prometheus.middlewares=auth"
      
      # auth middleware
      - "traefik.http.middlewares.auth.basicauth.users=admin:$$apr1$$.8iKBYyx$$QMe2EbLlYtjCNdjgGHyua/"

After that you have to configure Prometheus to consume data from traefik (exposed on prometheus.site.net):

  prometheus:
    image: "prom/prometheus"
    # ...
    command:
      - "--config.file=/prometheus.yml"
      # ...
3

This works for exposing the log files from prometheus... is there a way to get access to the metrics or graphs page in prometheus too? i assume you need more paths, but im unsure. maybe a way to expose all url paths?

I'm also just a bit confused why we are using the "internal" prometheus when i have a service @ docker, unless it's routing to that?

The other issue is that this exposes the "graph" panel if i visit ip:9090, from my understanding all services run under docker and prometheus / grafana should 'point' to it. Am i missing something? I only want these pages protected under my domain and inaccessible from external ip