Traefik as Reverse Proxy for Grafana and Prometheus

Hi,
so I have the following Problem. I am a Newbie with Traefik and I want to use it as reverse proxy for Prometheus and Grafana.

Forgive me for writing the configuration as YAML (you can convert it to TOML via various online tools):

---

providers:
  file:
    filename: /etc/traefik/traefik.yml

log:
  level: debug

api:
  dashboard: True

entryPoints:
  web:
    address: ":80"

http:
  routers:
    prometheus-router:
      rule: "PathPrefix(`/prometheus`)"
      service: prometheus
      entryPoints:
        - web
    grafana-router:
      rule: "PathPrefix(`/grafana`)"
      service: grafana
      entryPoints:
        - web
    dashboard:
      rule: "PathPrefix(`/api`) || PathPrefix(`/dashboard`)"
      service: dashboard@internal
      entryPoints:
        - web
  services:
    prometheus:
      loadBalancer:
        servers:
          - url: "http://127.0.0.1:9090/"
    grafana:
      loadBalancer:
        servers:
          - url: "http://127.0.0.1:3000/"

Basically I have the following goals:

  • traefik should run on port 80 with plain HTTP
  • If I visit http://mywebsite.com/prometheus I should be forwarded to the prometheus service that listens on 127.0.0.1:9090
  • If I vist http://mywebsite.com/grafana I should be forwarded to the grafana service that listens on 127.0.0.1:3000
  • If I visit http:/ /mywebsite.com/dashboard I want to see the traefik dashboard
  • If I visit http:/ /mywebsite.com/api I want to have access to the traefik API

The configuration above took me several hours, the documentation for Traefik v2 is okayish, but it's difficult to find realworld working examples without docker + there are some bugs in the documentation. For example the Key entrypoint in api doesn't exist.

Ok so what does my configuration do? I understand the configuration as the following:

  1. I set the file provider and set filename to the location of my configuration. The configuration as you see it is saved as /etc/traefik/traefik.yml. I have no idea if this is the correct way. I found various descriptions where everybody is just setting file: {} but this didn't work for me with traefik 2.0

  2. I set log level to debug (this works)

  3. I set api and dashboard to true (I have no idea if this works, because I can't see the dashboard, even on the host directly)

  4. I set an entrypoin called web listening on port 80 (this works as well)

  5. I set http configurations(?) (First I thought I need to use the entrypoint name here, I saw various examples on the web doing so.. but looks like I need to either choose http or tcp depending on the associated routers.

  6. I set the three routers for grafana, prometheus and the api/dashboard. I have no idea if dashboard@internal is correct, I just see errors when I spawn traefik and try to use only dashboard.. I will copy paste them in the end.

  7. I set up the services (looks like only loadbalancers are possible, even with only one server.. weird), there I set the URLs where the other services are listening to.

So what am I doing wrong? When I start traefik I see the following messages:

root@hive:~# traefik
INFO[0000] Configuration loaded from file: /etc/traefik/traefik.yml 
INFO[2019-10-29T15:23:45+01:00] Traefik version 2.0.4 built on 2019-10-28T20:23:57Z 
DEBU[2019-10-29T15:23:45+01:00] Static configuration loaded {"global":{"checkNewVersion":true},"serversTransport":{"maxIdleConnsPerHost":200},"entryPoints":{"web":{"address":":80","transport":{"lifeCycle":{"graceTimeOut":10000000000},"respondingTimeouts":{"idleTimeout":180000000000}},"forwardedHeaders":{}}},"providers":{"providersThrottleDuration":2000000000,"file":{"watch":true,"filename":"/etc/traefik/traefik.yml"}},"api":{"dashboard":true},"log":{"level":"debug","format":"common"}} 
INFO[2019-10-29T15:23:45+01:00] 
Stats collection is disabled.
Help us improve Traefik by turning this feature on :)
More details on: https://docs.traefik.io/v2.0/contributing/data-collection/ 
DEBU[2019-10-29T15:23:45+01:00] No default certificate, generating one       
INFO[2019-10-29T15:23:45+01:00] Starting provider aggregator.ProviderAggregator {} 
DEBU[2019-10-29T15:23:45+01:00] Start TCP Server                              entryPointName=web
INFO[2019-10-29T15:23:45+01:00] Starting provider *file.Provider {"watch":true,"filename":"/etc/traefik/traefik.yml"} 
DEBU[2019-10-29T15:23:45+01:00] Configuration received from provider file: {"http":{"routers":{"dashboard":{"entryPoints":["web"],"service":"dashboard@internal","rule":"PathPrefix(`/api`) || PathPrefix(`/dashboard`)"},"grafana-router":{"entryPoints":["web"],"service":"grafana","rule":"PathPrefix(`/grafana`)"},"prometheus-router":{"entryPoints":["web"],"service":"prometheus","rule":"PathPrefix(`/prometheus`)"}},"services":{"grafana":{"loadBalancer":{"servers":[{"url":"http://127.0.0.1:3000/"}],"passHostHeader":null}},"prometheus":{"loadBalancer":{"servers":[{"url":"http://127.0.0.1:9090/"}],"passHostHeader":null}}}},"tcp":{},"tls":{}}  providerName=file
ERRO[2019-10-29T15:23:45+01:00] the service "dashboard@internal" does not exist  routerName=dashboard@file entryPointName=web
DEBU[2019-10-29T15:23:45+01:00] Creating middleware                           middlewareType=Pipelining routerName=prometheus-router@file serviceName=prometheus entryPointName=web middlewareName=pipelining
DEBU[2019-10-29T15:23:45+01:00] Creating load-balancer                        entryPointName=web routerName=prometheus-router@file serviceName=prometheus
DEBU[2019-10-29T15:23:45+01:00] Creating server 0 http://127.0.0.1:9090/      entryPointName=web routerName=prometheus-router@file serviceName=prometheus serverName=0
DEBU[2019-10-29T15:23:45+01:00] Added outgoing tracing middleware prometheus  entryPointName=web routerName=prometheus-router@file middlewareName=tracing middlewareType=TracingForwarder
DEBU[2019-10-29T15:23:45+01:00] Creating middleware                           entryPointName=web routerName=grafana-router@file serviceName=grafana middlewareName=pipelining middlewareType=Pipelining
DEBU[2019-10-29T15:23:45+01:00] Creating load-balancer                        serviceName=grafana entryPointName=web routerName=grafana-router@file
DEBU[2019-10-29T15:23:45+01:00] Creating server 0 http://127.0.0.1:3000/      entryPointName=web routerName=grafana-router@file serviceName=grafana serverName=0
DEBU[2019-10-29T15:23:45+01:00] Added outgoing tracing middleware grafana     entryPointName=web routerName=grafana-router@file middlewareName=tracing middlewareType=TracingForwarder
DEBU[2019-10-29T15:23:45+01:00] Creating middleware                           entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
DEBU[2019-10-29T15:23:45+01:00] No default certificate, generating one       

When I do curl localhost. I get 404 page not found.
When I do curl localhost/grafana I get <a href="/login">Found</a>., but when I use curl -L localhost/grafana I get forwarded to localhost/login and then I get 404 page not found again.
what do I need to do to follow redirects? With caddy it's just 3 lines to setup such a reverse proxy.. (at least with caddy before caddy 2. I had trouble with caddy 2 as well.)

when I do curl localhost/prometheus I even get 404 page not found although I get a valid response when I do curl localhost:9090 -> <a href="/graph">Found</a>.

Some help would be really appreciated.

Ok, I have made some progress with the prometheus-router with using the stripPrefix middleware (at least I get forward to graph now, but it's still wrong and I will not get forwarded to the right path):

rometheus-router:
 rule: "PathPrefix(`/prometheus`)"
 service: prometheus   
 entryPoints:          
   - web
 middlewares:
   - prometheus-middleware
middlewares:            
  prometheus-middleware:
    stripPrefix:        
      prefixes:         
        - "/prometheus"

I guess I need to strip the prefix and then add it again to match the right service??? no idea..

Ok,
I have solved this issue by myself via specifying the subpath in prometheus and grafana directly. Now it's working as expected with this configuration:

api:
  dashboard: True

entryPoints:
  web:
    address: ":80"

http:
  routers:
    prometheus-router:
      rule: "PathPrefix(`/prometheus`)"
      service: prometheus   
      entryPoints:          
        - web
    grafana-router:
      rule: "PathPrefix(`/grafana`)"
      service: grafana               
      entryPoints:                   
        - web             
    dashboard:
      rule: "PathPrefix(`/api`) || PathPrefix(`/dashboard`)"
      service: dashboard@internal
      entryPoints:
        - web
  services:  
    prometheus:
      loadBalancer:            
        servers:               
          - url: "http://127.0.0.1:9090/"
    grafana:
      loadBalancer:
        servers:   
          - url: "http://127.0.0.1:3000/"

The only thing that still doesn't work is the access to the traefik dashboard or API... if anybody has an idea, I would be happy.

nevermind, I found the issue by myself.. I need to address api@internal and not dashboard@internal. Now it works as expected with localhost/dashboard/ (mind the trailing slash)

---

providers:
  file:
    filename: /etc/traefik/traefik.yml

log:
  level: debug

api:
  dashboard: True

entryPoints:
  web:
    address: ":80"

http:
  routers:
    prometheus-router:
      rule: "PathPrefix(`/prometheus`)"
      service: prometheus
      entryPoints:
        - web
    grafana-router:
      rule: "PathPrefix(`/grafana`)"
      service: grafana
      entryPoints:
        - web
    api:
      rule: "PathPrefix(`/api`) || PathPrefix(`/dashboard`)"
      service: api@internal
      entryPoints:
        - web
  services:
    prometheus:
      loadBalancer:
        servers:
          - url: "http://127.0.0.1:9090/"
    grafana:
      loadBalancer:
        servers:
          - url: "http://127.0.0.1:3000/"
2 Likes