Request header too large

Traefik Traefik v2 (latest)
,
1

Hi guys,

I've got an issue I'm not quite sure I'm trying to resolve correctly.

I've just loaded up Grafana in Docker and routed it via Traefik, but I'm getting the following error in one of my (quite large) panels:

image
image2479×764 60.7 KB

I've looked at https://docs.traefik.io/v2.0/middlewares/buffering/ - but I'm not quite sure if 1) that's the correct thing to look at and 2) if I'm doing it right if it is!

This is my Grafana section in docker-compose

  # Grafana - Graphical data visualization for InfluxDB data
  grafana:
    hostname: grafana
    container_name: Grafana
    restart: unless-stopped
    image: grafana/grafana
    # user: '472:472'
    networks:
      internal:
      pihole:
        ipv4_address: '172.22.0.112'
    ports:
      - 3000:3000
    volumes: 
      # - $USERDIR/TIGVStack/Grafana:/var/lib/grafana # sudo mkdir -p $USERDIR/TIGVStack/Grafana; chown 472:472 $USERDIR/TIGVStack/Grafana
      - $USERDIR/TIGVStack/Grafana:/config # sudo mkdir -p $USERDIR/TIGVStack/Grafana; chown 472:472 $USERDIR/TIGVStack/Grafana
      - $USERDIR/TIGVStack/Grafana/data/telegraf.conf:/etc/telegraf/telegraf.conf:ro
    environment:
      - GF_PATHS_DATA=/config/data
      - GF_PATHS_LOGS=/config/logs
      - GF_PATHS_PLUGINS=/config/plugins
      - GF_INSTALL_PLUGINS=grafana-clock-panel,grafana-simple-json-datasource,grafana-worldmap-panel,grafana-piechart-panel
      - PUID=$PUID
      - PGID=$PGID
      - TZ=$TZ
    labels:
      - "traefik.enable=true"
      ## HTTP Routers
      - "traefik.http.routers.grafana-rtr.entrypoints=https"
      - "traefik.http.routers.grafana-rtr.rule=HostHeader(`grafana.$DOMAINNAME`)"
      ## Middlewares
      - "traefik.http.routers.grafana-rtr.middlewares=chain-authelia@file"
      - "traefik.http.middlewares.limit.buffering.maxRequestBodyBytes=200000"
      - "traefik.http.middlewares.limit.buffering.maxResponseBodyBytes=200000"
      - "traefik.http.middlewares.limit.buffering.memRequestBodyBytes=200000"
      - "traefik.http.middlewares.limit.buffering.memResponseBodyBytes=200000"
      ## HTTP Services
      - "traefik.http.routers.grafana-rtr.service=grafana-svc"
      - "traefik.http.services.grafana-svc.loadbalancer.server.port=3000"
    depends_on:
      - influxdb
      - varken

(I'm aware that I put both disk and mem values in there, but just mem wasn't working, so was trying basically everything).

Any assistance would be fantastic!

Thanks in advance!

Edit:

Prior to putting Grafana into Docker, I had a rule in Traefik that pointed to my local Grafana install which was:

[http.routers]
  [http.routers.grafana-rtr]
      entryPoints = ["https"]
      rule = "Host(`grafana.mydomain.com`)"
      service = "grafana-svc"
      middlewares = ["chain-no-auth"]
      [http.routers.grafana-rtr.tls]
        certresolver = "dns-cloudflare"

[http.services]
  [http.services.grafana-svc]
    [http.services.grafana-svc.loadBalancer]
      passHostHeader = true
      [[http.services.grafana-svc.loadBalancer.servers]]
        url = "http://192.168.0.254:3000"
    [http.services.grafana-svc.loadBalancer.healthCheck]
      path = "/api/health"
      port = 3000
      interval = "10s"
      timeout = "3s"

And this did not exhibit the same symptoms.

This particular panel works if I load up Grafana locally which seems to indicate that the issue is with the r-proxy

2

I'm having the exact same issue. When I check Grafana on my LAN, all the data shows up just fine, but when I go to my external URL, a lot of info is missing. When I check the query in Grafana, it tells me the request entity is too large and I see the headers are huge.

3

Hi Mike,

The only workaround I've found so far, is to change the data source http method from GET to POST, then it seems to work.

But this certainly points to Traefik missing the ability to allow this/the ability not working as intended.

4

How are you changing the http method, from within Grafana?

5

Go to data sources, edit your data source, and at the bottom you'll find:

image

Change from GET to POST.

Hopefully that should resolve your issue as a workaround.

6

Excellent. Worth a shot. Thank you.