Chunking not functioning as expected

Traefik Traefik v2
, ,
1

I'm having a hard time getting chunk streams to work. As soon as I moved my service behind traefik the streaming stopped working - now all it does is send the whole response at once.

I've looked into this pretty throughly and tried multiple things:

  • trying it disable buffering by setting: traefik.http.middlewares.limit.buffering.memResponseBodyBytes=0
  • removed the redirect for http to see if it was a TLS issue
  • did a lot more debugging that basically has me fairly confident that traefik is the root cause

For reference my docker-compose:

services:
  api:
    image: ghcr.io/goperigon/pokey:latest
    stop_grace_period: 2m
    healthcheck:
      test: curl http://localhost:8080/ping || exit
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 1m
    command: sh -c "python scripts/init_db.py && fastapi run app/main.py --host 0.0.0.0 --port 8080"
    deploy:
      mode: replicated
      replicas: 2 ### TODO: determine ideal # of replicas under normal load
      restart_policy:
        condition: any
        delay: 5s
        max_attempts: 3
        window: 120s
      update_config:
        failure_action: pause
        monitor: 20s
        order: start-first
      placement:
        constraints:
          - node.labels.pokey.api.api_node==1
      labels:
        - traefik.enable=true
        - traefik.docker.network=traefik-public
        - traefik.constraint-label=traefik-public

        # - traefik.http.middlewares.pokey-api-https-redirect.redirectscheme.scheme=https
        # - traefik.http.middlewares.pokey-api-https-redirect.redirectscheme.permanent=true

        - traefik.http.middlewares.pokey-api-strip-prefix.stripprefix.prefixes=/pokey

        - traefik.http.routers.pokey-api-public-http.rule=Host(`sys.perigon.io`) && PathPrefix(`/pokey`)
        - traefik.http.routers.pokey-api-public-http.entrypoints=http
        - traefik.http.routers.pokey-api-public-http.middlewares=pokey-api-https-redirect
        - traefik.http.routers.pokey-api-public-http.middlewares=pokey-api-strip-prefix

        - traefik.http.routers.pokey-api-public-https.rule=Host(`sys.perigon.io`) && PathPrefix(`/pokey`)
        - traefik.http.routers.pokey-api-public-https.entrypoints=https
        - traefik.http.routers.pokey-api-public-https.tls=true
        - traefik.http.routers.pokey-api-public-https.tls.certresolver=le
        - traefik.http.routers.pokey-api-public-https.middlewares=pokey-api-strip-prefix
        
        - traefik.http.services.pokey-api-public.loadbalancer.server.port=8080
        # - traefik.http.services.pokey-api-public.loadbalancer.server.scheme=https
        # - traefik.http.services.pokey-api-public.loadbalancer.serverstransport=business-api-secure@file
        - traefik.http.middlewares.limit.buffering.memResponseBodyBytes=0

        - traefik.http.services.pokey-api-public.loadbalancer.healthcheck.path=/ping
        - traefik.http.services.pokey-api-public.loadbalancer.healthcheck.interval=10s
        - traefik.http.services.pokey-api-public.loadbalancer.healthcheck.timeout=5s
    environment:
      - ENV=prod
     
    networks:
      - traefik-public

networks:
  traefik-public:
    external: true
2

What’s the exact issue? Your target service is sending the response chunked, but Traefik collects all chunks and responds with a single message?

3
  1. Is the "whole response at once" , is a full response in 100% of times? or sometimes it might be a faulty response with missing parts?
  2. which version of Traefik do you use?
4

Traefik Version: v2.10.5

Sorry what I meant was normally I have essentially SSE - streaming events over the course of about a minute, new events every second or so. When I put traefik in front of it I get all the events sent at once at the very end, instead of the gradual "chunk-streaming" of the responses.

Here is what I mean in a simple example:

--- normally ---
Events streamed:
0:01: {"msg": "random event"}
0:05: {"msg": "another event"}
0:19: {"msg": "another event"}
1:00: {"msg": "last event"}

--- now (with traefik) ---
0:01:
0:05:
0:19:
1:00: {"msg": "random event"}\n{"msg": "another event"}\n{"msg": "another event"}\n{"msg": "last event"}

At first glance this seems to be some type of buffering that may be happening under the hood? But I'm not quite sure.

5

Yes that is exactly the issue.

6

I have seen issue that looked familiar on Github Buffering middleware breaks `chunked` encoded requests · Issue #7930 · traefik/traefik · GitHub

I'm not sure if you use the buffering middleware, but even if not , maybe one of suggestions there might help you.

Also I would try updating to latest Traefik 2 version , since I have seen some PRs that talked about some chunks related functionality that came after version 2.11

7

Finally got it fixed, turns out had an application level middleware saying the content would be gzipped causing traefik to have this behavior