Traefik ingress route on k8s, modifies chunked response from service

Traefik Traefik v2 (latest)
,
1

Problem:
We have an api service on k8s, serving compressed data over HTTP/1.1 with header transfer-encoding = chunked. When exposing this service just with an external-ip without TLS, api clients are able to stream chunks just fine and put each chunk item back together, unpacking the compressed content.

However, when we put our api behind a k8s traefik ingress route (web and websecure), somehow traefik seems to alter the original chunked response from the api, serving incomplete chunks, and false chunk terminations i.e. 0/r/n body length, thus making the clients unable to to put the chunk back together in the correct manner, and thus unable to decomress valid content.

Any pointers as to what may cause this?

Config:
Client:
apiclient: python3, aiohttp lib

Serverside:
Traefik is deployed with Helm

apiVersion: v2
appVersion: 2.2.8
description: A Traefik based Kubernetes ingress controller
home: https://traefik.io/
icon: https://raw.githubusercontent.com/containous/traefik/v2.2/docs/content/assets/img/traefik.logo.png
keywords:
  - traefik
  - ingress
maintainers:
  - email: emile@vauge.com
    name: emilevauge
  - email: daniel.tomcej@gmail.com
    name: dtomcej
  - email: ludovic@containo.us
    name: ldez
name: traefik
sources:
  - https://github.com/containous/traefik
type: application
version: 9.1.1

traefik v. 2.2.8 on k8s, with no middlewares, no compression
fastapi/HTTP/1.1 deployment

Ingressroute config:

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: <api-name>
  namespace: default
spec:
  entryPoints:
    - websecure
    - web
  routes:
    - match: (Host(`<hostname>`) && PathPrefix(`/move/schema`))
      kind: Rule
      services:
        - name: <api-name>
          port: 80

  tls:
    certResolver: default

Output:
Api chunks sent from API:

10.240.0.89:32916 - "GET /move/schema/raw/table/building_energinet_energy HTTP/1.1" 200

[]
{'OK': 'Loaded'} 2020-12-15T15:26:09.540Z
36657 2020-12-15T15:26:11.719Z
165960 2020-12-15T15:26:13.186Z

Api client requesting on http, external ip without traefik:

<ResponseHeaders('Date': 'Tue, 15 Dec 2020 15:22:05 GMT', 'Server': 'uvicorn', 'Transfer-Encoding': 'chunked')>
3738 False
4992 False
3744 False
3744 False
6240 False
9984 False
4215 True
Buffer complete: 36657, True
1241 False
1248 False
4992 False
2496 False
2496 False
1248 False
1248 False
2496 False
4992 False
3744 False
6240 False
1248 False
3744 False
1248 False
3744 False
3744 False
1248 False
3744 False
6240 False
6240 False
4992 False
3744 False
2496 False
3744 False
6240 False
2496 False
2496 False
3744 False
3744 False
1248 False
2496 False
3744 False
1248 False
1248 False
6240 False
3744 False
4992 False
1248 False
8736 False
4992 False
4992 False
2496 False
1248 False
4992 False
6240 False
1248 False
3744 False
3727 True
Buffer complete: 165960, True

Api client requesting on http/https, behind traefik ingress route (traefik reverse proxy)

<ResponseHeaders('Date': 'Tue, 15 Dec 2020 15:26:07 GMT', 'Server': 'uvicorn', 'Transfer-Encoding': 'chunked')>
2366 False
1724 False
4744 False
5930 False
7116 False
8302 False
2586 False
0 True
3889 True
Buffer complete: 36657, True
4090 False
13046 False
14232 False
1400 False
0 True
4088 False
16384 False
12296 True
Buffer complete: 65536, True

fails on decompress due to incomplete buffer: 
RuntimeError: Frame incomplete. LZ4F_decompress returned: 4455

In the output, the integers represent a chunk item binary length, the bool is if a chunk boundary is reached. Buffer complete represents one chunk and sould correspond to what the api sent. Note that for the first case, the buffer complete length matches the chunks sent from the api.

2

We also assume that Traefik somehow alters the original chunked respose. Therefore, client is unable to merge the chunks into the expected format.

Did you perform any further investigation? What did you find out?

3

Hi @kentore82 and @aleksvujic ! Thanks for your interest in Traefik!

@kentore82 is that still an issue for you with the latest version of Traefik?

@aleksvujic which version of Traefik do you use? Could you provide some DEBUG log?

Thanks!

4

@svx We performed an extensive research of the issue. It seems that Traefik has problems with delivering chunked responses to the client. We opened an issue in your GitHub repository: Chunked messages - missing data.