gRPC number of concurrent stream block at 250

What did I do

I created a simple gRPC client and server. The server is deployed inside our k8s cluster (with k3s) and traefik v2.4.8.
There is only one replica of the server. I got an ingress using h2c protocol and everything works great.

This is my proto, a simple stream from the server to the client:

service TelemetryService {
  rpc GetTelemetry(TelemetryMeta) returns (stream Telemetry) {}

When I create a large number of stream from the client by calling GetTelemetry(), the number get blocked to exactly 250.
If I use a NodePort instead of the ingress, everything works as expected.

Moreover, when I close the client, I dont receive the disconnect message in the server. Again, if I use the NodePort, it works.

Some application we have can use up to 5000 concurrent steam.

What did I see

  1. Limit of 250 of concurrent stream
  2. No disconnect message in the server when the client is closed

What I want

I want to be able to increase the limit of 250 to a high number.

Version of Traefik

traefik v2.4.8 with helm chart traefik-9.18.2

Environment & configuration

    port: 50051                                                                                                                                                                                                    
    protocol: TCP                                                                                                                                                                                                  
    expose: true                                                                                                                                                                                                   
    exposedPort: 50051

The rest is default from the helm chart.

We use kubernetes with k3s 1.21 deployed on ubuntu 20.04.