After many requests Chrome connections are pending and traefik raises errors

Traefik Traefik v2 (latest)
1

Hello Traefik followers,

I need your swarm knowledge today and hope you can help me a bit.
I use Leaflet to display 2D maps in an app. Similar to Google Maps, an image is split into small tiles in several zoom levels. If you zoom in and pan back and forth, more tiles are dynamically reloaded. Since we switched to Traefik, this no longer works properly. Everything works initially, but after about 3000 requests, not all tiles are loaded. Chrome Developer Network shows that there are several open connections in the "pending" state. After that, nothing really works. If I restart Traefik, it works again immediately.

From the structure:

Client > Traefik > NodeJS (working as HTTP to S3 public gateway) > S3 (Minio)

In the traefik log at level DEBUG I see the following error messages:

traefik  | time="2023-06-05T15:05:30Z" level=debug msg="Request has been aborted [my.ip.add.res:51420 - /projects/2/maps/tiles/1047679055122BXZQpCoHAdg5i3s8yKx0YLqaS4MDh9nFktIGOuUEm.pdf/3/5/1.png]: net/http: abort Handler" middlewareName=traefik-internal-recovery middlewareType=Recovery
traefik  | time="2023-06-05T15:05:31Z" level=debug msg="Request has been aborted [my.ip.add.res:51420 - /projects/2/maps/tiles/1047679055122BXZQpCoHAdg5i3s8yKx0YLqaS4MDh9nFktIGOuUEm.pdf/2/1/0.png]: net/http: abort Handler" middlewareName=traefik-internal-recovery middlewareType=Recovery
traefik  | time="2023-06-05T15:05:31Z" level=debug msg="Request has been aborted [my.ip.add.res:51420 - /projects/2/maps/tiles/1047679055122BXZQpCoHAdg5i3s8yKx0YLqaS4MDh9nFktIGOuUEm.pdf/2/0/0.png]: net/http: abort Handler" middlewareName=traefik-internal-recovery middlewareType=Recovery
traefik  | time="2023-06-05T15:05:31Z" level=debug msg="Request has been aborted [my.ip.add.res:51420 - /projects/2/maps/tiles/1047679055122BXZQpCoHAdg5i3s8yKx0YLqaS4MDh9nFktIGOuUEm.pdf/2/2/0.png]: net/http: abort Handler" middlewareName=traefik-internal-recovery middlewareType=Recovery
traefik  | time="2023-06-05T15:05:31Z" level=debug msg="Request has been aborted [my.ip.add.res:51420 - /projects/2/maps/tiles/1047679055122BXZQpCoHAdg5i3s8yKx0YLqaS4MDh9nFktIGOuUEm.pdf/2/0/1.png]: net/http: abort Handler" middlewareName=traefik-internal-recovery middlewareType=Recovery

My attempts and findings:

  • Traefik restart solves the problem in short term

  • Access from other computers or Chrome Incognito, for example, continues to work independently initially

  • If I stop Traefik and start nginx, this problem does not occure

  • The problem occurs with both Traefik binary and Docker

  • It doesn't matter whether NodeJS is addressed with http or https (with 1.1 or h2)

  • When I open NodeJS directly - without Traefik - the problem does not occur

  • If I force Chrome to use HTTP 1.1 instead of H2, the problem does not occur

  • I tried Traefik 3.0 beta 2 without change

Do you have an explanation or clue about these error messages?

2

Just an update: I've removed H2/HTTP2 support in traefik code and tried my compiled version which works fine. Any ideas?

3

The source of the problem is that with HTTP/2, the browser can now send all HTTP requests concurrently over a single connection.
In most cases the backend servers cannot handle to many concurrent requests and this leads to pending requests.
You can customize the number of concurrent request according to the backend servers performance using the http2.maxconcurrentstreams static parameter:

entryPoints:
  websecure:
    address: :443
    http2:
      maxconcurrentstreams: 50
1 Like
4

I am having a similar problem as described and changing the maximum concurrent streams as suggested by @ldaroczi does not resolve it.

I have a somewhat similar structure for serving tiles:
Client > Traefik > C# .Net > Mapcache > Mapserver

I am using the Docker version of Traefik as a reverse proxy for all my running containers.
Here is a list of my findings besides the ones described in the original post:

  • It only occurs in browsers based on the Chromium. I have not been able to reproduce the error state in Firefox. It shows up in both Chrome and Edge.

  • It does not occur when using HTTP instead of HTTPS. Indicating it has something to do with TLS.

  • According to the Traefik and Container logs the request does get to the container and a response is send back. However the browser ignores it and the request stays in the "pending state".

Any help to resolve this problem would be appreciated.

5

Hi @frevi,

Even if I'm sorry for you, I'm happy that I'm not the only one who feels this problem. In the meantime I have given up the analysis, from my point of view the cause is clearly identified related to HTTP2. Other reasons could be excluded if we use an NGINX container instead of Traefik.

My approach, which I'm currently using very well:

  1. Download Traefik source code
  2. Disable HTTP2
  3. Create your own Traefik binary (really easy)
  4. Exchange the file in the Docker container image

This solves my problem stably since weeks.

BTW: The same problem occurs in Traefik v3.

I suspect your MapServer sounds similar to leaflet = many requests that are started and may be canceled on the client side.

Regards
Timo

1 Like