Requests on dashboard to `overview` work for 35 seconds, then get no response for 70 seconds, then work again for 35 seconds, etc

[ringsten]

Running v2.10.5 in a one node swarm.

When visiting the dashboard, I can see in the dev tools network tab that polling the overview endpoint happens every 5 seconds.

These requests will 200 OK for 35 seconds, then receive no response for 70 seconds, then work normally again for 35 seconds, then no response for 70 seconds (they're not dropped or rejected, just hang there), etc ad nauseam.

I have the same issue with another application (paperless-ngx), if I hit the "refresh" button in the tasks queue, it will eventually stop responding at all for 70 seconds and then work again.

If I run the same requests with curl, no problem I always get a response (even while the browser is not).

Wireshark just shows TCP retransmissions and resets:

The requests just hang:

This only happens with HTTPS enabled (it's fine with HTTP), so it could be a HTTP/2 problem too. No, it also happens with HTTP1.1 over HTTPS.

I've tried upping maxIdleConnsPerHost to 10 with the same result.

Any ideas?

# traefik.yml
api:
  dashboard: true
entryPoints:
  web:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: "websecure"
          scheme: "https"
  websecure:
    address: ":443"
serversTransport:
  insecureSkipVerify: true
providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
    network: "traefik-net"
    swarmmode: true
    swarmmodeRefreshSeconds: 15
    watch: true
  file:
    filename: "/etc/traefik/config.yml"
    watch: true
certificatesResolvers:
  staging:
    acme:
      email: "<redacted>"
      storage: "/etc/traefik/certificates/acme.json"
      caServer: "https://acme-staging-v02.api.letsencrypt.org/directory"
      dnsChallenge:
        provider: "hurricane"
  production:
    acme:
      email: "<redacted>"
      storage: "/etc/traefik/certificates/acme.json"
      caServer: "https://acme-v02.api.letsencrypt.org/directory"
      dnsChallenge:
        provider: "hurricane"

# config.yml
http:
  middlewares:
    traefikAuth:
      basicAuth:
        users:
          - "admin:$apr1$Fg8R5.rj$O28A0voS0SdkYCq2EYJ3H1"

# docker-compose.yml for traefik
version: "3.4"
services:
  traefik:
    image: traefik:2.10
    volumes:
      - ./traefik.yml:/etc/traefik/traefik.yml
      - ./config.yml:/etc/traefik/config.yml
      - certificates:/etc/traefik/certificates
      - /var/run/docker.sock:/var/run/docker.sock
      - /etc/localtime:/etc/localtime:ro
    environment:
      HURRICANE_TOKENS_FILE: "/run/secrets/traefik_hurricane_tokens"
    secrets:
      - traefik_hurricane_tokens
    ports:
      - "80:80"
      - "443:443"
    networks:
      - traefik-net
    deploy:
      labels:
        traefik.http.routers.traefik.tls.certResolver: "staging"
        traefik.enable: "true"
        traefik.http.routers.traefik.middlewares: "traefikAuth@file" #,default@file"
        traefik.http.routers.traefik.rule: "Host(`${TRAEFIK_FQDN}`)"
        traefik.http.routers.traefik.tls.domains[0].sans: "${TRAEFIK_SANS}"
        traefik.http.routers.traefik.service: "api@internal"
        traefik.http.services.traefik.loadbalancer.server.port: 8080
        traefik.http.services.traefik.loadbalancer.sticky.cookie.secure: "true"
      placement:
        constraints:
          - node.role == manager

networks:
  traefik-net:
    external: true

secrets:
  traefik_hurricane_tokens:
    external: true

volumes:
  certificates:

[bluepuma77]

Looks good in general. I would add a replicas or mode:global to deploy section. Usually TLS is defined with main and sans. In yaml we usually don’t put values in quotes. Not sure if that all matters.

Compare with simple Traefik Swarm example.

[ringsten]

This was a layer 4 issue after all. My Opnsense router was set to the default optimization mode. Instead I had to set it to "conservative", which "Tries to avoid dropping any legitimate idle connections at the expense of increased memory usage and CPU utilization."

I don't know why that isn't the default, why would you want your firewall to drop legitimate idle connections??

Anyway, for anyone else having this issue, it's in Firewall > Settings > Advanced > Miscellaneous > Firewall Optimization, set it to "conservative" instead of "normal".