Provider connection error context deadline exceeded

Traefik Traefik v2
1

After traefik has been running for a couple of hours, it stops working. When I try to access one of the websites it gives me this error:
503 Service Unavailable
And in the logs of traefik appears this many times consecutively:
Provider connection error context deadline exceeded (Client.Timeout or context cancellation while reading body), retrying in 451.831812ms

My configuration:

Docker compose:

version: '3'

services:
  traefik:
    image: traefik:latest
    restart: unless-stopped
    container_name: traefik
    extra_hosts:
      - "host.docker.internal:host-gateway"
    networks:
      - traefik
    ports:
      - "80:80/tcp"
      - "443:443/tcp"
      - "443:443/udp"  # HTTP3
      - "127.0.0.1:8080:8080/tcp"
    volumes:
      - /opt/traefik:/etc/traefik
      - /var/log/traefik/:/logs
      - acme:/etc/traefik/acme
      - /var/run/docker.sock:/var/run/docker.sock
    security_opt:
      - "no-new-privileges:true"

volumes:
  acme:
    driver: local

networks:
  traefik:
    external: true

Note:

  • the traefik network is a bridge network, ipv4 only
  • host.docker.internal is used to connect to service on the same machine that cannot be put inside a container

Configuration:

################################################################
# Global configuration
################################################################
global:
  checkNewVersion: false
  sendAnonymousUsage: false

experimental:
  http3: true
  plugins:
    geoblock:
      moduleName: "github.com/PascalMinder/geoblock"
      version: "v0.2.7"
    sablier:
      moduleName: "github.com/acouvreur/sablier"
      version: "v1.4.1-beta.5"
    souin:
      moduleName: "github.com/darkweak/souin"
      version: "v1.6.43"

################################################################
# EntryPoints configuration
################################################################

entryPoints:
  web:
    address: :80
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https

  websecure:
    address: :443
    http3: {}
    http2:
      maxConcurrentStreams: 250
    forwardedHeaders:
      trustedIPs:
        - "127.0.0.1/8"
    http:
      middlewares:
        - ratelimit@file
        - latency-check@file

################################################################
# Traefik logs configuration
################################################################

log:
  format: common
  level: WARNING

################################################################
# Access logs configuration
################################################################

accessLog:
  filePath: "/logs/traefik.log"
  format: json
  fields:
    headers:
      defaultMode: drop
      names:
        User-Agent: keep # log user agent strings

################################################################
# Docker configuration backend
################################################################

providers:
  docker:
    exposedByDefault: false
    network: traefik
    #useBindPortIP: true
    httpClientTimeout: 300
    defaultRule: "Host(`{{ .ContainerName }}.example.com`)"
  file:
    directory: /etc/traefik/file
    watch: true

################################################################
# API and dashboard configuration
################################################################

api:
  insecure: true
  dashboard: true

################################################################
# Certificates
################################################################

tls:
  options:
    default:
      minVersion: VersionTLS12
      sniStrict: true
    mintls13:
      minVersion: VersionTLS13
  stores:
    default:
      defaultGeneratedCert:
        resolver: main

certificatesResolvers:
  main:
    acme:
      #caServer: "https://acme-staging-v02.api.letsencrypt.org/directory"
      email: example@example.com
      storage: /etc/traefik/acme/acme.json
      keyType: 'EC384'
      preferredChain: 'ISRG Root X2'
      tlsChallenge: {}

Logs:

time="2023-11-23T09:49:20Z" level=info msg="Configuration loaded from file: /etc/traefik/traefik.yml"
2023/11/23 09:52:06 reverseproxy.go:666: suppressing panic for copyResponse error in test; copy error: H3_REQUEST_CANCELLED
time="2023-11-23T09:54:22Z" level=error msg="Provider connection error context deadline exceeded (Client.Timeout or context cancellation while reading body), retrying in 584.836694ms" providerName=docker
2023/11/23 09:55:42 reverseproxy.go:666: suppressing panic for copyResponse error in test; copy error: context canceled
time="2023-11-23T09:57:13Z" level=error msg="accept tcp [::]:443: use of closed network connection" entryPointName=websecure
time="2023-11-23T09:57:13Z" level=error msg="accept tcp [::]:80: use of closed network connection" entryPointName=web
time="2023-11-23T09:57:13Z" level=error msg="accept tcp [::]:8080: use of closed network connection" entryPointName=traefik
time="2023-11-23T09:57:13Z" level=error msg="Error while starting server: accept tcp [::]:80: use of closed network connection" entryPointName=web
time="2023-11-23T09:57:13Z" level=error msg="Error while starting server: accept tcp [::]:8080: use of closed network connection" entryPointName=traefik
time="2023-11-23T09:57:13Z" level=error msg="close tcp [::]:443: use of closed network connection" entryPointName=websecure
time="2023-11-23T09:59:15Z" level=info msg="Configuration loaded from file: /etc/traefik/traefik.yml"
time="2023-11-23T10:04:55Z" level=error msg="Provider connection error context deadline exceeded (Client.Timeout or context cancellation while reading body), retrying in 672.352453ms" providerName=docker
time="2023-11-23T10:09:56Z" level=error msg="Provider connection error context deadline exceeded (Client.Timeout or context cancellation while reading body), retrying in 565.826376ms" providerName=docker
time="2023-11-23T10:14:56Z" level=error msg="Provider connection error context deadline exceeded (Client.Timeout or context cancellation while reading body), retrying in 704.029977ms" providerName=docker

time="2023-11-24T13:47:57Z" level=error msg="Provider connection error context deadline exceeded (Client.Timeout or context cancellation while reading body), retrying in 491.962196ms" providerName=docker
time="2023-11-24T13:52:58Z" level=error msg="Provider connection error context deadline exceeded (Client.Timeout or context cancellation while reading body), retrying in 657.455584ms" providerName=docker
time="2023-11-24T13:57:58Z" level=error msg="Provider connection error context deadline exceeded (Client.Timeout or context cancellation while reading body), retrying in 361.730211ms" providerName=docker
time="2023-11-24T14:02:59Z" level=error msg="Provider connection error context deadline exceeded (Client.Timeout or context cancellation while reading body), retrying in 501.792534ms" providerName=docker
time="2023-11-24T14:07:59Z" level=error msg="Provider connection error context deadline exceeded (Client.Timeout or context cancellation while reading body), retrying in 271.412027ms" providerName=docker
time="2023-11-24T14:13:00Z" level=error msg="Provider connection error context deadline exceeded (Client.Timeout or context cancellation while reading body), retrying in 451.831812ms" providerName=docker