"no space left on device" after upgrade to 3.x

Traefik Traefik v3 (latest)
1

I upgraded Traefik from (I do not remember, it was a version from about 6 months) to 3.3.5 and I get an error at startup:

traefik-1   | 2025-04-02T15:22:13+02:00 ERR Cannot start the provider *file.Provider error="error adding file watcher: no space left on device"

I found one reference to the problem on SO that also points to a Traefik Github Issue.

The proposed solution (or workaround) is to set, on the host (Debian in my case)

sysctl -w fs.inotify.max_user_watches=12288

but this does not work for me.

EDIT: I went through the release notes and migration guide and did not find anything relevant, a break is quite surprising.

How can I make the file configurations work again? (short of downgrading to a vague previous version)

Note: there is of course ample space on the disk.

2

Share your full Traefik static and dynamic config, and Docker compose file(s) if used.

Share df output.

3

Sure, if that can help

compose.yaml

  hass:
    depends_on:
      - mqtt
    devices:
      - /dev/serial/by-id/usb-1a86_USB_Serial-if00-port0:/dev/serial/by-id/usb-1a86_USB_Serial-if00-port0
    image: homeassistant/home-assistant:2025.3
    labels:
      - traefik.enable=true
      - traefik.http.routers.hass.rule=Host(`hass.XXX`)
      - traefik.http.routers.hass.service=hass
      - traefik.http.services.hass.loadbalancer.server.port=8123
      # - traefik.http.services.hass.loadbalancer.server.url=http://192.168.10.2
    network_mode: host
    restart: unless-stopped
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - hass:/config

traefik.yaml

log:
  level: INFO

global:
  sendAnonymousUsage: true

api:
  dashboard: true
  insecure: true

providers:
  docker:
    exposedByDefault: false
  file:
    directory: /config/config.d
    
certificatesresolvers:
  letsencrypt:
    acme:
      dnschallenge:
        provider: ovh
      email: XXX
      storage: /config/acme.json

entryPoints:
  web:
    address: :80
    http:
      redirections:
        entrypoint:
          to: websecure
          scheme: https
  websecure:
    address: :443
    http3: {}
    asDefault: true
    http:
      tls:
        certResolver: letsencrypt
        domains:
          - main: XXX
            sans:
              - "*.XXX"
              - "*.XXX"
              - "*.XXX"
      middlewares:
        - authelia
    forwardedHeaders:
      trustedIPs: 
        - 10.0.0.0/8
        - 172.16.0.0/12
        - 192.168.0.0/16
        - fc00::/7

An example of the file based config (that is not loaded)

http:
  routers:
    hass:
      rule: Host(`hass.XXX`)
      service: hass
  services:
    hass:
      loadBalancer:
        servers:
          - url: http://192.168.10.2:8123

The logs with the error:

traefik-1   | 2025-04-02T15:22:13+02:00 INF Traefik version 3.3.5 built on 2025-03-31T08:45:53Z version=3.3.5
traefik-1   | 2025-04-02T15:22:13+02:00 INF Stats collection is enabled.
traefik-1   | 2025-04-02T15:22:13+02:00 INF Many thanks for contributing to Traefik's improvement by allowing us to receive anonymous information from your configuration.
traefik-1   | 2025-04-02T15:22:13+02:00 INF Help us improve Traefik by leaving this feature on :)
traefik-1   | 2025-04-02T15:22:13+02:00 INF More details on: https://doc.traefik.io/traefik/contributing/data-collection/
traefik-1   | 2025-04-02T15:22:13+02:00 INF Starting provider aggregator *aggregator.ProviderAggregator
traefik-1   | 2025-04-02T15:22:13+02:00 INF Starting provider *file.Provider
traefik-1   | 2025-04-02T15:22:13+02:00 ERR Cannot start the provider *file.Provider error="error adding file watcher: no space left on device"
traefik-1   | 2025-04-02T15:22:13+02:00 INF Starting provider *traefik.Provider
traefik-1   | 2025-04-02T15:22:13+02:00 INF Starting provider *acme.ChallengeTLSALPN
traefik-1   | 2025-04-02T15:22:13+02:00 INF Starting provider *docker.Provider
traefik-1   | 2025-04-02T15:22:13+02:00 INF Starting provider *acme.Provider
traefik-1   | 2025-04-02T15:22:13+02:00 INF Testing certificate renew... acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme

The disk space:

root@srv # df -h | grep -v overlay
Filesystem             Size  Used Avail Use% Mounted on
udev                    12G     0   12G   0% /dev
tmpfs                  2.4G  4.4M  2.4G   1% /run
/dev/nvme0n1p2         915G  520G  348G  60% /
tmpfs                   12G     0   12G   0% /dev/shm
tmpfs                  5.0M     0  5.0M   0% /run/lock
/dev/nvme0n1p1         511M  5.9M  506M   2% /boot/efi
/dev/sde1              1.8T  454G  1.3T  27% /backup
/dev/mapper/data-data  8.1T  6.9T  827G  90% /data
tmpfs                  2.4G     0  2.4G   0% /run/user/0
4

error adding file watcher -> related to inotify and not to actual disk space.
try raising it from your docker-compose file

services:
  traefik:
    image: traefik:latest
    sysctls:
      - fs.inotify.max_user_watches=524288

(you might need to update your system inotify as well)

also you can try to disable the watch for config changes, if you don't need it to watch for changes all the time.