"no space left on device" after upgrade to 3.x

Traefik Traefik v3 (latest)
1

I upgraded Traefik from (I do not remember, it was a version from about 6 months) to 3.3.5 and I get an error at startup:

traefik-1   | 2025-04-02T15:22:13+02:00 ERR Cannot start the provider *file.Provider error="error adding file watcher: no space left on device"

I found one reference to the problem on SO that also points to a Traefik Github Issue.

The proposed solution (or workaround) is to set, on the host (Debian in my case)

sysctl -w fs.inotify.max_user_watches=12288

but this does not work for me.

EDIT: I went through the release notes and migration guide and did not find anything relevant, a break is quite surprising.

How can I make the file configurations work again? (short of downgrading to a vague previous version)

Note: there is of course ample space on the disk.

2

Share your full Traefik static and dynamic config, and Docker compose file(s) if used.

Share df output.

3

Sure, if that can help

compose.yaml

  hass:
    depends_on:
      - mqtt
    devices:
      - /dev/serial/by-id/usb-1a86_USB_Serial-if00-port0:/dev/serial/by-id/usb-1a86_USB_Serial-if00-port0
    image: homeassistant/home-assistant:2025.3
    labels:
      - traefik.enable=true
      - traefik.http.routers.hass.rule=Host(`hass.XXX`)
      - traefik.http.routers.hass.service=hass
      - traefik.http.services.hass.loadbalancer.server.port=8123
      # - traefik.http.services.hass.loadbalancer.server.url=http://192.168.10.2
    network_mode: host
    restart: unless-stopped
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - hass:/config

traefik.yaml

log:
  level: INFO

global:
  sendAnonymousUsage: true

api:
  dashboard: true
  insecure: true

providers:
  docker:
    exposedByDefault: false
  file:
    directory: /config/config.d
    
certificatesresolvers:
  letsencrypt:
    acme:
      dnschallenge:
        provider: ovh
      email: XXX
      storage: /config/acme.json

entryPoints:
  web:
    address: :80
    http:
      redirections:
        entrypoint:
          to: websecure
          scheme: https
  websecure:
    address: :443
    http3: {}
    asDefault: true
    http:
      tls:
        certResolver: letsencrypt
        domains:
          - main: XXX
            sans:
              - "*.XXX"
              - "*.XXX"
              - "*.XXX"
      middlewares:
        - authelia
    forwardedHeaders:
      trustedIPs: 
        - 10.0.0.0/8
        - 172.16.0.0/12
        - 192.168.0.0/16
        - fc00::/7

An example of the file based config (that is not loaded)

http:
  routers:
    hass:
      rule: Host(`hass.XXX`)
      service: hass
  services:
    hass:
      loadBalancer:
        servers:
          - url: http://192.168.10.2:8123

The logs with the error:

traefik-1   | 2025-04-02T15:22:13+02:00 INF Traefik version 3.3.5 built on 2025-03-31T08:45:53Z version=3.3.5
traefik-1   | 2025-04-02T15:22:13+02:00 INF Stats collection is enabled.
traefik-1   | 2025-04-02T15:22:13+02:00 INF Many thanks for contributing to Traefik's improvement by allowing us to receive anonymous information from your configuration.
traefik-1   | 2025-04-02T15:22:13+02:00 INF Help us improve Traefik by leaving this feature on :)
traefik-1   | 2025-04-02T15:22:13+02:00 INF More details on: https://doc.traefik.io/traefik/contributing/data-collection/
traefik-1   | 2025-04-02T15:22:13+02:00 INF Starting provider aggregator *aggregator.ProviderAggregator
traefik-1   | 2025-04-02T15:22:13+02:00 INF Starting provider *file.Provider
traefik-1   | 2025-04-02T15:22:13+02:00 ERR Cannot start the provider *file.Provider error="error adding file watcher: no space left on device"
traefik-1   | 2025-04-02T15:22:13+02:00 INF Starting provider *traefik.Provider
traefik-1   | 2025-04-02T15:22:13+02:00 INF Starting provider *acme.ChallengeTLSALPN
traefik-1   | 2025-04-02T15:22:13+02:00 INF Starting provider *docker.Provider
traefik-1   | 2025-04-02T15:22:13+02:00 INF Starting provider *acme.Provider
traefik-1   | 2025-04-02T15:22:13+02:00 INF Testing certificate renew... acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme

The disk space:

root@srv # df -h | grep -v overlay
Filesystem             Size  Used Avail Use% Mounted on
udev                    12G     0   12G   0% /dev
tmpfs                  2.4G  4.4M  2.4G   1% /run
/dev/nvme0n1p2         915G  520G  348G  60% /
tmpfs                   12G     0   12G   0% /dev/shm
tmpfs                  5.0M     0  5.0M   0% /run/lock
/dev/nvme0n1p1         511M  5.9M  506M   2% /boot/efi
/dev/sde1              1.8T  454G  1.3T  27% /backup
/dev/mapper/data-data  8.1T  6.9T  827G  90% /data
tmpfs                  2.4G     0  2.4G   0% /run/user/0
4

error adding file watcher -> related to inotify and not to actual disk space.
try raising it from your docker-compose file

services:
  traefik:
    image: traefik:latest
    sysctls:
      - fs.inotify.max_user_watches=524288

(you might need to update your system inotify as well)

also you can try to disable the watch for config changes, if you don't need it to watch for changes all the time.

5

I'm also experiencing this on my synology NAS. I have a script to update traefik nightly. The symptom i'm getting is that it's just using the default traefik certificate instead of my full chain / key. Version 3.3.6

traefik  | 2025-05-04T14:49:49Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *file.Provider
traefik  | 2025-05-04T14:49:49Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *file.Provider provider configuration config={"filename":"/etc/traefik/dynamic.yml","watch":true}
traefik  | 2025-05-04T14:49:49Z DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:122 > add watcher on: /etc/traefik
traefik  | 2025-05-04T14:49:49Z DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:122 > add watcher on: /etc/traefik/dynamic.yml
traefik  | 2025-05-04T14:49:49Z ERR github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:206 > Cannot start the provider *file.Provider error="error adding file watcher: no space left on device"
traefik  | 2025-05-04T14:49:49Z DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:231 > Starting TCP Server entryPointName=web
traefik  | 2025-05-04T14:49:49Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *docker.Provider

I'm only using the dynamic config to assign my wildcard cert chain and pem. (not sure why I can't get this to function without using the file provider) Everything else is commented out, and using docker tags for configuration.

When doing a docker-compose up with the sysctls above I get

Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: sysctl "fs.inotify.max_user_watches" is not in a separate kernel namespace: unknown

There seems to be some regression / changes to how traefik is using the file watching. I removed the filewatch and now it works.

watch: false

I'm also running traefik on an arch linux vm on esxi with esentially the same config - no issues.