Traefik is not hot reloading the certificates

Hi,

I'm trying to find how to reload traefik when my certificates (that are stored in files updated via rsync every now and then) are renewed.
But it seems that when the certificate files are updated on the host, traefik doesn't load the new ones, and that is an issue because I have to manually restart traefik in order to update them.

Here is my compose traefik config:

version: '3.6'

services:

  traefik:
    image: traefik:v2.0
    command:
      - "--log.level=DEBUG"
      - --entrypoints.web.address=:80
      - --entrypoints.websecure.address=:443
      - --providers.docker=true
      - --providers.docker.exposedByDefault=false
      - --providers.docker.network=docker
      - --providers.file.filename=config.toml
      - --providers.file.watch=true
      - --api.insecure=true
      - --accesslog=true
    networks:
      - traefik
    restart: always
    ports:
      - '80:80'
      - '443:443'
      - '8080:8080'
    volumes:
      - /home/web/data/config.toml:/config.toml
      - /var/run/docker.sock:/var/run/docker.sock
      - /etc/ssl/eri/:/etc/ssl/eri/:ro
    labels:
      - "traefik.enable=true"
      - "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"
      - "traefik.http.routers.http-catchall.rule=hostregexp(`{host:.+}`)"
      - "traefik.http.routers.http-catchall.entrypoints=web"
      - "traefik.http.routers.http-catchall.middlewares=redirect-to-https"


networks:
  traefik:
    external: true

And the config.toml file

defaultEntryPoints = ["http", "https"]

# Connection to docker host system (docker.sock)
[docker]
domain = "eri.network"
watch = true
# This will hide all docker containers that don't have explicitly
# set label to "enable"
exposedbydefault = false

# Force HTTPS
[entryPoints]
  [entryPoints.http]
  address = ":80"
    [entryPoints.http.redirect]
    entryPoint = "websecure"
  [entryPoints.websecure]
  address = ":443"
    [entryPoints.https.tls]
      [[entryPoints.https.tls.certificates]]
      certFile = "/etc/ssl/eri/fullchain.pem"
      keyFile = "/etc/ssl/eri/privkey.pem"

[[tls.certificates]]
  certFile = "/etc/ssl/eri/fullchain.pem"
  keyFile  = "/etc/ssl/eri/privkey.pem"

Hoping that someone can help me..

Kind regards,

Dorian

1 Like

It is better if you use providers.file.directory=/etc/traefik, mount your configuration into that path, and Traefik will load all the configuration files within said path.

Due to fsnotify being unreliable, Traefik will not watch individual certificate files, however, if you touch config.toml, this will force Traefik to reload the provider configuration (which includes the certificates), and those will be reloaded.

It's also worth noting that you have a mix of v1 and v2 traefik in your configuration file, so you may want to take the time to remove some of the irrelevant pieces (pretty much everything except [[tls.certificates]]). Also, please update Traefik to the latest version for security updates and bug-fixes image: traefik:v2.3.2

Thanks for using Traefik and let us know if you have any other questions.

Hi,

Sorry for the (quite) late reply, the touch of the config.toml file doesn't seem to work, despite using providers.file.directory=/etc/traefik instead.

Any idea ?

Thanks in advance.

1 Like

Hey,

I also have the same issue, Traefik just does not hot-reload certificates:

traefik.yml:

entryPoints:
    web:
        address: ":80"
    websecure:
        address: ":443"
    traefik:
        address: ":9090"

api:
    insecure: true
    dashboard: true

providers:
    docker:
        exposedByDefault: false
        network: traefik
    file:
        directory: /etc/traefik/dynamic
        watch: true

accessLog: {}

dynamic/ssl.yml:

tls:
  certificates:
    - certFile: "/opt/certs/chain.crt"
      keyFile: "/opt/certs/key.key"

Traefik container mounts (/opt/traefik has /opt/traefik/dynamic with ssl.yml specifying my certificate locations):

"Mounts": [
    {
        "Type": "bind",
        "Source": "/opt/traefik",
        "Destination": "/etc/traefik",
        "Mode": "ro",
        "RW": false,
        "Propagation": "rprivate"
    },
    {
        "Type": "bind",
        "Source": "/opt/certs",
        "Destination": "/opt/certs",
        "Mode": "z",
        "RW": true,
        "Propagation": "rprivate"
    },
    {
        "Type": "bind",
        "Source": "/var/run/docker.sock",
        "Destination": "/var/run/docker.sock",
        "Mode": "ro",
        "RW": false,
        "Propagation": "rprivate"
    }
],

I update the certificate manually, then just to be sure, I touch the dynamic/ssl.yml file to make traefik notice changes, but it does not. It keeps serving the old certificate and my only option is to restart traefik, after which point it loads the new certificate. I do not want to restart traefik, as that means some active connections are dropped, which is bad, I need hot-reloading for certificates.

I do not understand why it does not hot-reload, when every tutorial/doc/guide on the net says it should work. Does anyone have any idea what I am doing wrong?

Thanks!

PS: I made sure that the new certificates are present and also reflected inside of the container, I also verified the date on the certificate with openssl and yep, it is the new certificate. I also verified the 'date modified' field has changed on the dynamic configuration file when using touch both outside and inside of the container.

Works for me on Linux. We update the cert files, touch the dynamic config file in a watched folder, Traefik reloads the TLS certs.

Do you see any reload activity in Traefik debug log when you touch the dynamic config file?

What’s your OS, Docker version, virtualization?

@bluepuma77

VM host: Proxmox 8.3.3
OS: Debian 12
Docker:

Client: Docker Engine - Community
 Version:           27.5.1
 API version:       1.47
 Go version:        go1.22.11
 Git commit:        9f9e405
 Built:             Wed Jan 22 13:41:17 2025
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          27.5.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.11
  Git commit:       4c9b3b0
  Built:            Wed Jan 22 13:41:17 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.25
  GitCommit:        bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
 runc:
  Version:          1.2.4
  GitCommit:        v1.2.4-0-g6c52b3f
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Traefik: 3.3.3

Log when debug active, and after touching dynamic/ssl.yml:

2025-02-22T22:48:56+01:00 DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{"middlewares":{"wwwtohttps":{"redirectRegex":{"permanent":true,"regex":"^https?://(?:www\\.)?(.+)","replacement":"https://${1}"}}},"routers":{"http-catchall":{"entryPoints":["web"],"middlewares":["wwwtohttps"],"rule":"HostRegexp(`{host:(www\\.)?.+}`)","service":"noop@internal"},"wwwsecure-catchall":{"entryPoints":["websecure"],"middlewares":["wwwtohttps"],"rule":"HostRegexp(`{host:(www\\.).+}`)","service":"noop@internal","tls":{}}}},"tcp":{},"tls":{},"udp":{}} providerName=file
2025-02-22T22:48:56+01:00 DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:127 > Skipping unchanged configuration providerName=file
2025-02-22T22:48:58+01:00 DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{"middlewares":{"wwwtohttps":{"redirectRegex":{"permanent":true,"regex":"^https?://(?:www\\.)?(.+)","replacement":"https://${1}"}}},"routers":{"http-catchall":{"entryPoints":["web"],"middlewares":["wwwtohttps"],"rule":"HostRegexp(`{host:(www\\.)?.+}`)","service":"noop@internal"},"wwwsecure-catchall":{"entryPoints":["websecure"],"middlewares":["wwwtohttps"],"rule":"HostRegexp(`{host:(www\\.).+}`)","service":"noop@internal","tls":{}}}},"tcp":{},"tls":{},"udp":{}} providerName=file
2025-02-22T22:48:58+01:00 DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:127 > Skipping unchanged configuration providerName=file

I can see updates for the secondary file, which is www-redirect.yml which I did not touch (literally) and did not edit. All I did was touch /opt/traefik/dynamic/ssl.yml. So it is weird that traefik did not register that, and instead tried to re-read a different file with which I did nothing.

File permissions should be fine:

drwxr-x--- 2 root root 4096 Feb 22 17:24 .
drwxr-x--- 3 root root 4096 Feb 22 21:47 ..
-rw-r--r-- 1 root root  128 Feb 22 21:48 ssl.yml
-rw-r----- 1 root root  685 Feb 22 17:19 www-redirect.yml

@bluepuma77 Any idea what I am doing wrong? Maybe the name of the dynamic file is somehow causing problems? I can try to rename it from ssl.yml to something else to see if changes get noticed when I touch the file.

Where are the dynamic files located? On the host or within the VM, outside Docker? Watching files through multiple layers is complicated and not always works.

@bluepuma77

The files are located on the VM (which is running docker and traefik), as all other configuration files, there is a main directory in /opt/traefik, which has the traefik.yml and /opt/traefik/dynamic which has the ssl.yml and www-redirect.yml.

Then I map the main folder into the traefik container like so: /opt/traefik:/opt/traefik:ro. I also map a directory with the certificates like so: /opt/certs:/opt/certs:ro.

So there is only one layer - how Docker maps files into containers.

There is one mistake in my message:

Then I map the main folder into the traefik container like so: /opt/traefik:/opt/traefik:ro .

I map the main folder into traefik like this: /opt/traefik:/etc/traefik:ro.

I am having exactly the same problem because I use grafana for metrics and I still see my old certificate expiration date. I did a touch every time when the certificate is renewed (acme.sh have that option)

Works for me, I update the TLS cert files and then touch the Traefik dynamic config file, which triggers a reload.

Share your full Traefik static and dynamic config, and docker-compose.yml if used.

Enable and check Traefik debug log.

this is the debug log after do the touch:

2025-03-12T10:34:12+01:00 DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config=providerName=file
2025-03-12T10:34:12+01:00 DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:127 > Skipping unchanged configuration providerName=file

reload is done correctly from my metrics I can see it: Last Reload: 12/03/2025, 10:34:10

my docker-compose (I have a lot of certificates in diferent locations that's why I am mapping on that way, can be the problem?)

services:
  traefik:
    container_name: traefik
    image: traefik:v3.3.4
    ports:
      - 380:80
      - 3443:443
      - 82:8080
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /volume1/docker/traefik/config:/etc/traefik
      - /volume1/docker/traefik/certs:/etc/traefik_certs
      - /volume1/docker/traefik/logs:/etc/traefik_logs
      - /volume1/docker/traefik/pages:/pages
      - /usr/syno/etc/certificate/_archive/IXSW77/fullchain.pem:/etc/traefik_certs/xx_fullchain.pem:r
      - /usr/syno/etc/certificate/_archive/IXSW77/privkey.pem:/etc/traefik_certs/xx_privkey.pem:r
    environment:
      - TZ=Europe/Madrid
    networks:
      traefik_macvlan:
        ipv4_address: 192.168.0.233
      default:
        ipv4_address: 172.25.50.30
    extra_hosts:
      - host.docker.internal:172.17.0.1

networks:
  default:
    name: traefik
    ipam:
      config:
        - subnet: 172.25.50.0/24
  traefik_macvlan:
    name: traefik_macvlan
    driver: macvlan      
    driver_opts:
      parent: ovs_eth0
    ipam:
      driver: default
      config:
        - subnet: 192.168.0.0/24 
          ip_range: 192.168.0.235/30
          gateway: 192.168.0.1
          aux_addresses:
            host: 192.168.0.234

my traeefi.yml

providers:
  file:
    directory: /etc/traefik/
    watch: true
  docker:
    endpoint: "unix:///var/run/docker.sock"
    watch: true
    exposedByDefault: false

and my dyanmic_config.yml refering to TLS

tls:
  certificates:
    - certFile: /etc/traefik_certs/xx_fullchain.pem
      keyFile: /etc/traefik_certs/xx_privkey.pem
  options:
    default:
      minVersion: VersionTLS12
      curvePreferences:
        - X25519
        - CurveP256
        - CurveP384
      cipherSuites:
        - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
        - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305

the cerficiate inside my container (end of privkey):

2rPRsAeNHApi/FtOofawtQ==
-----END PRIVATE KEY-----

and the one from my NAS:

oONcOmuvHdUq12S+405iyKzj
-----END PRIVATE KEY-----

also I can double check from metrics or checking the certificate from the browser url that its still the old expiration.

[UPDATE]

I modified my docker-compose as this to remove old mapping of /etc/treafik_certs:

    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /volume1/docker/traefik/config:/etc/traefik
      #- /volume1/docker/traefik/certs:/etc/traefik_certs
      - /volume1/docker/traefik/logs:/etc/traefik_logs
      - /volume1/docker/traefik/pages:/pages
      - /usr/syno/etc/certificate/_archive/IXSW77/fullchain.pem:/etc/traefik_certs/xx_fullchain.pem
      - /usr/syno/etc/certificate/_archive/IXSW77/privkey.pem:/etc/traefik_certs/xx_privkey.pem

and I receive this in logs but the certificate is still the old expiration:

2025-03-12T11:00:09+01:00 DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config=providerName=file

2025-03-12T11:00:09+01:00 DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:97 > No store is defined to add the certificate MIIFEzCCA/ugAwIBAgISAwdbPF5Moed8NsCoCeckaHbPMA0GCS, it will be added to the default store

2025-03-12T11:00:09+01:00 DBG github.com/traefik/traefik/v3/pkg/tls/certificate.go:132 > Adding certificate for domain(s) *.xx.synology.me,xx.synology.me

my certs inside the container are listed as here with these rights. The one I am testing is still showing the old date of Mar 10 instead of Mar 12 (the last 2 ones with 0 hardlinks)

my traefik config folder rights:

I found the problem and the solution, it seems to be related to the certificate permissions on the host system. Finally I run a scheduled task every day to copy all my certificates to /volem1/docker/traefik/certs/ folder and give them 644 permissions and use that folder for the volume mapping in my docker-compose. Now traefik can read them again if I touch a config file for the hot reloading.

The only concern is that after being renewed now I can see both certificates in my metrics and my chrome browser where I already had the old certificate it still shows the old one (also with closing the browser or restarting the computer but without a cache clean), any other new browser shows the new one.

who can be that? Restarting traefik solves it in my metrics. going from these 5 certificates to my 4 existing ones (removing the duplicated expired one)

You use the same filenames and overwrite the files every time?

yes

I have that script

cp /usr/syno/etc/certificate/_archive/5smsQF/fullchain.pem /volume1/docker/traefik/certs/xx_fullchain.pem

cp /usr/syno/etc/certificate/_archive/5smsQF/privkey.pem /volume1/docker/traefik/certs/xx_privkey.pem

chmod 644 /volume1/docker/traefik/certs/xx_fullchain.pem
chmod 644 /volume1/docker/traefik/certs/xx_privkey.pem

chown admin:users /volume1/docker/traefik/certs/xx_fullchain.pem
chown admin:users /volume1/docker/traefik/certs/xx_privkey.pem

touch /volume1/docker/traefik/config/dynamic_config.yml

where then in my docker compose:

volumes:
      - /volume1/docker/traefik/certs:/etc/traefik_certs

and in my dynamic config:

tls:
  certificates:
    - certFile: /etc/traefik_certs/xx_fullchain.pem
      keyFile: /etc/traefik_certs/xx_privkey.pem

so in my volume and container I only see one file, not two duplicated.