Help with allowEmptyServices in Docker Swarm

I've configured traefik v3 to allow empty services:

  traefik-proxy:
    image: traefik:v3.3.3
    command:
      - "--global.checknewversion=true"
      - "--global.sendanonymoususage=false"
      # Enable Swarm Provider (Replaces Docker Swarm Mode from v2)
      - "--providers.swarm=true"
      - "--providers.swarm.allowEmptyServices=true"
      - "--providers.swarm.exposedbydefault=false"
      - "--providers.swarm.network=traefik-external"

services:

admin@swarm-mgr03:~$ docker service ls
ID             NAME                              MODE         REPLICAS   IMAGE                                                 PORTS
p6chig0lyofr   stirlingpdf_db                    replicated   1/1        postgres:17.3-alpine                                  
3ny82s9hydre   stirlingpdf_stirling-pdf          replicated   1/1        stirlingtools/stirling-pdf:0.42.0-fat                 
xfagdejgr3mp   traefik_sablier                   replicated   1/1        sablierapp/sablier:1.8.4                              
hav8u2a83nre   traefik_traefik-proxy             replicated   1/1        traefik:v3.3.3                                        

if i go to the service, no issues:

 ~  curl -L http://pdf.mydomain.local                                                                                                                                                                                      <!DOCTYPE html>
<html lang="en" dir="ltr" data-language="en_US">
  <head>
  
  <!-- Title -->
  <title>Stirling-PDF - Sign in</title>

  <!-- Metadata -->
  <meta charset="utf-8">
  <meta name="description" content="Stirling-PDF - Sign in">
  <meta name="msapplication-TileColor" content="#2d89ef">
  <meta name="theme-color" content="#ffffff">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">

  <!-- Icons -->
  <link rel="apple-touch-icon" sizes="180x180" href="/apple-touch-icon.png">
  <link rel="icon" type="image/png" sizes="32x32" href="/favicon-32x32.png">

if i scale down:

admin@swarm-mgr03:~$ docker service ls
ID             NAME                              MODE         REPLICAS   IMAGE                                                 PORTS
p6chig0lyofr   stirlingpdf_db                    replicated   0/0        postgres:17.3-alpine                                  
3ny82s9hydre   stirlingpdf_stirling-pdf          replicated   0/0        stirlingtools/stirling-pdf:0.42.0-fat                 
xfagdejgr3mp   traefik_sablier                   replicated   1/1        sablierapp/sablier:1.8.4                              
hav8u2a83nre   traefik_traefik-proxy             replicated   1/1        traefik:v3.3.3                                        

then i get a 404 error, but i thought it says the service should still be reachable?

 ~  curl -L http://pdf.mydomain.local                                                                                                                                                                                       ✔  01:33:35 PM 
404 page not found

I looked at this thread, but I know my healthchecks are working...

anyone have any hints that may help?

According to doc my understanding is you should get a 503 instead of 404 when no instance/replica is available.

That's what i thought as well, but it doesn't seem to work on my end. I even see the service disappear from the traefik dashboard when i scale to 0. is there a way to verify that the allowEmptyServices is working?

I quickly tested with Traefik v3.3, v3.0, v2.11, v2.9 with allowEmptyServices (v2 using providers.docker.swarmMode=true) and it always has the same behavior: when the service is scaled to 0, the Traefik router and service is removed, resulting in http status 404.

Not sure if this is a bug (report here) or if we understand the functionality wrong. Maybe it's just for unhealthy services, and they must still exist:

allowEmptyServices
If the parameter is set to true, any servers load balancer defined for Docker containers is created regardless of the healthiness of the corresponding containers.

sorry for the delay, will open a report. thank you for all the testing.