Help with allowEmptyServices in Docker Swarm

I've configured traefik v3 to allow empty services:

  traefik-proxy:
    image: traefik:v3.3.3
    command:
      - "--global.checknewversion=true"
      - "--global.sendanonymoususage=false"
      # Enable Swarm Provider (Replaces Docker Swarm Mode from v2)
      - "--providers.swarm=true"
      - "--providers.swarm.allowEmptyServices=true"
      - "--providers.swarm.exposedbydefault=false"
      - "--providers.swarm.network=traefik-external"

services:

admin@swarm-mgr03:~$ docker service ls
ID             NAME                              MODE         REPLICAS   IMAGE                                                 PORTS
p6chig0lyofr   stirlingpdf_db                    replicated   1/1        postgres:17.3-alpine                                  
3ny82s9hydre   stirlingpdf_stirling-pdf          replicated   1/1        stirlingtools/stirling-pdf:0.42.0-fat                 
xfagdejgr3mp   traefik_sablier                   replicated   1/1        sablierapp/sablier:1.8.4                              
hav8u2a83nre   traefik_traefik-proxy             replicated   1/1        traefik:v3.3.3                                        

if i go to the service, no issues:

 ~  curl -L http://pdf.mydomain.local                                                                                                                                                                                      <!DOCTYPE html>
<html lang="en" dir="ltr" data-language="en_US">
  <head>
  
  <!-- Title -->
  <title>Stirling-PDF - Sign in</title>

  <!-- Metadata -->
  <meta charset="utf-8">
  <meta name="description" content="Stirling-PDF - Sign in">
  <meta name="msapplication-TileColor" content="#2d89ef">
  <meta name="theme-color" content="#ffffff">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">

  <!-- Icons -->
  <link rel="apple-touch-icon" sizes="180x180" href="/apple-touch-icon.png">
  <link rel="icon" type="image/png" sizes="32x32" href="/favicon-32x32.png">

if i scale down:

admin@swarm-mgr03:~$ docker service ls
ID             NAME                              MODE         REPLICAS   IMAGE                                                 PORTS
p6chig0lyofr   stirlingpdf_db                    replicated   0/0        postgres:17.3-alpine                                  
3ny82s9hydre   stirlingpdf_stirling-pdf          replicated   0/0        stirlingtools/stirling-pdf:0.42.0-fat                 
xfagdejgr3mp   traefik_sablier                   replicated   1/1        sablierapp/sablier:1.8.4                              
hav8u2a83nre   traefik_traefik-proxy             replicated   1/1        traefik:v3.3.3                                        

then i get a 404 error, but i thought it says the service should still be reachable?

 ~  curl -L http://pdf.mydomain.local                                                                                                                                                                                       ✔  01:33:35 PM 
404 page not found

I looked at this thread, but I know my healthchecks are working...

anyone have any hints that may help?

According to doc my understanding is you should get a 503 instead of 404 when no instance/replica is available.

That's what i thought as well, but it doesn't seem to work on my end. I even see the service disappear from the traefik dashboard when i scale to 0. is there a way to verify that the allowEmptyServices is working?

I quickly tested with Traefik v3.3, v3.0, v2.11, v2.9 with allowEmptyServices (v2 using providers.docker.swarmMode=true) and it always has the same behavior: when the service is scaled to 0, the Traefik router and service is removed, resulting in http status 404.

Not sure if this is a bug (report here) or if we understand the functionality wrong. Maybe it's just for unhealthy services, and they must still exist:

allowEmptyServices
If the parameter is set to true, any servers load balancer defined for Docker containers is created regardless of the healthiness of the corresponding containers.