Docker Compose restart not refreshing labels

I have traefik and several other containers running from the same docker-compose file. I updated the labels in the file on container X, for example adding these lines:

      - traefik.http.routers.hello-service-router.middlewares=test-auth
      - traefik.http.middlewares.test-auth.basicauth.users=[redacted]

If I run docker compose restart traefik hello-service, the two containers are restarted, but the labels are not picked up. The basic auth is not enforced.
But when I run docker compose down && docker compose up -d, the containers are restarted and the labels are picked up, with basic auth being enforced.

Anyone have any idea why? Isn't the point of docker compose restart that it restarts the container and picks up the updates in docker-compose? Or is this something specific to traefik behavior? Is there some cache that's not being cleaned?

Ah, found the problem here.

If you make changes to your compose.yml configuration, these changes are not reflected after running this command.