Service IP address is off by one, Docker network is correct


Similar to an issue, that was unfortunately closed without an answer, I ran into a problem that I'm currently not able to solve myself:

I wanted to add a graphite/statsd service to my swarm which could later be used to get metrics from traefik itself. Adding the service works fine and it is picked up by traefik, however, the IP address is off by one.

I know that a lot of times a wrong IP address comes up, the culprit is the wrong docker network. However, the network seems to be correct:

Inspect Output of Service

        "Endpoint": {
            "Spec": {
                "Mode": "vip"
            "VirtualIPs": [
                    "NetworkID": "y1q27t0bdvx05e8ul2dpj0piv",
                    "Addr": ""

Dashboard Service View
CleanShot 2022-05-02 at 14.10.31

Both services (traefik and graphite) are defined in the same docker-compose file and deployed as one stack.

Services from other Stacks are picked up correctly by traefik with their correct IP addresses.

Am I overlooking something really simple here or is this unexpected behaviour?

Thanks a lot in advance!

The complete docker-compose.yml:

version: '3.3'

    # Use the latest v2.7.x Traefik image available
    image: traefik:v2.7
      # Listen on port 80, default for HTTP, necessary to redirect to HTTPS
      - 80:80
      # Listen on port 443, default for HTTPS
      - 443:443
          # Only deploy traefik to the manager node as it has the certificate volume
          # TODO: Use a specific node id if there are multiple managers
          - node.role == manager
        # Enable Traefik for this service, to make it available in the public network
        traefik.enable: "true"
        # Use the traefik-public network (declared below) traefik-public
        # Use the custom label "traefik.constraint-label=traefik-public"
        # This public Traefik will only use services with this label
        # That way you can add other internal Traefik instances per stack if needed
        traefik.constraint-label: traefik-public
        # admin-auth middleware with HTTP Basic auth
        # Using the environment variables USERNAME and HASHED_PASSWORD
        traefik.http.middlewares.admin-auth.basicauth.users: ${USERNAME_AND_HASHED_PASSWORD?Variable not set}
        # https-redirect middleware to redirect HTTP to HTTPS
        # It can be re-used by other stacks in other Docker Compose files
        traefik.http.middlewares.https-redirect.redirectscheme.scheme: https
        traefik.http.middlewares.https-redirect.redirectscheme.permanent: "true"
        # traefik-http set up only to use the middleware to redirect to https
        # Uses the environment variable DOMAIN
        traefik.http.routers.traefik-public-http.rule: Host(`${DOMAIN?Variable not set}`)
        traefik.http.routers.traefik-public-http.entrypoints: http
        traefik.http.routers.traefik-public-http.middlewares: https-redirect
        # traefik-https the actual router using HTTPS
        # Uses the environment variable DOMAIN
        traefik.http.routers.traefik-public-https.rule: Host(`${DOMAIN?Variable not set}`)
        traefik.http.routers.traefik-public-https.entrypoints: https
        traefik.http.routers.traefik-public-https.tls: "true"
        # Use the special Traefik service api@internal with the web UI/Dashboard
        traefik.http.routers.traefik-public-https.service: api@internal
        # Use the "le" (Let's Encrypt) resolver created below
        traefik.http.routers.traefik-public-https.tls.certresolver: le
        # Enable HTTP Basic auth, using the middleware created above
        traefik.http.routers.traefik-public-https.middlewares: admin-auth
        # Define the port inside of the Docker service to use 8080
      # Add Docker as a mounted volume, so that Traefik can read the labels of other services
      - /var/run/docker.sock:/var/run/docker.sock:ro
      # Mount the volume to store the certificates
      - traefik-public-certificates:/certificates
      # Enable Docker in Traefik, so that it reads labels from Docker services
      - --providers.docker
      # Add a constraint to only use services with the label "traefik.constraint-label=traefik-public"
      - --providers.docker.constraints=Label(`traefik.constraint-label`, `traefik-public`)
      # Do not expose all Docker services, only the ones explicitly exposed
      - --providers.docker.exposedbydefault=false
      # Enable Docker Swarm mode
      - --providers.docker.swarmmode
      # Create an entrypoint "http" listening on port 80
      - --entrypoints.http.address=:80
      # Create an entrypoint "https" listening on port 443
      - --entrypoints.https.address=:443
      # Create the certificate resolver "le" for Let's Encrypt, uses the environment variable EMAIL
      -${EMAIL?Variable not set}
      # Store the Let's Encrypt certificates in the mounted volume
      # Use the TLS Challenge for Let's Encrypt
      - --certificatesresolvers.le.acme.tlschallenge=true
      # Enable the access log, with HTTP requests
      - --accesslog
      # Enable the Traefik log, for configurations and errors
      - --log
      # Enable the Dashboard and API
      - --api
      # Use the public network created to be shared between Traefik and
      # any other service that needs to be publicly available with HTTPS
      - traefik-public
  # Optional graphite + statsd    
    image: graphiteapp/graphite-statsd
      - traefik-public
        traefik.enable: "true" traefik-public
        traefik.constraint-label: traefik-public
        # HTTP requests, redirect to HTTPS
        traefik.http.routers.graphite-public-http.rule: Host(`${DOMAIN?Variable not set}`) && Path(`/graphite`)
        traefik.http.routers.graphite-public-http.entrypoints: http
        traefik.http.routers.graphite-public-http.middlewares: https-redirect
        # HTTPS requests, use a letsencrypt certificate
        traefik.http.routers.graphite-public-https.tls.certresolver: le
        traefik.http.routers.graphite-public-https.rule: Host(`${DOMAIN?Variable not set}`) && Path(`/graphite`)
        traefik.http.routers.graphite-public-https.entrypoints: https
        traefik.http.routers.graphite-public-https.tls: "true"
        # Forward the traffic to the nginx inside the container 80
        # add the same basic auth to graphite over HTTPS that we use for traefik itself
        traefik.http.routers.graphite-public-https.middlewares: admin-auth

  # Create a volume to store the certificates, there is a constraint to make sure
  # Traefik is always deployed to the same Docker node with the same volume containing
  # the HTTPS certificates

  # Use the previously created public network "traefik-public", shared with other
  # services that need to be publicly available via this Traefik
    external: true

Traefik will use the container ips not the vip of the service.

If you inspect the running container of the service you will observe it matches.

1 Like

You are absolutely correct... the problem lies within the graphite container itself, it's sometimes a bit confusing where an error message comes from.

Thanks a lot, makes perfectly sense.

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.