Traefik works but only if loadbalancer.server.port is included

I'm very puzzled by the following, hopefully someone can shine a light on why this happening:

Traefik works but only if I include the label:
traefik.http.services.random-xbcz.loadbalancer.server.port: "80"

But this label should not be needed...

Without this label, going to the Traefik url results in

404 page not found

Whatever I use as an service name it works. The service name itself is nowhere used/referenced in the compose file or traefik configurations files.

I don't understand why I can't leave out the label:
traefik.http.services.random-xbcz.loadbalancer.server.port: "80"

Anybody knows what could be the reason?

Setup information

Network macvlan.
Ping from the host to the traefik IP 192.168.1.231 works.

The traefik dashboard shows (if it works):

|| status || name            || type       || servers || providers ||
|   ok     | api@internal    | -            | 0       | ae          |
|   ok     | random-xbcz     | loadbalancer | 1       | docker      |

Going to the random-xbcz service page shows:

status: down (red)
url: http://192.168.1.231
weight: 1

Although the added link refers to service that is down, Traefik is working...

The relevant compose parts

   labels:
     traefik.enable: "true"
     traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto: https
     traefik.http.routers.traefik01-web-club-secure.entrypoints: https
     traefik.http.routers.traefik01-web-club-secure.middlewares: traefik-auth,sslheader
     traefik.http.routers.traefik01-web-club-secure.rule: Host(`traefik01.web.example.com`)
     traefik.http.routers.traefik01-web-club-secure.service: api@internal
     traefik.http.routers.traefik01-web-club-secure.tls: "true"
     traefik.http.routers.traefik01-web-club-secure.tls.certresolver: mijnhost
     traefik.http.services.random-xbcz.loadbalancer.server.port: "80"

   networks:
     macvlan01:
       ipv4_address: 192.168.1.231

networks: 
 macvlan01: 
   name: macvlan01 
   external: true

Traefik config:

$ cat config/traefik.yml

api:
  dashboard: true
  debug: true


entryPoints:
  http:
    address: ":80"

  http-external:
    address: ":81"

  https:
    address: ":443"
  
  https-external:
    address: ":444"

serversTransport:
  insecureSkipVerify: true

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false

  file:
    filename: /config.yml
    watch: true

  redis:
    endpoints:
      - "redis:6379"

Hopefully someone can shine a light on why this happening.

Traefik needs to know which port is used by the internal target service. When using Docker containers, it will try to find a port defined in Dockerfile with EXPOSE. If multiple ports are defined, it will use the first one. So it's always recommended to define the target port via labels to make sure it's the correct one (example).

@bluepuma77 thanks for your support.

I'm still confused as the container exposes only port 80.

docker inspect traefik01 \
  --format '{{json .Config.ExposedPorts}}'

{"80/tcp":{}}

The Traefik v3.7.7 image exposes only 80/tcp. My router already points to api@internal, yet the dashboard only works when I define an otherwise unused Docker service with loadbalancer.server.port=80. With the result that there is now a service which is down. That does not feel correct.

Is this intended or expected behavior, or is it a bug/regression in the Docker provider?

Check Port Detection docs.

Alright checked the documentation:

Port Detection

Traefik retrieves the private IP and port of containers from the Docker API.

Port detection for private communication works as follows:

  • If a container exposes a single port, then Traefik uses this port.

Only one port is exposed.
This is what should happen.

  • If a container multiple ports, then Traefik uses the lowest port. E.g. if 80 and 8080 are exposed, Traefik will use 80

This is not my situation.
If it would be, the lowest port (80) should be used. Does not happen.

  • If a container does not expose any port, or the selection from multiple ports does not fit, then you must manually specify which port Traefik should use for communication by using the label traefik.http.services.<service_name>.loadbalancer.server.port (Read more on this label in the dedicated section in routing).

This makes it work, but it should not work this way.
It results in a "dead" service (status down), but the dashboard is working.

Host networking

When exposing containers that are configured with host networking, the IP address of the host is resolved as follows:

  • try a lookup of host.docker.internal
  • if the lookup was unsuccessful, try a lookup of host.containers.internal, (Podman equivalent of host.docker.internal)
  • if that lookup was also unsuccessful, fall back to 127.0.0.1

On Linux, for versions of Docker older than 20.10.0, for host.docker.internal to be defined, it should be provided as an extra_host to the Traefik container, using the --add-host flag. For example, to set it to the IP address of the bridge interface (docker0 by default): --add-host=host.docker.internal:172.17.0.1.

Tried this with:

labels:
  traefik.http.services.host-test.loadbalancer.servers.url: "http://host.docker.internal:80"

The dashboard works, but that is because port 80 is again defined.

This is really about the dashboard.
Only the dashboard does not work, the other services or working correctly.

This looks like a bug to me, or do I miss to see something (less) obvious?

If you want to dig deeper, then you should provide full Traefik static and dynamic config, and full compose file(s). And describe what you want to do, are you talking about the Traefik dashboard being unavailable, from the same Traefik instance?

Hi @bluepuma77

thanks for your support, below my Traefik setup.
I have rewritten the 2 domains that I use with respectively example.org and mydomain.org.

Hopefully you see what is causing the issue I face with my Traefik Dashboard.

Information to investigate why api@internal is not working

Traefik itself is working

Traefik dashboard from the same Traefik instance url: https://traefik01.web.example.org/dashboard/

Traefik version: 3.7.7

Only one port exposed:

docker inspect traefik01 --format '{{json .Config.ExposedPorts}}'

{"80/tcp":{}}

Expected is that the Traefik dashboard can be reached via api@internal, but it isn't. An additional service must be specified to address port 80.

Compose.yaml

services:
  traefik:
    image: traefik:v3.7
    container_name: traefik01
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    networks:
      macvlan01:
        ipv4_address: 192.168.1.xxx

    environment:
      CF_API_EMAIL: ${CF_API_EMAIL}
      CF_DNS_API_TOKEN: ${CF_DNS_API_TOKEN}
      MIJNHOST_API_KEY: ${MIJNHOST_API_KEY}
      TRAEFIK_DASHBOARD_CREDENTIALS: ${TRAEFIK_DASHBOARD_CREDENTIALS}

    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./config/traefik.yml:/traefik.yml:ro
      - ./config/config.yml:/config.yml:ro

      - /srv/docker/swarm/.volumes/router/traefik01/pki:/etc/pki
      - /srv/docker/swarm/.volumes/router/traefik01/logs:/var/logs

    labels:
      dockhand.url: "[traefik](https://traefik01.web.example.org)"

      traefik.enable: true
      traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto: "https"
      traefik.http.routers.traefik01-blue-secure.middlewares: "traefik-auth,sslheader"
      traefik.http.routers.traefik01-web-example-org-secure.entrypoints: "https"
      traefik.http.routers.traefik01-web-example-org-secure.rule: "Host(`traefik01.web.example.org`)"   
      traefik.http.routers.traefik01-web-example-org-secure.tls: true
      traefik.http.routers.traefik01-web-example-org-secure.tls.certresolver: "mijnhost"
      traefik.http.routers.traefik01-web-example-org-secure.middlewares: "traefik-auth,sslheader"
      traefik.http.routers.traefik01-web-example-org-secure.service: "api@internal"
      traefik.http.services.api.loadbalancer.server.port: 80

networks:
  macvlan01:
    external: true

config/Traefik.yml

api:
  dashboard: true
  debug: true

log:
   level: DEBUG
   format: json
   filePath: /var/logs/traefik.log
 
accesslog:
  filePath: /var/logs/access.log
  fields:
    names:
      StartUTC: drop

entryPoints:

  ssh:
    address: ":22"

  ssh24:
    address: ":24"

  http:
    address: ":80"

  http-external:
    address: ":81"

  https:
    address: ":443"
  
  https-external:
    address: ":444"

serversTransport:
  insecureSkipVerify: true

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false

  file:
    filename: /config.yml
    watch: true

  redis:
    endpoints:
      - "redis:6379"

certificatesResolvers:

...

config/config.yml

http:

  routers:

    grafana-cloud:
      entryPoints:
         - https
      rule: "Host(`db01.web.mydomain.org`)"
      tls: true
      service: grafana-cloud

  services:
      

    grafana-cloud:
      loadbalancer:
        servers:
          - url: "http://cloud.mydomain.org/:3000"

  middlewares:

    grafana-db01-example-to-mydomain-rewrite:
      redirectRegex:
        regex: "^https://db01.example.org:444/(.*)"
        replacement: "https://db01.mydomain.org/${1}"

    grafana-db01-example-headers-manipulaton:
      headers:
        customrequestheaders:
          X-Script-Name: "X-Forwarded-For"

    redirect-http-to-https-once:
      redirectScheme:
        scheme: https

    redirect-http-to-https-permanent:
      redirectScheme:
        scheme: https
        permanent: true

    default-headers:
      headers:
        frameDeny: true
        browserXssFilter: true
        contentTypeNosniff: true
        forceSTSHeader: true
        stsIncludeSubdomains: true
        stsPreload: true
        stsSeconds: 15552000
        customFrameOptionsValue: SAMEORIGIN
        customRequestHeaders:
          X-Forwarded-Proto: https

    default-allowlist:
      ipAllowList:
        sourceRange:
        - "192.168.1.0/16"
        - "172.16.0.0/12"
        - "10.0.0.0/8"

    secured:
      chain:
        middlewares:
        - default-allowlist
        - default-headers

tcp:
  routers:

    owncloud:
      entryPoints: "https-external"
      rule: "HostSNI(`owncloud.mydomain.org`)"
      service: owncloud
      tls:
        passthrough: true

  services:
    owncloud:
      loadbalancer:
        servers:
          - address: "cloud.mydomain.org:443"