Traefik not honoring traefik.docker.network

Traefik Traefik v2 (latest)
1

Greetings,

I'm running Traefik v2.0.2 and I've run into what feels like a bug, but I want to make sure I'm not screwing something simple up.

I have several containers that need to talk to a database, and I want traefik in front of them. I've set them up with multiple networks like this :

version: '3.6'
services:
  www-example-com:
    image: drupal:7
    container_name: www-example-com
    restart: always
    volumes:
      - /srv/drupal7/all:/var/www/html/sites/all
      - /srv/drupal7/example.com/:/var/www/html/sites/example.com
    labels:
      traefik.docker.network: "web"
      traefik.enable: true

      traefik.http.services.www-example-com-https.loadbalancer.server.port: 80
      traefik.http.services.www-example-com-https.loadbalancer.server.scheme: "http"

      traefik.http.routers.www-example-com.entrypoints: "http,https"
      traefik.http.routers.www-example-com.rule: "Host(`example.com`,`www.example.com`)"

      traefik.http.routers.www-example-com-https.entrypoints: "http,https"
      traefik.http.routers.www-example-com-https.rule: "Host(`example.com`,`www.example.com`)"
      traefik.http.routers.www-example-com-https.service: "www-example-com-https"
      traefik.http.routers.www-example-com-https.tls.certresolver: "letsEncrypt"
    networks:
      web:
        aliases:
          - www-example-com
      database:
networks:
  web:
  database:
    internal: true

There's nothing special in my traefik config, and if I remove the database network from the above config, things work as you would expect, though of course I can't get to the database. When I add in the database network, however, traefik starts having some problems. Occasionally it will work fine, and other times I get a gateway timeout. When I'm getting a gateway timeout, I can look at the traefik dashboard and see that the www-example-com-https service has an IP address in the database network, which traefik doesn't have access to.

Is this a known issue? Or have I somehow configured something wrong? My understanding is that the traefik.docker.network label identifies the network traefik should use to communicate with the container. Is that not correct?

Thanks!

2

Hi @XenoPhage, thanks for your interest. As it is hard to analyze the problem with partial information, can you share the whole setup (including Traefik configuration AND compose definition) please?

3

Sure. Here's the docker compose file :

version: '3.6'
services:
  traefik:
    image: traefik:latest
    container_name: traefik
    restart: always
    volumes:
      - /srv/traefik/traefik.toml:/traefik.toml:Z
      - /srv/traefik/servers.toml:/servers.toml:Z
      - /srv/traefik/acme.json:/acme.json:Z
      - /var/run/docker.sock:/var/run/docker.sock
    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
      - target: 8443
        published: 8443
        protocol: tcp
        mode: host
    networks:
      - web
      - bridge

networks:
  web:
  bridge:
    driver: bridge

Next up, the traefik.toml :

[global]
  checkNewVersion = true
  sendAnonymousUsage = false

[log]
  #level = "DEBUG"

[accessLog]

[serversTransport]
  insecureSkipVerify = true

[entryPoints]
  [entryPoints.http]
    address = ":80"

  [entryPoints.https]
    address = ":443"

  [entryPoints.health]
    address = ":8080"

  [entryPoints.traefik]
    address = ":8443"

[providers.docker]
  endpoint = "unix:///var/run/docker.sock"
  exposedByDefault = false
  defaultRule = "Host(`{{ .Name }}.{{ index .Labels \"customLabel\"}}`)"
  watch = true

[providers.file]
  filename = "servers.toml"

[api]
  dashboard = true
  #insecure = true

[ping]
  entryPoint = "health"

[certificatesResolvers.letsEncrypt.acme]
  email = "hostmaster@example.com"
  storage = "acme.json"
  [certificatesResolvers.letsEncrypt.acme.tlsChallenge]
  [certificatesResolvers.letsEncrypt.acme.httpChallenge]
    entryPoint = "http"

And finally servers.toml :

[http.routers]
  [http.routers.traefik]
    entryPoints = ["traefik"]
    rule="Host(`api.localhost`)"
    service="api@internal"

  [http.routers.traefik-https]
    entryPoints = ["traefik"]
    rule="PathPrefix(`/api`) || PathPrefix(`/dashboard`)"
    service="api@internal"
    middlewares=["traefikAuth@file","terry@file"]
    [http.routers.traefik-https.tls]

[http.services]

[http.middlewares]
  [http.middlewares.terry.headers]
    [http.middlewares.terry.headers.customResponseHeaders]
      X-Clacks-Overhead = "GNU Terry Pratchett"

  [http.middlewares.httpsredirect.redirectScheme]
    scheme = "https"
    port = "443"
    permanent = true

  [http.middlewares.traefikAuth.basicAuth]
    users = [
        "myuser:myseekritpassword",
      ]

[tls.options]
  [tls.options.default]
4

@dduportal Any thoughts on this?

1 Like
5

Sorry, Did not had time (yet) to work on this case? I'll let you know :slight_smile:

1 Like
7

when use multiple networks, maybe traefik get wrong ip

image
image991×284 18.8 KB

image
image1467×276 23.3 KB

update at 2019-11-07 19:05:56

https://docs.traefik.io/providers/docker/#network

solved

--providers.docker.network=overlay-network-internal
8

providers.docker.network provides a default to use for all containers, it's overridden by traefik.docker.network. In my situation, I'm defining traefik.docker.network and it's still getting the wrong IP.

1 Like
9

Are there any clues in the debug log?

10

image
image793×120 8.25 KB

my overlay-network-internal is external network. but your network is created by docker-compsoe.

11

Hi @XenoPhage, I was able to reproduce the behavior and locate the error.

What @minbaby said is true, the "warning" screenshot from the documentation should tip you: you have to change the value of traefik.docker.network to the real name of the network, not the reference name from within the docker-compose.yml alas. I was able to make your setup work by retrieving the name with the command docker network ls.

This is a limitation from docker-compose, whom "namespace" the networks by prefixing their names with <project name of the stack> and an underscore _.

In order to have a consistent and deterministic name, I used to specify the flag -p for the docker-compose (or set the environment variable $COMPOSE_PROJECT_NAME). More on this: https://docs.docker.com/compose/reference/overview/#use--p-to-specify-a-project-name.

1 Like
12

Why does it matter if it is an external network or an external network created by a docker-compose file (traefik's docker-compose)? I'm having a similar issue where treafik.docker.network isn't working, but I get the error:

error msg="Skip container service-COMPOSE_PROJECT_NAME_evaluated: field not found, node: network:{NETWORK}" providerName=docker