Traefik is trying to target IP from different docker network

Traefik Traefik v2
1

Hello. I've successfully spun up keycloak with Teaefik. However only if those services are on the same network and doesn't have some internal subnetworks.

Now i wanted to split those two services to different docker-compose files. In one, i have:

version: "3"
services:

  traefik:
    image: "traefik:latest"
    container_name: "traefik"
    command:
      - "--log.level=DEBUG"
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    stdin_open: true # docker run -i
    tty: true        # docker run -t
    ports:
      - "80:80"
      - "443:443"
      - "4130:8080"
    environment:
      - CF_API_EMAIL=petko252@gmail.com
      - CF_API_KEY=d8549cc82a4cd7c439e86aaee9a3a23155072
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - /home/docker/docker_traefik:/etc/traefik
      - traefik_ssl_certs:/etc/traefik/certs
      - /home/docker/docker_traefik/logs:/var/log/traefik

  crowdsec:
    container_name: crowdsec
    image: crowdsecurity/crowdsec:latest
    stdin_open: true # docker run -i
    tty: true        # docker run -t
    environment:
      GID: "1001"
      COLLECTIONS: "crowdsecurity/linux crowdsecurity/traefik crowdsecurity/http-cve crowdsecurity/whitelist-good-actors crowdsecurity/sshd"
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /home/docker/docker_crowdsec/crowdsec:/etc/crowdsec
      - /var/log/auth.log:/var/log/auth.log:ro
      - /var/log/crowdsec:/var/log/crowdsec:ro
      - /home/docker/docker_crowdsec/database:/var/lib/crowdsec/data
    restart: unless-stopped
    security_opt:
      - no-new-privileges=true
    hostname: crowdsec
    
  bouncer-traefik:
    container_name: crowdsec-bouncer-traefik
    image: fbonalair/traefik-crowdsec-bouncer:latest
    stdin_open: true # docker run -i
    tty: true        # docker run -t
    environment:
      CROWDSEC_BOUNCER_API_KEY: 51c497e5aca2fd552fa6c384954182c3 # the api key needs to be created of the crowdsec container with `docker compose exec -t crowdsec cscli bouncers add bouncer-traefik`
      CROWDSEC_AGENT_HOST: crowdsec:8080  
    restart: unless-stopped
    depends_on:
      - crowdsec
    hostname: crowdsec-bouncer-traefik


volumes:
  traefik_ssl_certs:

Which generates traefik_default network. Now, i deploy second docker compose file with Keycloak setup:

version: '3.9'

services:
  keycloak:
    image: docker.io/bitnami/keycloak:21
    container_name: keycloak
    environment:
      KEYCLOAK_DATABASE_HOST: "postgres"
      KEYCLOAK_DATABASE_NAME: "keycloak"
      DB_USER: "admin"
      DB_PASSWORD: "keycloak"
      KEYCLOAK_ADMIN_USER: "admin"
      KEYCLOAK_ADMIN_PASSWORD: "keycloak"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.keycloak-local.rule=Host(`keycloak.local.dashrave.eu`)"
      - "traefik.http.routers.keycloak-local.entrypoints=web, websecure"
      - "traefik.http.routers.keycloak-local.tls=true"
      - "traefik.http.routers.keycloak-local.tls.certresolver=cloudflare"
      - "traefik.http.routers.keycloak-local.tls.domains[0].main=local.dashrave.eu"
      - "traefik.http.routers.keycloak-local.tls.domains[0].sans=*.local.dashrave.eu"
    ports:
      - "4150:8080"
    depends_on:
      - postgres
    restart: unless-stopped
    networks:
      - local-keycloak
      - traefik_default

  postgres:
    image: postgres:11
    volumes:
      - /home/docker/docker_keycloak/pgdata:/var/lib/postgresql/data
    environment:
      POSTGRES_DB: "keycloak"
      POSTGRES_USER: "admin"
      POSTGRES_PASSWORD: "keycloak"
    restart: unless-stopped
    networks:
      - local-keycloak

networks:
  local-keycloak:
  traefik_default:
    external: true

I make it accessible from traefik default network, however only the frontend for security purposes. At this point the Keycloak frontend lives in two networks - local-keycloak (where it connects with it's database) and traefik_default (to be accessible for traefik) as shown here (traefik_default network):
traefik_default network
However, in the traefik logs, traefik wants to access an IP attached to keycloak frontend in local-keycloak network

level=debug msg="'504 Gateway Timeout' caused by: dial tcp 172.21.0.3:8080: i/o timeout"

Which traefik can't access understandably. How do i force traefik to access IP that is acessible in traefik_default network (172.19.0.5:8080)?

2

I recommend to use a dedicated Docker network, not default. Set the docker.network on provider or service, see simple Traefik example.

Why do you do this?

    stdin_open: true # docker run -i
    tty: true        # docker run -t

You can’t use traefik.yml and command at the same time for static config, decide for one.

3

By docker.network you mean setting in static configuration of traefik or rather manually creating / specifying docker network?

I know. I needed to use it to check if some files are present inside the filesystem.

Update: As i read the docker-compose file that you sent, i defined what network should docker create and now i have the same setup but instead of traefik_default as name, it is proxy as the name of the network. Anyway, nothing changed really ald traefik still target wrong IP. Shouldn't i specify the network as label inside of keycloak docker compose file?

4

Attach both Traefik and your target service to the same Docker network, set docker.network.

You need to use name when defining the Docker network in compose, without the network name will be changed to include the project/folder name.

5

Yup, i did that.
Traefik docker compose file:

version: "3"
services:

  traefik:
    image: "traefik:latest"
    container_name: "traefik"
    command:
      - "--log.level=DEBUG"
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    stdin_open: true # docker run -i
    tty: true        # docker run -t
    ports:
      - "80:80"
      - "443:443"
      - "4130:8080"
    networks:
      - proxy
    environment:
      - CF_API_EMAIL=petko252@gmail.com
      - CF_API_KEY=d8549cc82a4cd7c439e86aaee9a3a23155072
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - /home/docker/docker_traefik:/etc/traefik
      - traefik_ssl_certs:/etc/traefik/certs
      - /home/docker/docker_traefik/logs:/var/log/traefik

  crowdsec:
    container_name: crowdsec
    image: crowdsecurity/crowdsec:latest
    stdin_open: true # docker run -i
    tty: true        # docker run -t
    environment:
      GID: "1001"
      COLLECTIONS: "crowdsecurity/linux crowdsecurity/traefik crowdsecurity/http-cve crowdsecurity/whitelist-good-actors crowdsecurity/sshd"
    networks:
      - proxy
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /home/docker/docker_crowdsec/crowdsec:/etc/crowdsec
      - /var/log/auth.log:/var/log/auth.log:ro
      - /var/log/crowdsec:/var/log/crowdsec:ro
      - /home/docker/docker_crowdsec/database:/var/lib/crowdsec/data
    restart: unless-stopped
    security_opt:
      - no-new-privileges=true
    hostname: crowdsec
    
  bouncer-traefik:
    container_name: crowdsec-bouncer-traefik
    image: fbonalair/traefik-crowdsec-bouncer:latest
    stdin_open: true # docker run -i
    tty: true        # docker run -t
    environment:
      CROWDSEC_BOUNCER_API_KEY: 51c497e5aca2fd552fa6c384954182c3 # the api key needs to be created of the crowdsec container with `docker compose exec -t crowdsec cscli bouncers add bouncer-traefik`
      CROWDSEC_AGENT_HOST: crowdsec:8080
    networks:
      - proxy
    restart: unless-stopped
    depends_on:
      - crowdsec
    hostname: crowdsec-bouncer-traefik

networks:
  proxy:
    name: proxy

volumes:
  traefik_ssl_certs:

I defined the network inside the traefik docker compose file and now i just need to specify the network to join inside second service container (Keycloak in my case):

version: '3.9'

services:
  keycloak:
    image: docker.io/bitnami/keycloak:21
    container_name: keycloak
    environment:
      KEYCLOAK_DATABASE_HOST: "postgres"
      KEYCLOAK_DATABASE_NAME: "keycloak"
      DB_USER: "admin"
      DB_PASSWORD: "keycloak"
      KEYCLOAK_ADMIN_USER: "admin"
      KEYCLOAK_ADMIN_PASSWORD: "keycloak"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.keycloak-local.rule=Host(`keycloak.local.dashrave.eu`)"
      - "traefik.http.routers.keycloak-local.entrypoints=web, websecure"
      - "traefik.http.routers.keycloak-local.tls=true"
      - "traefik.http.routers.keycloak-local.tls.certresolver=cloudflare"
      - "traefik.http.routers.keycloak-local.tls.domains[0].main=local.dashrave.eu"
      - "traefik.http.routers.keycloak-local.tls.domains[0].sans=*.local.dashrave.eu"
    ports:
      - "4150:8080"
    depends_on:
      - postgres
    restart: unless-stopped
    networks:
      - local-keycloak
      - proxy

  postgres:
    image: postgres:11
    volumes:
      - /home/docker/docker_keycloak/pgdata:/var/lib/postgresql/data
    environment:
      POSTGRES_DB: "keycloak"
      POSTGRES_USER: "admin"
      POSTGRES_PASSWORD: "keycloak"
    restart: unless-stopped
    networks:
      - local-keycloak

networks:
  local-keycloak:
  proxy:
    external: true

Inside the "proxy" network i see these containers:
image
Which is correct. However my issue is that traefik is targeting IP 172.23.0.3:8080 which is the IP of Keycloak but not inside the "proxy" network, but inside the "local-keycloak" network. I want traefik to target 172.22.0.5:8080 so the reachable IP.

local-keycloak network:
image

6

FIXED.

I misunderstood what you said by docker.network. Later i noticed there's label traefik.docker.network=proxy which is defining which network traefik should use to target IPs.

So yeah, you were right, and i misunderstood your statement. I'm sorry and also glad that we could solve it! Thank you, @bluepuma77 !

7

UPDATE, @bluepuma77

I don't know how but as soon as i started other container that do work perfectly with traefik and i had 0 problems with it, keycloak stopped working behind traefik and now i get:

time="2023-06-10T16:01:36Z" level=debug msg="'502 Bad Gateway' caused by: dial tcp 192.168.32.5:8080: connect: connection refused"time="2023-06-10T16:01:36Z" level=debug msg="'502 Bad Gateway' caused by: dial tcp 192.168.32.5:8080: connect: connection refused"

The IP in the log is targeting Keycloak's IP inside the "proxy" network which is correct and it should be like that. However, Keycloak somehow refuses the connection. Actually, i don't know if the connection is refused by Keycloak itself or there's issue in the configuration again.

Any clue?

PS: I set the traefik.docker.network=proxy inside the service's (Keycloak's) docker compose which is the right way to do it i guess.

(another) UPDATE:
So it works again. Keycloak probably makes fun of me cause it started working out of nowhere. However, after Keycloak docker compose restart, i noticed there's some delay. So it first writes Bad gateway and after couple of seconds it start to work properly. It's probably issue with Keycloak so i'm yet again closing this topic as solved by your first comment.

8

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.