Containers within traefik network unable to connect when using domain names

sanjibukai · April 19, 2022, 4:07am

Hello everybody..

I have traefik (docker) running in my server using its own docker-compose file where I also have a named traefik network set as external for other container/services/app/etc.

Then I have many different applications running on that same server within their own docker-compose files.

I'm using traefik labels and everything is working great. Namely my services/app are correctly reverse proxied to the outside world over the internet.

However I just noticed that I'm unable to actually access one service from another. I never had that use-case but it seems it was always like that..

For example I have a gitea and a gitlab server, both running from their own docker-compose files.. And I'm trying to run a woodpecker CI server. But I noticed that the woodpecker server is unable to reach either the gitea or the gitlab server (using the domain name I have for them).

If I run the exact same woodpecker docker-compose file on another machine, it's working. So really this seems related to the fact they are all in the same machine.

So I wonder what should I tweak to have services within traefik to talk to each other using their domain names?

And I also wonder if this is more an OS like config to tweak (the server is running debian), Though I'm able to access any of those services using their domain names e.g. in the terminal, so I don't think it's related to the OS level.

Thank you very much for any detail or direction.

Note:

Here is my traefik docker-compose:

version: "3"

services:
  traefik:
    restart: always
    image: traefik:v2.5
    ports:
      - "443:443"
      - "80:80"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./traefik.yml:/etc/traefik/traefik.yml
      - ./acme.json:/acme.json

    labels:
      - "traefik.http.middlewares.auth.basicauth.users=__USERNAME__:__PASSWORD_HASH__"
      - "traefik.http.routers.api.middlewares=auth"
      - "traefik.http.routers.api.rule=Host(`__HOSTNAME__`)"
      - "traefik.http.routers.api.service=api@internal"
      - "traefik.http.routers.api.entrypoints=https"
      - "traefik.http.routers.api.tls=true"
      - "traefik.http.routers.api.tls.certresolver=traefik_resolver"

networks:
  default:
    external:
      name: traefik

Here is an example of my gitea docker-compose:

version: "3"

services:
  server:
    image: gitea/gitea
    container_name: gitea
    environment:
      - USER_UID=1000
      - USER_GID=1000
    restart: always
    volumes:
      - ./gitea:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    ports:
      - "222:22"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.gitea.rule=Host(`__GITEA_HOST__`)"
      - "traefik.http.routers.gitea.entrypoints=https"
      - "traefik.http.routers.gitea.tls=true"
      - "traefik.http.routers.gitea.tls.certresolver=traefik_resolver"
      - "traefik.http.services.gitea.loadbalancer.server.port=3000"

networks:
  default:
    external:
      name: traefik

And here is my woodpecker docker-compose:

version: '3'

services:
  woodpecker-server:
    image: woodpeckerci/woodpecker-server:latest
    volumes:
      - woodpecker-server-data:/var/lib/woodpecker/
    environment:
      - WOODPECKER_OPEN=true
      - WOODPECKER_HOST=__WP_HOST__
      - WOODPECKER_GITEA=true
      - WOODPECKER_GITEA_URL=__GITEA_HOST__
      - WOODPECKER_GITEA_CLIENT=__GITEA_CLIENT__
      - WOODPECKER_GITEA_SECRET=__GITEA_SECRET__
      - WOODPECKER_AGENT_SECRET=__AGENT_SECRET__
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.woodpecker.rule=Host(`__GITEA_HOST__`)"
      - "traefik.http.routers.woodpecker.entrypoints=https"
      - "traefik.http.routers.woodpecker.tls=true"
      - "traefik.http.routers.woodpecker.tls.certresolver=traefik_resolver"
      - "traefik.http.services.woodpecker.loadbalancer.server.port=8000"

  woodpecker-agent:
    image: woodpeckerci/woodpecker-agent:latest
    command: agent
    restart: always
    depends_on:
      - woodpecker-server
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - WOODPECKER_SERVER=woodpecker-server:9000
      - WOODPECKER_AGENT_SECRET=__AGENT_SECRET__
    labels:
      - "traefik.enable=false"

volumes:
  woodpecker-server-data:

networks:
  default:
    external:
      name: traefik

sanjibukai · April 23, 2022, 3:54am

Does anybody else encountered this problem?

jamesharding · April 24, 2022, 7:52pm

I am having the same issue (running on Ubuntu 20.04). Any luck solving this?

sanjibukai · April 25, 2022, 6:59pm

Hello,
I found this other post that is related Container to Container communication by host rule
And also that SO ssl - Docker Traefik can't resolve DNS (Fails reaching server and obtaining certificates) - Stack Overflow
And all possible solutions seem to be listed in this repo (I'm unable to find the original post where I saw this link) GitHub - flexguse/traefik-inter-container-routing: Demo setup for showing how Docker container communication with traefik routing can be used.

Though I didn't tested any of those solutions, I think any of those won't really help (unless I'm not understanding well)..
Let me explain..
For me having a solution that needs me, after each app/service I launch, to go back to the Traefik config and add some options (the container alias names method) remove the interest of having Traefik handling everything automatically.
The same goes for the Add host entry to container method, which needs me to launch the app/service and grab its IP and then go back to the config in order to add the IP.. Not very dynamic nor automatic IMHO..
The set DNS servermethod, I'm not sure it will work, since I don't have any problem to reach the internet (e.g. google.com or whatever domain) from the services..

So the problem I still have is that if I want one service to access another service with its FQDN (when both of my services are behinf traefik) it doesn't work.

I could have use the services name if all my services where within the same docker-compose file but they are not..

sanjibukai · September 11, 2022, 9:29pm

Hello everybody, does anyone still have no idea how to solve this?

danielritter · December 13, 2022, 3:42pm

I have the same issue. I have for each docker-compose.yml file a separate vm and one vm for a traefik instance which balances between the vm's and points to each other services/vm's. I cannot reach any service/vm by domain name on my root server or on the vm's with it's domain name. When I call the page outside from the internal network by domain name I can reach the service/vm.
Anybody a solution for that or knows what occures the problem? Is this a traefik or firewall issue? Many thanks!

bluepuma77 · December 13, 2022, 4:31pm

How should we tell you anything if you don’t show your Traefik static and dynamic config, even your docker-compose.yml? Traefik has about a 100 options you can set wrong

danielritter · December 15, 2022, 4:19pm

The files look like:

STATIC CONFIGURATION

log:
  level: INFO
global:
  checkNewVersion: true
  sendAnonymousUsage: false
api:
  insecure: true
  dashboard: true
  debug: true
entryPoints:
  web:
    address: :80
  webSecure:
    address: :443
providers:
  docker:
    endpoint: unix:///var/run/docker.sock
    exposedByDefault: false
  file:
    filename: traefik-dynamic.yml
    watch: true
certificatesResolvers:
  letsencript:
    acme:
      httpChallenge:
        entryPoint: web
          # caServer: https://acme-v02.api.letsencript.org/directory
      storage: /certificates/acme.json
      email: mail@domain.tdl
tls:
  options:
    default:
      sniStrict: true
      minVersion: VersionTLS12
      cipherSuites:
        - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

DYNAMIC CONFIGURATION

http:
  middlewares:
    https-redirect:
      redirectScheme:
        scheme: https
  routers:
    uptime-kuma-redirect:
      rule: Host(`uptime-kuma.domain.tdl`)
      service: uptime-kuma-service
      entryPoints:
        - web
      middlewares:
        - https-redirect
    uptime-kuma-router:
      rule: Host(`uptime-kuma.domain.tdl`)
      service: uptime-kuma-service
      entryPoints:
        - webSecure
      tls:
        certResolver: letsencript
        domains:
          - main: uptime-kuma.domain.tdl
        options: default
    nexus-redirect:
      rule: Host(`nexus.domain.tdl`)
      service: nexus-service
      entryPoints:
        - web
      middlewares:
        - https-redirect
    nexus-router:
      rule: Host(`nexus.domain.tdl`)
      service: nexus-service
      entryPoints:
        - webSecure
      tls:
        certResolver: letsencript
        domains:
          - main: nexus.domain.tdl
        options: default
  services:
    uptime-kuma-service:
      loadBalancer:
        servers:
          - url: http://10.0.20.30:3001
    nexus-service:
      loadBalancer:
        servers:
          - url: http://10.0.20.21:8081

docker-compoye.yml

version: "3.9"
services:
  traefik:
    image: traefik:v2.9.6
    container_name: traefik
    hostname: traefik
    restart: always
    env_file:
      - .env
    ports:
      - 80:80
      - 443:443
      - 8080:8080
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./traefik/traefik.yml:/traefik.yml:ro
      - ./traefik/traefik-dynamic.yml/:/traefik-dynamic.yml:ro
      - ./certificates/acme.json:/certificates/acme.json
      - ./security/users-credentials:/security/users-credentials:ro
networks:
  $NETWORK:
    external: true

.env

NETWORK=networkname

bluepuma77 · December 15, 2022, 7:51pm

You have dashboard insecure enabled (http://host:8080/dashboard/), what does it tell you? Did you look at the Traefik logs?

Are you sure your dynamic config is read without an absolute path in provider.file?

You can make your config a lot simpler by moving the http-redirect to entrypoints:

entryPoints:
  web:
    address: :80
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https

  websecure:
    address: :443

Furthermore you don't need domains: -name:, LE will take the domain from existing rule:Host().

Have you tried a curl or wget from within the Traefik container to target http://10.0.20.30:3001/?

ianhinder · May 6, 2023, 3:16pm

As far as I can see, you haven’t told docker compose to use the network you have defined in any of your services. You have only declared the network as available to use. So the services will be using the default per-project network, and will be unable to see services in other projects. In fact, I’m surprised that traefik can even connect to the other services. So you need to add a networks: key to each service. You also then need to tell traefik which network to use with a label traefik.docker.network. There is an example here:

If they are all on the same network, they should be able to see each other.

Topic		Replies	Views
Need Help with Traefik and DuckDNS Setup for remote access - Can’t Access Services Externally Traefik v2 docker	11	2463	January 22, 2024
Trying to communicate between services through traefik dns Traefik v3 (latest) docker , tcp	6	517	May 22, 2024
Docker Networking Traefik v1 docker	2	500	November 11, 2019
Cannot access services from local network Traefik v2 docker	6	3427	February 13, 2023
Question about container_name Traefik v2 docker	8	483	September 3, 2023

Containers within traefik network unable to connect when using domain names

STATIC CONFIGURATION

DYNAMIC CONFIGURATION

docker-compoye.yml

.env

Related topics