Hello all,
My question is in the subject, but let me explain why I'm asking.
I'm trying to set up multiple docker compose application stacks, with a single traefik container forwarding HTTP requests to the relevant stacks.
Each docker compose stack creates its own network, as does traefik. Hence I had to make the 'frontend' application in each stack sit on two networks: the one that traefik sits on, and the backend network shared with the other containers in that stack. Fine so far.
However, what I observed when I tried this out, was that traefik would attempt to forward incoming requests to the "wrong" IP address on the frontend container - the one which belongs to the backend network - and this traffic was being blackholed. I was able to demonstrate this directly using tcpdump on the traefik network; I could see the incoming request fine, and then the attempted proxied connections with the wrong destination IP address (just TCP SYN packets). Furthermore, looking in the Traefik API web UI, I could also see the service target address http://x.x.x.x where x.x.x.x was on the backend network.
Unfortunately, this appears to be inconsistent. When tearing this down and recreating it from scratch, sometimes the problem goes away, and sometimes it recurs. Hence I really want to understand how traefik (or docker) choses the address on such a container to proxy to.
Let me make things more concrete. I'm using Ubuntu 22.04 with docker-ce 5:20.10.22~3-0~ubuntu-jammy with docker-compose-plugin 2.14.1~ubuntu-jammy, i.e. "docker compose" instead of "docker-compose".
Here's how I deployed traefik:
==> traefik/docker-compose.yml <==
version: '3'
services:
reverse-proxy:
image: traefik:v2.5
ports:
- "80:80"
- "443:443"
- "8080:8080"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ./:/etc/traefik
networks:
default:
name: aaa_traefik_proxy
==> traefik/traefik.yml <==
## traefik.yml
# Docker configuration backend
providers:
docker:
defaultRule: "Host(`{{ trimPrefix `/` .Name }}.docker.localhost`)"
exposedByDefault: false
# API and dashboard configuration
api:
insecure: true
And here's a minimal example of the "whoami" application (although the real application where I was experiencing the problem was netbox)
==> project/docker-compose.yml <==
version: '3'
services:
whoami:
image: "traefik/whoami"
container_name: "simple-service"
labels:
- "traefik.enable=true"
- "traefik.http.routers.whoami.rule=Host(`whoami.localhost`)"
- "traefik.http.routers.whoami.entrypoints=http"
networks:
default:
traefik:
priority: 100
networks:
traefik:
name: aaa_traefik_proxy
When the problem was happening, I could do
curl -H host:whoami.localhost localhost
and it would hang for 30 seconds before giving up with "Gateway Timeout". As I described before, finding the traefik bridge interface and running tcpdump on it, the SYN packets were going to the simple-service container on its 'default' network IP address rather than the 'traefik' network IP address.
Aside: although the network name was originally "traefik_proxy", I changed it to "aaa_traefik_proxy" because Docker creates network interfaces on a container in alphabetical order (in principle "priority: N" should override that, but it's safest to fix the name as well). Having said that, I don't actually care which interface is eth0 and which is eth1. What I care about is how traefik picks which interface IP address to forward traffic to.
I've read the documentation for the traefik docker provider. It talks about how the host itself is resolved (host.docker.internal), but I can't see how the service IP address is chosen for a container which has traefik.enable on it. Does it just ask docker's DNS to resolve the container name? If so, the question then becomes "how does docker choose which IP address(es) to return for this query, when there are multiple networks?" Does it return both, as it round-robin DNS? (Combined with caching, this would make the results unpredictable)
At one point I tried changing the network name back to "traefik_proxy", leaving the "priority: 100" in place, and the problem recurs.
$ time curl -H host:whoami.localhost localhost
Gateway Timeout
real 0m30.018s
user 0m0.006s
sys 0m0.014s
$
This does seem to make it more likely to fail, and so if anyone wants to try reproducing this independently, I recommend this is what you do. However if I keep restarting the "simple-service" stack like this, sometimes it does work. So I don't think it's guaranteed that the IP address of the lexically ealiest interface is always chosen.
Switching back to aaa_traefik_proxy it seems to reproduce less often, but sometimes it does. (One in ten perhaps?)
I also tried the netbox stack again, and it seems to be easier to reproduce. Here is my docker-compose.override.yml:
version: '3.4'
services:
netbox:
#ports:
# - 8001:8080
labels:
- "traefik.enable=true"
- "traefik.http.services.netbox.loadbalancer.server.port=8080"
- "traefik.http.routers.netbox.entrypoints=http"
#- "traefik.http.routers.netbox.rule=HostRegexp(`netbox[-.].*`)"
- "traefik.http.routers.netbox.rule=Host(`netbox.example.com`)"
networks:
traefik:
default:
networks:
default:
#enable_ipv6: true
traefik:
name: aaa_traefik_proxy
(the remainder of the stack is the official git repo)
This reproduces even with network name "aaa_traefik_proxy". Here's tcpdump on the traefik network when it does.
20:05:06.222518 IP 172.27.0.1.46942 > 172.27.0.2.80: Flags [S], seq 229048584, win 64240, options [mss 1460,sackOK,TS val 1172738899 ecr 0,nop,wscale 7], length 0
20:05:06.222547 IP 172.27.0.2.80 > 172.27.0.1.46942: Flags [S.], seq 3369705287, ack 229048585, win 65160, options [mss 1460,sackOK,TS val 3772116655 ecr 1172738899,nop,wscale 7], length 0
20:05:06.222573 IP 172.27.0.1.46942 > 172.27.0.2.80: Flags [.], ack 1, win 502, options [nop,nop,TS val 1172738899 ecr 3772116655], length 0
20:05:06.222721 IP 172.27.0.1.46942 > 172.27.0.2.80: Flags [P.], seq 1:82, ack 1, win 502, options [nop,nop,TS val 1172738900 ecr 3772116655], length 81: HTTP: GET / HTTP/1.1
20:05:06.222744 IP 172.27.0.2.80 > 172.27.0.1.46942: Flags [.], ack 82, win 509, options [nop,nop,TS val 3772116656 ecr 1172738900], length 0
20:05:06.223065 IP 172.27.0.2.57504 > 172.31.0.7.8080: Flags [S], seq 1803610685, win 64240, options [mss 1460,sackOK,TS val 2116218632 ecr 0,nop,wscale 7], length 0
20:05:07.237886 IP 172.27.0.2.57504 > 172.31.0.7.8080: Flags [S], seq 1803610685, win 64240, options [mss 1460,sackOK,TS val 2116219647 ecr 0,nop,wscale 7], length 0
20:05:09.253886 IP 172.27.0.2.57504 > 172.31.0.7.8080: Flags [S], seq 1803610685, win 64240, options [mss 1460,sackOK,TS val 2116221663 ecr 0,nop,wscale 7], length 0
(note how traefik is on the 172.27 network, but it is trying to connect outbound to the 172.31 network)
I can keep recreating the netbox stack with docker compose down; docker compose up -d. Sometimes it works, and sometimes I see the problem described here. It reproduces maybe half of the time?
When it's working:
$ curl -fsS localhost:8080/api/http/services | python3 -mjson.tool | grep 172
"url": "http://172.27.0.4:8080"
"http://172.27.0.4:8080": "UP"
"url": "http://172.27.0.3:80"
"http://172.27.0.3:80": "UP"
When it's not working:
$ curl -fsS localhost:8080/api/http/services | python3 -mjson.tool | grep 172
"url": "http://172.22.0.7:8080"
"http://172.22.0.7:8080": "UP"
"url": "http://172.27.0.3:80"
"http://172.27.0.3:80": "UP"
So my conclusion is: it's non-deterministic how the target IP address is chosen. If I'm doing something wrong, and there's a way to make it deterministic, I would be very grateful for any clues! 
I'm sure I can't be the only one to try this.
And if you've read through all of this, many thanks for taking the trouble to do so.
Regards,
Brian.