Conventions:
- Client: The host trying to access whoami service through traefik
- Client IP: 10.0.0.80
- Server: The host running traefik and whoami services
- Server IP: 10.0.0.15
While following the initial Getting Started tutorial, I created a simple docker-compose.yml containing an almost identical configuration file except for the Host router rule of the whoami service, that has been replaced with whoami.tld, representing a custom domain to be used locally:
version: '3'
services:
reverse-proxy:
# The official v2 Traefik docker image
image: traefik:v2.10
# Enables the web UI and tells Traefik to listen to docker
command: --api.insecure=true --providers.docker
ports:
# The HTTP port
- "80:80"
# The Web UI (enabled by --api.insecure=true)
- "8080:8080"
volumes:
# So that Traefik can listen to the Docker events
- /var/run/docker.sock:/var/run/docker.sock
whoami:
# A container that exposes an API to show its IP address
image: traefik/whoami
labels:
- "traefik.http.routers.whoami.rule=Host(`whoami.tld`)"
In theory this service should be accessible for any Clients in the same network as the Server by the curl command using the Host header, but in practice it is only possible to access the whoami service running the curl command from within the Server itself.
curl -H Host:whoami.tld http://10.0.0.15/
The above command returns 301 Moved Permanently for every Client (and opens the traefik dashboard if accessed from the browser), while in the Server it returns the expected output containing the data of the whoami service.
I tried creating an exclusive network for traefik in docker but no different results. Also performed a packet capture with tcpdump to trace the HTTP request and the results was different for each one of the hosts.
# Format:
# network_interface src > dst
# Packet flow when running curl from the *Server*
br-e31f1e726d9e 10.0.0.15.39076 > 172.29.0.3.80
veth651aab0 10.0.0.15.39076 > 172.29.0.3.80
veth651aab0 172.29.0.3.58268 > 172.29.0.2.80
vethba1f5e8 172.29.0.3.58268 > 172.29.0.2.80
vethba1f5e8 172.29.0.2.80 > 172.29.0.3.58268
veth651aab0 172.29.0.2.80 > 172.29.0.3.58268
veth651aab0 172.29.0.3.80 > 10.0.0.15.39076
br-e31f1e726d9e 172.29.0.3.80 > 10.0.0.15.39076
--
# Packet flow when running curl from the *Client*
eth0 10.0.0.80.45088 > 10.0.0.15.80
br-e31f1e726d9e 172.29.0.1.54682 > 172.29.0.3.8080
veth651aab0 172.29.0.1.54682 > 172.29.0.3.8080
veth651aab0 172.29.0.3.8080 > 172.29.0.1.54682
br-e31f1e726d9e 172.29.0.3.8080 > 172.29.0.1.54682
eth0 10.0.0.15.80 > 10.0.0.80.45088
Apparently the package is being directly redirected to port 8080 when the request comes from the eth0 network interface.
iptables rules (just to make sure if this could be the reason of this weird behavior)
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:80
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:80
Chain FORWARD (policy DROP)
target prot opt source destination
DOCKER-USER all -- 0.0.0.0/0 0.0.0.0/0
DOCKER-ISOLATION-STAGE-1 all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
DOCKER all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
DOCKER all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
DOCKER all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
DOCKER all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain DOCKER (4 references)
target prot opt source destination
ACCEPT tcp -- 0.0.0.0/0 172.25.0.2 tcp dpt:80
ACCEPT tcp -- 0.0.0.0/0 172.25.0.3 tcp dpt:8080
ACCEPT tcp -- 0.0.0.0/0 172.25.0.3 tcp dpt:80
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target prot opt source destination
DOCKER-ISOLATION-STAGE-2 all -- 0.0.0.0/0 0.0.0.0/0
DOCKER-ISOLATION-STAGE-2 all -- 0.0.0.0/0 0.0.0.0/0
DOCKER-ISOLATION-STAGE-2 all -- 0.0.0.0/0 0.0.0.0/0
RETURN all -- 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-ISOLATION-STAGE-2 (3 references)
target prot opt source destination
DROP all -- 0.0.0.0/0 0.0.0.0/0
DROP all -- 0.0.0.0/0 0.0.0.0/0
DROP all -- 0.0.0.0/0 0.0.0.0/0
RETURN all -- 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-USER (1 references)
target prot opt source destination
RETURN all -- 0.0.0.0/0 0.0.0.0/0
Also tried using an invalid Host header, and when running the curl command from a Client, the response is always the 301 Moved Permanently while running from the Server, the response is 404 page not found, which makes me believe that any requests are being redirected to the dashboard regardless of the Host header.
I spent over 30h trying to debug this, but could not figure why this is happening, if anyone could help with some suggestion I would really appreciate!