I'm setting up Traefik for a home server setup. Some of these containers are exposed externally and the others are not. I've hit an issue where, randomly it seems, some containers will result in 502 Bad Gateway: Connection Refused and on a more rare occasion 404.
I've had some luck with resolving the issue by bringing down docker, bringing down firewalld, flushing iptables, the bringing up firewalld, and then bringing up docker. The 404 seemed to be resolved after I did the same thing but also removed the docker network the containers were on. However, one container continues to throw a 502:
msg="'502 Bad Gateway' caused by: dial tcp 172.20.0.2:1980: connect: connection refused"
My Traefik docker compose:
version: '3'
services:
traefik:
image: traefik:latest
container_name: traefik
restart: unless-stopped
networks:
- traefik
ports:
- "80:80"
- "8080:8080"
- "443:443"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /media/containers/traefik/traefik-static.yaml:/traefik.yaml
- /media/containers/traefik/acme.json:/acme.json
- /media/containers/traefik/certs:/certs
labels:
- traefik.http.routers.dashboard.tls=true
- "traefik.http.routers.dashboard.rule=Host(`traefik.$DOMAIN`)"
- "traefik.http.routers.dashboard.service=api@internal"
networks:
traefik:
external: true
The docker compose of the affected container;
version: '3'
services:
pihole:
container_name: pihole
image: pihole/pihole:latest
ports:
- "53:53/tcp"
- "53:53/udp"
- "67:67/udp"
#- "80:80/tcp" # 80 conflicts with traefik
- "1980:80/tcp"
environment:
[...]
volumes:
[...]
cap_add:
- NET_ADMIN
networks:
- traefik
dns:
[...]
labels:
- traefik.http.routers.pihole.rule=Host(`pihole.$DOMAIN`)
- traefik.http.routers.pihole.tls=true
- traefik.http.services.pihole.loadbalancer.server.port=1980
restart: unless-stopped
networks:
traefik:
external: true
Below is a comparison docker compose for a similar internal container which works;
version: '3'
services:
radarr:
image: linuxserver/radarr
container_name: radarr
environment:
[...]
restart: unless-stopped
volumes:
[...]
networks:
- traefik
ports:
- 7878:7878
labels:
- traefik.http.routers.radarr.rule=Host(`radarr.$DOMAIN`)
- traefik.http.routers.radarr.tls=true
- traefik.http.services.radarr.loadbalancer.server.port=7878
Given the error is connection refused, I checked the container but it seems to have the IP address and port traefik wants;
❯ docker inspect pihole
[...]
"NetworkSettings": {
"Bridge": "",
"SandboxID": "<UUID>",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {
"53/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "53"
},
{
"HostIp": "::",
"HostPort": "53"
}
],
"53/udp": [
{
"HostIp": "0.0.0.0",
"HostPort": "53"
},
{
"HostIp": "::",
"HostPort": "53"
}
],
"67/udp": [
{
"HostIp": "0.0.0.0",
"HostPort": "67"
},
{
"HostIp": "::",
"HostPort": "67"
}
],
"80/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "1980"
},
{
"HostIp": "::",
"HostPort": "1980"
}
]
},
"SandboxKey": "/var/run/docker/netns/<UUID>",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"traefik": {
"IPAMConfig": null,
"Links": null,
"Aliases": [
"<UUID>",
"pihole"
],
"NetworkID": "<UUID>",
"EndpointID": "<UUID>",
"Gateway": "172.20.0.1",
"IPAddress": "172.20.0.2",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "XX:XX:XX:XX:XX",
"DriverOpts": null
}
}
}
Note The <UUID> bits above do not all have the same value.
The traefik docker network is below;
❯ docker network inspect traefik
[
{
"Name": "traefik",
"Id": "<UUID>",
"Created": "2022-05-19T02:55:50.041992949-04:00",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": {},
"Config": [
{
"Subnet": "172.20.0.0/16",
"Gateway": "172.20.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
[...]
"<RADARR UUID>": {
"Name": "radarr",
"EndpointID": "<<UUID>",
"MacAddress": "XX.XX.XX.XX.XX",
"IPv4Address": "172.20.0.5/16",
"IPv6Address": ""
},
[...]
"<TRAEFIK UUID>": {
"Name": "traefik",
"EndpointID": "<UUID>",
"MacAddress": "02:42:ac:14:00:09",
"IPv4Address": "172.20.0.9/16",
"IPv6Address": ""
},
[...]
"<PIHOLE UUID>": {
"Name": "pihole",
"EndpointID": "<UUID>",
"MacAddress": "YY.YY.YY.YY",
"IPv4Address": "172.20.0.2/16",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {}
}
]
I can confirm I can reach that container from both the host and a remote system;
❯ curl 127.0.0.1:1980 # from host
<!doctype html>
<html lang='en'>
<head>
<meta charset='utf-8'>
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>● 127.0.0.1</title>
<link rel='stylesheet' href='/pihole/blockingpage.css'>
<link rel='shortcut icon' href='/admin/img/favicons/favicon.ico' type='image/x-icon'>
</head>
<body id='splashpage'>
<div id="pihole_card">
<img src='/admin/img/logo.svg' alt='Pi-hole logo' id="pihole_logo_splash" />
<p>Pi-<strong>hole</strong>: Your black hole for Internet advertisements</p>
<a href='/admin'>Did you mean to go to the admin panel?</a>
</div>
</body>
</html>
❯ curl 192.168.1.2:1980 # from a system elsewhere on my network
<!doctype html>
<html lang='en'>
<head>
<meta charset='utf-8'>
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>● 192.168.1.2</title>
<link rel='stylesheet' href='/pihole/blockingpage.css'>
<link rel='shortcut icon' href='/admin/img/favicons/favicon.ico' type='image/x-icon'>
</head>
<body id='splashpage'>
<div id="pihole_card">
<img src='/admin/img/logo.svg' alt='Pi-hole logo' id="pihole_logo_splash" />
<p>Pi-<strong>hole</strong>: Your black hole for Internet advertisements</p>
<a href='/admin'>Did you mean to go to the admin panel?</a>
</div>
</body>
</html>
A lot of what I've found in searching is this issue can arise if there's firewall issues, if the containers are on different docker networks, the the ports are incorrect, etc. Unless I am blind (and it is quite late as I type this so I may very well be at least hard of seeing), I am not sure I am seeing anything that points to the issue; the containers are all on the same network, the ports are open, etc. I even tried - traefik.docker.network=traefik with no luck.