Random containers give Bad Gateway

I'm setting up Traefik for a home server setup. Some of these containers are exposed externally and the others are not. I've hit an issue where, randomly it seems, some containers will result in 502 Bad Gateway: Connection Refused and on a more rare occasion 404.

I've had some luck with resolving the issue by bringing down docker, bringing down firewalld, flushing iptables, the bringing up firewalld, and then bringing up docker. The 404 seemed to be resolved after I did the same thing but also removed the docker network the containers were on. However, one container continues to throw a 502:

msg="'502 Bad Gateway' caused by: dial tcp 172.20.0.2:1980: connect: connection refused"

My Traefik docker compose:

version: '3'

services:
    traefik:
        image: traefik:latest
        container_name: traefik
        restart: unless-stopped
        networks:
            - traefik
        ports:
            - "80:80"
            - "8080:8080"
            - "443:443"

        volumes:
            - /var/run/docker.sock:/var/run/docker.sock
            - /media/containers/traefik/traefik-static.yaml:/traefik.yaml
            - /media/containers/traefik/acme.json:/acme.json
            - /media/containers/traefik/certs:/certs

        labels: 
            - traefik.http.routers.dashboard.tls=true
            - "traefik.http.routers.dashboard.rule=Host(`traefik.$DOMAIN`)"
            - "traefik.http.routers.dashboard.service=api@internal"


networks:
    traefik:
        external: true

The docker compose of the affected container;

version: '3'
services:
    pihole:
        container_name: pihole
        image: pihole/pihole:latest
        ports:
            - "53:53/tcp"
            - "53:53/udp"
            - "67:67/udp"
            #- "80:80/tcp"  # 80 conflicts with traefik
            - "1980:80/tcp"
        environment:
           [...]
        volumes:
           [...]
        cap_add:
            - NET_ADMIN
        networks:
            - traefik
        dns:
           [...]
        labels:
            - traefik.http.routers.pihole.rule=Host(`pihole.$DOMAIN`)
            - traefik.http.routers.pihole.tls=true
            - traefik.http.services.pihole.loadbalancer.server.port=1980
        restart: unless-stopped

networks:
    traefik:
        external: true 

Below is a comparison docker compose for a similar internal container which works;

version: '3'
services:
    radarr:
        image: linuxserver/radarr
        container_name: radarr
        environment:
            [...]
        restart: unless-stopped
        volumes:
            [...]
        networks:
            - traefik
        ports:
            - 7878:7878
        labels:
            - traefik.http.routers.radarr.rule=Host(`radarr.$DOMAIN`)
            - traefik.http.routers.radarr.tls=true
            - traefik.http.services.radarr.loadbalancer.server.port=7878

Given the error is connection refused, I checked the container but it seems to have the IP address and port traefik wants;

❯ docker inspect pihole
[...]
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "<UUID>",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "53/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "53"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "53"
                    }
                ],
                "53/udp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "53"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "53"
                    }
                ],
                "67/udp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "67"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "67"
                    }
                ],
                "80/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "1980"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "1980"
                    }
                ]
            },
            "SandboxKey": "/var/run/docker/netns/<UUID>",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "traefik": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "<UUID>",
                        "pihole"
                    ],
                    "NetworkID": "<UUID>",
                    "EndpointID": "<UUID>",
                    "Gateway": "172.20.0.1",
                    "IPAddress": "172.20.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "XX:XX:XX:XX:XX",
                    "DriverOpts": null
                }
            }
        }

Note The <UUID> bits above do not all have the same value.

The traefik docker network is below;

❯ docker network inspect traefik
[
    {
        "Name": "traefik",
        "Id": "<UUID>",
        "Created": "2022-05-19T02:55:50.041992949-04:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.20.0.0/16",
                    "Gateway": "172.20.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
             [...]
            "<RADARR UUID>": {
                "Name": "radarr",
                "EndpointID": "<<UUID>",
                "MacAddress": "XX.XX.XX.XX.XX",
                "IPv4Address": "172.20.0.5/16",
                "IPv6Address": ""
            },
                 [...]
            "<TRAEFIK UUID>": {
                "Name": "traefik",
                "EndpointID": "<UUID>",
                "MacAddress": "02:42:ac:14:00:09",
                "IPv4Address": "172.20.0.9/16",
                "IPv6Address": ""
            },
                [...]
            "<PIHOLE UUID>": {
                "Name": "pihole",
                "EndpointID": "<UUID>",
                "MacAddress": "YY.YY.YY.YY",
                "IPv4Address": "172.20.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]

I can confirm I can reach that container from both the host and a remote system;

❯ curl 127.0.0.1:1980    # from host
    <!doctype html>
    <html lang='en'>
        <head>
            <meta charset='utf-8'>
            <meta name="viewport" content="width=device-width, initial-scale=1">
            <title>● 127.0.0.1</title>
            <link rel='stylesheet' href='/pihole/blockingpage.css'>
            <link rel='shortcut icon' href='/admin/img/favicons/favicon.ico' type='image/x-icon'>
        </head>
        <body id='splashpage'>
            <div id="pihole_card">
              <img src='/admin/img/logo.svg' alt='Pi-hole logo' id="pihole_logo_splash" />
              <p>Pi-<strong>hole</strong>: Your black hole for Internet advertisements</p>
              <a href='/admin'>Did you mean to go to the admin panel?</a>
            </div>
        </body>
    </html>
❯ curl 192.168.1.2:1980    # from a system elsewhere on my network
    <!doctype html>
    <html lang='en'>
        <head>
            <meta charset='utf-8'>
            <meta name="viewport" content="width=device-width, initial-scale=1">
            <title>● 192.168.1.2</title>
            <link rel='stylesheet' href='/pihole/blockingpage.css'>
            <link rel='shortcut icon' href='/admin/img/favicons/favicon.ico' type='image/x-icon'>
        </head>
        <body id='splashpage'>
            <div id="pihole_card">
              <img src='/admin/img/logo.svg' alt='Pi-hole logo' id="pihole_logo_splash" />
              <p>Pi-<strong>hole</strong>: Your black hole for Internet advertisements</p>
              <a href='/admin'>Did you mean to go to the admin panel?</a>
            </div>
        </body>
    </html>

A lot of what I've found in searching is this issue can arise if there's firewall issues, if the containers are on different docker networks, the the ports are incorrect, etc. Unless I am blind (and it is quite late as I type this so I may very well be at least hard of seeing), I am not sure I am seeing anything that points to the issue; the containers are all on the same network, the ports are open, etc. I even tried - traefik.docker.network=traefik with no luck.

Did you ever figure this out? I'm having the same problem with Traefik and pi-hole.