Traefik returning 502 Bad Gateway cause EOF when accessing HTTPS routes with docker network

Traefik Traefik v2
1

Description:

I'm encountering an issue with Traefik where it returns a 502 Bad Gateway error caused by EOF when trying to access HTTPS routes via my domain (https://my-domain.com) or any path prefixes configured through Traefik labels. The issue persists even when I try to ping the containers' IPs directly.

Here's an excerpt from the Traefik logs showing the error:

time="2024-03-25T22:17:39Z" level=debug msg="'502 Bad Gateway' caused by: EOF"70.93.248.216 - - [25/Mar/2024:22:17:39 +0000] "GET / HTTP/2.0" 502 11 "-" "-" 2"webapp@docker" "http://172.18.0.3:3001" 2ms

Here are some additional details about my setup:

  • I have four containers running on a custom Docker internal network called "internal-net": traefik, webapp, chatbot, and flask_api.
  • I've verified that all services are running properly within their containers and that they are all running only on the network "internal-net". But I'm still having networking issues.
  • I'm using Docker networks to facilitate container-to-container communication without having to manually change host IPs for each container. Thus, I have configured my code to make container-to-container requests using https://<container-name>:<port> as the host DNS.
  • My setup is hosted on Google Compute Engine. I have configured and tested that my custom DNS is working on this server.
  • I've tried the following with no success:
    ** expose each container's ports on docker run using -p flag
    ** tell traefik which port to go to for each service by labeling the image with traefik.http.services.<container-name>.loadbalancer.server.port=<port>
    ** running one service container with traefik at a time

Could someone please help me diagnose and resolve this issue? I suspect there might be a configuration problem either with Traefik or Docker networking.

Code Snippets:

webapp.dockerfile

FROM <base docker image>

EXPOSE 3001

# Install node and npm
RUN curl -fsSL https://deb.nodesource.com/setup_current.x | bash - \
    && apt-get install -y nodejs
# Go to repo
RUN mkdir -p /home/dreamdai/webapp-react/web_service/node_modules
WORKDIR /home/dreamdai/webapp-react/web_service

# Install npm packages
COPY ./webapp-react/web_service/package*.json .
RUN npm install

# Copy codebase
COPY . /home/dreamdai

### Configure traefik ###
# Allows Traefik to discover containers
LABEL traefik.enable=true
# Sends traffic through https
LABEL traefik.http.routers.webapp.entrypoints=websecure
# Allows Traefik to route requests from <my-dns>.com (host static IP accessible by http)
LABEL traefik.http.routers.webapp.rule=Host(`<my-dns>.com`)
# Tell it to use the internal network
LABEL traefik.docker.network=internal-net
# Enables HTTPS
LABEL traefik.http.routers.webapp.tls=true
# Sets certificate issuer as LetsEncrypt
LABEL traefik.http.routers.webapp.tls.certresolver=letsencrypt

# Run the programs
CMD npm start

chatbot.dockerfile

FROM <base docker image>

EXPOSE 8501

# Copy repo and install reqs
WORKDIR /home/dreamdai
COPY . /home/dreamdai

### Configure traefik ###
# Allows Traefik to discover containers
LABEL traefik.enable=true
# Tell it to use the internal network
LABEL traefik.docker.network=internal-net
# Sends traffic through https
LABEL traefik.http.routers.chatbot.entrypoints=websecure
# Allows Traefik to route requests from host/chat
LABEL traefik.http.routers.chatbot.rule="Host(`<my-dns>.com`) && PathPrefix(`/chat`)"
# Enables HTTPS
LABEL traefik.http.routers.chatbot.tls=true
# Sets certificate issuer as LetsEncrypt
LABEL traefik.http.routers.chatbot.tls.certresolver=letsencrypt


# Run the programs
CMD python3 chatbot_runner.py -e prod

flask_api.dockerfile

NOTE: The flask_api is meant to be accessible by https container-to-container traffic only. I'm not yet sure how to address the Host rule because once it complained that I had .com on both this container label and the webapp container label.

FROM <base docker image>

EXPOSE 4000

# Go to repo
WORKDIR /home/dreamdai
COPY . /home/dreamdai

### Configure traefik ###
# Allows Traefik to discover containers
LABEL traefik.enable=true
# Sends traffic through https
LABEL traefik.http.routers.flask_api.entrypoints=websecure
# Allows Traefik to route requests from host (host static IP accessible by http)
LABEL traefik.http.routers.flask_api.rule=Host(`flask_api`) # THIS ISN'T WORKING YET WITH TLS
# Tell it to use the internal network
LABEL traefik.docker.network=internal-net
# Enables HTTPS
LABEL traefik.http.routers.flask_api.tls=true
# Sets certificate issuer as LetsEncrypt
LABEL traefik.http.routers.flask_api.tls.certresolver=letsencrypt

# Run the programs
CMD gunicorn --bind 0.0.0.0:4000 --env "ENV=prod" "flask_api:create_app()"

traefik.dockerfile

FROM traefik:v2.10

EXPOSE 80 443
## --- Comment out the following lines for dev environment (default prod) ---
COPY ./config/traefik.prod.toml /etc/traefik/traefik.toml
# Creates a directory for certificates that Traefik can access, puts an empty json in it,
# and configures permissions to be restrictive enough that LetsEncrypt doesn't complain
RUN mkdir /certificates && touch /certificates/acme.json && echo "{}" > /certificates/acme.json && chmod 600 /certificates/acme.json

# Allows Traefik to discover other containers
LABEL traefik.enable=true
LABEL traefik.docker.network=internal-net
# Define host so it can be accessed at host/traefik-dashboard
LABEL traefik.http.routers.dashboard.rule="Host(`<my-dns>.com`) && PathPrefix(`/traefik-dashboard`)"
# Enables HTTPS
LABEL traefik.http.routers.dashboard.tls=true
# Sends traffic through https
LABEL traefik.http.routers.dashboard.entrypoints=websecure
# Sets certificate resolver as LetsEncrypt
LABEL traefik.http.routers.dashboard.tls.certresolver=letsencrypt
# Enables middleware features for dashboard
LABEL traefik.http.routers.dashboard.service=api@internal

traefik.prod.toml

[entryPoints]
  [entryPoints.web]
    address = ":80"
  [entryPoints.web.http]
    [entryPoints.web.http.redirections]
      [entryPoints.web.http.redirections.entryPoint]
        to = "websecure"
        scheme = "https"

  [entryPoints.websecure]
    address = ":443"
    
[accessLog]
[log]
level = "DEBUG"

[api]
dashboard = true

[providers]
  [providers.docker]
    exposedByDefault = false
    network = "internal-net"

[certificatesResolvers.letsencrypt.acme]
  email = "example@gmail.com"
  storage = "/certificates/acme.json"
  [certificatesResolvers.letsencrypt.acme.httpChallenge]
    entryPoint = "web"

Docker run commands via VM startup script

sudo docker network create internal-net
sudo docker run --name flask_api --rm -d -v /secrets:/secrets:ro --network internal-net flask-api:release-prod
sudo docker run --name webapp --rm -d -v /secrets:/secrets:ro --network internal-net webapp:release-prod
sudo docker run --name chatbot --rm -d -v /secrets:/secrets:ro --network internal-net chatbot:release-prod
sudo docker run --name traefik --rm -d -v /var/run/docker.sock:/var/run/docker.sock:ro -v /certificates:/certificates -p "80:80" -p "443:443" --network internal-net traefik:release-prod

Note: redacted custom DNS and docker image names for anonymity.

Any help is greatly appreciated as I've exhausted my research and trial-and-error paths.

2

Try to run a simple Traefik example with docker compose up. See if it works in your infrastructure.

Get rid of your love to pure docker run CLI commands, I made the switch only a year ago :wink: :clap: Use one or multiple docker compose.yml files.

I would recommend to not bake the config and labels into the Dockerfile, but place them in docker-compose.yml file, that’s more consistent when you want to spin up same service with different URL for testing etc, I think it’s more best practice.

Personally I always add the port to the labels, to make sure Traefik uses the right one in case the image exposes multiple.

Another personal thing: use yaml over toml, it’s so much more readable and less writing. Or move to pure inline config, see example on top.

3

Which method do you use to add the port to the labels?
traefik.port= or traefik.http.services.my-container-service.loadbalancer.server.port=

4

Again, see traefik-best-practice/docker-traefik-dashboard-letsencrypt at main · bluepuma77/traefik-best-practice · GitHub

And for tcp use the according labels.