Gateway Timeout when using different port mapping

Traefik Traefik v2 (latest)
1

I am unable to reach one docker container using Traefik:

The config of the docker container that is unreachable is:

version: "3"

# More info at https://github.com/pi-hole/docker-pi-hole/ and https://docs.pi-hole.net/
services:
  pihole:
    container_name: pihole
    image: pihole/pihole:latest
    ports:
      - "53:53/tcp"
      - "53:53/udp"
      - "67:67/udp"
      - "8153:80/tcp"
    environment:
      TZ: 'Europe/Amsterdam'
      VIRTUAL_HOST: pi-hole.docker-core.domain.com
      # WEBPASSWORD: 'set a secure password here or it will be random'
    # Volumes store your data between container upgrades
    volumes:
       - '/opt/pi-hole/etc-pihole/:/etc/pihole/'
       - '/opt/pi-hole/etc-dnsmasq.d/:/etc/dnsmasq.d/'
    dns:
      - 127.0.0.1
      - 8.8.8.8
    # Recommended but not required (DHCP needs NET_ADMIN)
    #   https://github.com/pi-hole/docker-pi-hole#note-on-capabilities
    cap_add:
      - NET_ADMIN
    restart: unless-stopped
    labels:
      - traefik.enable=true
      - traefik.http.routers.pi-hole.rule=Host(`pi-hole.docker-core.domain.com`)
      - traefik.http.services.pi-hole.loadbalancer.server.port=80

This is my Traefik config:

version: '3'

services:
  reverse-proxy:
    # The official v2.0 Traefik docker image
    image: traefik:v2.0
    # Enables the web UI and tells Traefik to listen to docker
    command: --api.insecure=true --providers.docker --providers.docker.defaultRule="Host(`{{ trimPrefix `/` .Name }}.docker-core.domain.com`)"
    ports:
      # The HTTP port
      - "80:80"
      # The Web UI (enabled by --api.insecure=true)
      - "8080:8080"
    volumes:
      # So that Traefik can listen to the Docker events
      - /var/run/docker.sock:/var/run/docker.sock

Whenever I visit pi-hole.docker-core.domain.com I get: Gateway Timeout

2

Looks like you specified wrong port. There is a mismatch: the ports section says that your container is listening on port 80, yet in the label section you ask traefik to connect to 81.

1 Like
3

Hello Zespri,

Even when setting it to port 80 I get a gateway timeout.

I changed my config to reflect this.

4

This error means that the traefik cannot connect to where ever you asked it to connect. You need to troubleshoot that.

In particular I'd recommend enabling debug logs, trying to connecting to the expected endpoint externally and also trying to connect to that exact port / host from with traefik container (e.g. with curl).

5

From the host itself I can reach the container:

[root@docker-core traefik]#  curl 'http://172.18.0.2:80'

    <html><head>
        <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1"/>
        <link rel='stylesheet' href='/pihole/blockingpage.css' type='text/css'/>
    </head><body id='splashpage'><img src='/admin/img/logo.svg'/><br/>Pi-<b>hole</b>: Your black hole for Internet advertisements<br><a href='/admin'>Did you mean to go to the admin panel?</a></body></html>

It is also reachable from outside the host by using this command:

[root@docker-core pi-hole]# curl --header 'Host:pi-hole.docker-core.corp.pootenterprise.com' 'http://localhost:8053/'

    <html><head>
        <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1"/>
        <link rel='stylesheet' href='/pihole/blockingpage.css' type='text/css'/>
    </head><body id='splashpage'><img src='/admin/img/logo.svg'/><br/>Pi-<b>hole</b>: Your black hole for Internet advertisements<br><a href='/admin'>Did you mean to go to the admin panel?</a></body></html>

It seems I cannot ping the pi-hole container from inside the traefik container:

/ # ping 172.18.0.2
PING 172.18.0.2 (172.18.0.2): 56 data bytes

I can ping the portainer container from inside the traefik container

/ # ping 172.17.0.2
PING 172.17.0.2 (172.17.0.2): 56 data bytes
64 bytes from 172.17.0.2: seq=0 ttl=63 time=0.225 ms
64 bytes from 172.17.0.2: seq=1 ttl=63 time=0.200 ms

How do I enable debug logging?

6

This may not be true in your case but in general with containers, I would not rely on ping at all. Ping uses ICMP low level protocol which has nothing to do with HTTP (I think it's not even TCP). You can have a successful ping but not HTTP server on an address and you can have an HTTP server that would not respond to ping. Also with containers, often a virtual network layer is used, for example in kubernetes pinging certain addresses that you can connect over HTTP is not possible in principle since they are not real IP address and traffic for them are not routed, in the sense that the IP address never appears in an IP packet header.. When you connect to them via HTTP though the virtual IP address get transparently substituted for a corresponding real one. So in general I would not trust a lot to ping as a diagnostic tool when you are trying to troubleshoot containerized web applications.

7

Thanks again Zespri,

This is what I am getting with the debug:

192.168.2.17 - - [01/Oct/2019:20:19:57 +0000] "GET / HTTP/1.1" 504 15 "-" "-" 16 "pi-hole@docker" "http://172.18.0.2:80" 30000ms


time="2019-10-01T20:20:27Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/favicon.ico\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"image/webp,image/apng,image/*,*/*;q=0.8\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Accept-Language\":[\"en-GB,en-US;q=0.9,en;q=0.8\"],\"Connection\":[\"keep-alive\"],\"Cookie\":[\"PHPSESSID=oe79f4ogmbckrouv047j5h1437\"],\"Referer\":[\"http://pi-hole.docker-core.example.com/\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36\"],\"X-Forwarded-Host\":[\"pi-hole.docker-core.example.com\"],\"X-Forwarded-Port\":[\"80\"],\"X-Forwarded-Proto\":[\"http\"],\"X-Forwarded-Server\":[\"571d9ecbae91\"],\"X-Real-Ip\":[\"192.168.2.17\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"pi-hole.docker-core.example.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"192.168.2.17:1984\",\"RequestURI\":\"/favicon.ico\",\"TLS\":null}"


time="2019-10-01T20:20:27Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/favicon.ico\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"image/webp,image/apng,image/*,*/*;q=0.8\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Accept-Language\":[\"en-GB,en-US;q=0.9,en;q=0.8\"],\"Connection\":[\"keep-alive\"],\"Cookie\":[\"PHPSESSID=oe79f4ogmbckrouv047j5h1437\"],\"Referer\":[\"http://pi-hole.docker-core.example.com/\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36\"],\"X-Forwarded-Host\":[\"pi-hole.docker-core.example.com\"],\"X-Forwarded-Port\":[\"80\"],\"X-Forwarded-Proto\":[\"http\"],\"X-Forwarded-Server\":[\"571d9ecbae91\"],\"X-Real-Ip\":[\"192.168.2.17\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"pi-hole.docker-core.example.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"192.168.2.17:1984\",\"RequestURI\":\"/favicon.ico\",\"TLS\":null}" ForwardURL="http://172.18.0.2:80"

How can I try to make a connection from inside the traefik container? There is no curl installed.

8

Try installing it, may be? apk add -U curl I do not remember the right syntax off hand but I did that before :wink:

9

I figured out why it wasn't reachable, both containers were in different stacks (networks). By placing them both under the same stack it worked.

1 Like