Bad Gateway 502 with docker containers

Traefik Traefik v2 (latest)
1

Been using docker for a while now and have now switched to docker-compose and have been using traefik to access Nextcloud on my local server from external locations. This was working fine (with some help from this community :slight_smile:), but when I added another container (ombi) to be exposed externally via traefik, I only get a "502 Bad Gateway" error, despite having the exact same configuration. I tried using only the new container, in case a duplicate load-balancer port is not allowed but still the same.

Here is the relevant snippet of my docker-compose.yml (the remainder being containers not exposed via traefik):

version: '3'

networks:
  proxy:
    external: true

  traefik:
    image: traefik:v2.0.4
    container_name: traefik
    restart: unless-stopped
    networks:
      - proxy
    ports:
      - 80:80
      - 443:443
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /opt/traefik/acme.json:/acme.json
      - /opt/traefik/traefik.yml:/traefik.yml:ro
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.entrypoints=http"
      - "traefik.http.routers.traefik.rule=Host(`traefik.domain.net`)"
      - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
      - "traefik.http.routers.traefik-secure.entrypoints=https"
      - "traefik.http.routers.traefik-secure.rule=Host(`traefik.domain.net`)"
      - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
      - "traefik.http.routers.traefik-secure.tls=true"
      - "traefik.http.routers.traefik-secure.tls.certresolver=http"
      - "traefik.http.routers.traefik-secure.service=api@internal"

  whoami:
    image: "containous/whoami"
    container_name: "simple-whoami"
    networks:
      - proxy
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami.rule=Host(`whoami.domain.net`)"
      - "traefik.http.routers.whoami.entrypoints=https"
      - "traefik.http.routers.whoami.tls.certresolver=http"

  ombi:
    image: linuxserver/ombi
    container_name: ombi
    networks:
      - proxy
    environment:
      - PUID=1001
      - PGID=1001
      - TZ=Europe/Berlin
    volumes:
      - /opt/ombi:/config
    restart: unless-stopped
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.ombi-secure.entrypoints=https"
      - "traefik.http.routers.ombi-secure.rule=Host(`ombi.domain.net`)"
      - "traefik.http.routers.ombi-secure.tls=true"
      - "traefik.http.routers.ombi-secure.tls.certresolver=http"
      - "traefik.http.routers.ombi-secure.service=ombi"
      - "traefik.http.routers.ombi.entrypoints=http"
      - "traefik.http.routers.ombi.rule=Host(`ombi.domain.net`)"
      - "traefik.http.middlewares.ombi-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.routers.ombi.middlewares=ombi-https-redirect"
      - "traefik.http.services.ombi.loadbalancer.server.port=443"
      - "traefik.http.services.ombi.loadbalancer.server.scheme=https"

  nextcloud:
    image: linuxserver/nextcloud
    container_name: nextcloud
    networks:
      - proxy
    environment:
      - PUID=1001
      - PGID=1001
      - TZ=Europe/Berlin
    volumes:
      - /opt/nextcloud:/config
      - /opt/nextcloud_data:/data
    restart: unless-stopped
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.nextcloud-secure.entrypoints=https"
      - "traefik.http.routers.nextcloud-secure.rule=Host(`nextcloud.domain.net`)"
      - "traefik.http.routers.nextcloud-secure.tls=true"
      - "traefik.http.routers.nextcloud-secure.tls.certresolver=http"
      - "traefik.http.routers.nextcloud-secure.service=nextcloud"
      - "traefik.http.services.nextcloud.loadbalancer.server.port=443"
      - "traefik.http.services.nextcloud.loadbalancer.server.scheme=https"
      - "traefik.http.routers.nextcloud.entrypoints=http"
      - "traefik.http.routers.nextcloud.rule=Host(`nextcloud.domain.net`)"
      - "traefik.http.middlewares.nextcloud-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.routers.nextcloud.middlewares=nextcloud-https-redirect"

traefik.yml:

api:
  dashboard: true

log:
  level: DEBUG

entryPoints:
  http:
    address: ":80"
  https:
    address: ":443"

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
    network: proxy

certificatesResolvers:
  http:
    acme:
      email: name@gmail.com
      storage: acme.json
      httpChallenge:
        entryPoint: http
        
serversTransport:
  insecureSkipVerify: true

Part of log:

traefik      | time="2019-11-20T10:50:31Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"*/*\"],\"User-Agent\":[\"curl/7.58.0\"],\"X-Forwarded-Host\":[\"ombi.domain.net\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"c6e63aeff7e6\"],\"X-Real-Ip\":[\"1.2.3.4\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"ombi.domain.net\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"1.2.3.4:42966\",\"RequestURI\":\"/\",\"TLS\":null}"

Any ideas? I am surely missing something stupid...

2

With a 502/HTTP error, it looks like that the backend container is answering an HTTP error to Traefik.

What are the logs of the containers?

1 Like
3

You can also do this:

  • Go to dashboard and find the service you are expecting to work that is not working. Note down the service url
  • Exec into the traefik conteiner and curl this url. Chances are that you won't be able to, that is that it errors out.
  • Find out why and eliminate the reason

Come back if you need help with the above!

1 Like
4

Thanks for the replies - you both might me right here, I did not think of actually looking at the container's logs, just traefik's logs. So, ombi's log says:

ombi         | We are running on
ombi         | /opt/ombi
ombi         | Hosting environment: Production
ombi         | Content root path: /opt/ombi
ombi         | Now listening on: http://[::]:3579
ombi         | Application started. Press Ctrl+C to shut down.

And the service in question (as per traefik's dashboard) is listening on https://172.18.0.3:443.
Seems to me that traefik does not know where to forward the incoming requests and tries to contact the container on port 443, could that be it?

curl --verbose https://172.18.0.3:443 results in
curl: (7) Failed to connect to 172.18.0.3 port 443: Connection refused,
whereas curl --verbose https://172.18.0.3:3579 gives the following:

* Rebuilt URL to: https://172.18.0.3:3579/
*   Trying 172.18.0.3...
* TCP_NODELAY set
* Connected to 172.18.0.3 (172.18.0.3) port 3579 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* error:1408F10B:SSL routines:ssl3_get_record:wrong version number
* stopped the pause stream!
* Closing connection 0
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

Simply changing the loadbalancer port to 3579 does not work however. Since traefik.port is deprecated in 2.0, how does one specify the required port? Add an entrypoint apart from http & https in traefik.yml?

5

Have a look here: https://docs.traefik.io/routing/providers/docker/#services for the full reference, but yes you need to change the port. Also check if ombi receives TLS connection or not and update your config accordingly.

1 Like
6

Change the port via "traefik.http.services.ombi.loadbalancer.server.port=3579" (did not work at first try) or via the entry points?
curl --verbose http://172.18.0.3:3579 results in the regular ombi start page being shown.

7

Yep, that's right. Carry on! :wink:

1 Like
8

You're almost there!

Just a tip to remember:

  • "Entrypoint" (in Traefik) allows to define the public port to access traefik from the outside
  • Label traefik.http.services.ombi.loadbalancer.server.port=3579 associated to a docker container describes the private port that Traefik will use to contact this container.
1 Like
9

@dduportal not to confuse everyone, but I start thinking that "service" may be not the best name for what it represents in traefik. First, when dealing with kubernetes provider people confuse kubernetes and traefik services all the time, and secondly, commonly in IT when you are a service you provide the service. E.g. Windows Services are applications that are running in the background which other entities can call (if the service allow it). In kubernetes Service is a collection of endpoints other entities can use to reach an application, but traefik has it backwards, in traefik a service is the the "address" of an external entity that traefik itself calls. This probably one of the reason why quite a few people have trouble wrapping their heads around the concept.

I'm not suggesting to change the name, mind you, probably too late for that, just voicing some thoughts.

2 Likes
10

Okay, read the reference and gathered I do not need to add an additional entrypoint but rather need to specify the port traefik is forwarding the traffic to. However, I cannot for the life of me find another thing that I might need to do apart from traefik.http.services.ombi.loadbalancer.server.port=3579. The fact that the container responds to http requests makes me think that the container cannot respond to https requests.
When I remove the secure entrypoints and the https-redirect, everything works fine.

11

You might want to add the label

- traefik.http.services.ombi.loadbalancer.server.scheme=https

to specify the scheme for communicating with the backend ombi container, as I understand ombi is doing HTTPS right?

Also, if ombi provides a self signed certificate, you might want to skip the tls verify, or better: providing the certificate to Traefik so it can dial with ombi in TLS securely. Both options can be found on https://docs.traefik.io/reference/static-configuration/file/ (clue: check [serversTransport] section :slight_smile: )

1 Like
12

Thanks, that simple line did the work. A few minutes ago I thought that ombi might not support HTTPS but apparently it does. So I need both entrypoints for the loadbalancer - any idea why nextcloud works with only the http-version? Because NC listens on ports 80/443 by default and ombi does not?
Anyway, I'm happy now! :slight_smile:

13

There is absolutely no link between entrypoint and loadbalancer. Take time to read https://docs.traefik.io/v2.0/routing/overview/ and look at the diagram :slight_smile:

For ombi, you need 2 routers, one for HTTP traffic and another for HTTPS traffic. The first one is bound to the entrypoint for public HTTP (e.g. external port 80), and the other on to 443. But whatever "entry point" is used, the request should always be sent to the private port 3579 of you container, and only in HTTPS: both router share the same "Traefik Service", which defines the settings to communicate with the backend (e.g. the port and scheme).

It means that, for the "HTTPS" from the public port 443, Traefik will terminate TLS, and then initate a new client -> server HTTPS connection to the backend.

I'm not sure to understand your question about nextcloud? Traefik listens to ports 80 and 443, publishing these ports. If requests come to Traefik with the right hostname, then it's sent to the corresponding router.