Bad Gateway 502 with docker containers

Been using docker for a while now and have now switched to docker-compose and have been using traefik to access Nextcloud on my local server from external locations. This was working fine (with some help from this community :slight_smile:), but when I added another container (ombi) to be exposed externally via traefik, I only get a "502 Bad Gateway" error, despite having the exact same configuration. I tried using only the new container, in case a duplicate load-balancer port is not allowed but still the same.

Here is the relevant snippet of my docker-compose.yml (the remainder being containers not exposed via traefik):

version: '3'

    external: true

    image: traefik:v2.0.4
    container_name: traefik
    restart: unless-stopped
      - proxy
      - 80:80
      - 443:443
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /opt/traefik/acme.json:/acme.json
      - /opt/traefik/traefik.yml:/traefik.yml:ro
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.entrypoints=http"
      - "traefik.http.routers.traefik.rule=Host(``)"
      - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
      - "traefik.http.routers.traefik-secure.entrypoints=https"
      - "traefik.http.routers.traefik-secure.rule=Host(``)"
      - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
      - "traefik.http.routers.traefik-secure.tls=true"
      - "traefik.http.routers.traefik-secure.tls.certresolver=http"
      - "traefik.http.routers.traefik-secure.service=api@internal"

    image: "containous/whoami"
    container_name: "simple-whoami"
      - proxy
      - "traefik.enable=true"
      - "traefik.http.routers.whoami.rule=Host(``)"
      - "traefik.http.routers.whoami.entrypoints=https"
      - "traefik.http.routers.whoami.tls.certresolver=http"

    image: linuxserver/ombi
    container_name: ombi
      - proxy
      - PUID=1001
      - PGID=1001
      - TZ=Europe/Berlin
      - /opt/ombi:/config
    restart: unless-stopped
      - "traefik.enable=true"
      - "traefik.http.routers.ombi-secure.entrypoints=https"
      - "traefik.http.routers.ombi-secure.rule=Host(``)"
      - "traefik.http.routers.ombi-secure.tls=true"
      - "traefik.http.routers.ombi-secure.tls.certresolver=http"
      - "traefik.http.routers.ombi-secure.service=ombi"
      - "traefik.http.routers.ombi.entrypoints=http"
      - "traefik.http.routers.ombi.rule=Host(``)"
      - "traefik.http.middlewares.ombi-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.routers.ombi.middlewares=ombi-https-redirect"
      - ""
      - ""

    image: linuxserver/nextcloud
    container_name: nextcloud
      - proxy
      - PUID=1001
      - PGID=1001
      - TZ=Europe/Berlin
      - /opt/nextcloud:/config
      - /opt/nextcloud_data:/data
    restart: unless-stopped
      - "traefik.enable=true"
      - "traefik.http.routers.nextcloud-secure.entrypoints=https"
      - "traefik.http.routers.nextcloud-secure.rule=Host(``)"
      - "traefik.http.routers.nextcloud-secure.tls=true"
      - "traefik.http.routers.nextcloud-secure.tls.certresolver=http"
      - "traefik.http.routers.nextcloud-secure.service=nextcloud"
      - ""
      - ""
      - "traefik.http.routers.nextcloud.entrypoints=http"
      - "traefik.http.routers.nextcloud.rule=Host(``)"
      - "traefik.http.middlewares.nextcloud-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.routers.nextcloud.middlewares=nextcloud-https-redirect"


  dashboard: true

  level: DEBUG

    address: ":80"
    address: ":443"

    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
    network: proxy

      storage: acme.json
        entryPoint: http
  insecureSkipVerify: true

Part of log:

traefik      | time="2019-11-20T10:50:31Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"*/*\"],\"User-Agent\":[\"curl/7.58.0\"],\"X-Forwarded-Host\":[\"\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"c6e63aeff7e6\"],\"X-Real-Ip\":[\"\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"\",\"RequestURI\":\"/\",\"TLS\":null}"

Any ideas? I am surely missing something stupid...

With a 502/HTTP error, it looks like that the backend container is answering an HTTP error to Traefik.

What are the logs of the containers?

1 Like

You can also do this:

  • Go to dashboard and find the service you are expecting to work that is not working. Note down the service url
  • Exec into the traefik conteiner and curl this url. Chances are that you won't be able to, that is that it errors out.
  • Find out why and eliminate the reason

Come back if you need help with the above!

1 Like

Thanks for the replies - you both might me right here, I did not think of actually looking at the container's logs, just traefik's logs. So, ombi's log says:

ombi         | We are running on
ombi         | /opt/ombi
ombi         | Hosting environment: Production
ombi         | Content root path: /opt/ombi
ombi         | Now listening on: http://[::]:3579
ombi         | Application started. Press Ctrl+C to shut down.

And the service in question (as per traefik's dashboard) is listening on
Seems to me that traefik does not know where to forward the incoming requests and tries to contact the container on port 443, could that be it?

curl --verbose results in
curl: (7) Failed to connect to port 443: Connection refused,
whereas curl --verbose gives the following:

* Rebuilt URL to:
*   Trying
* Connected to ( port 3579 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* error:1408F10B:SSL routines:ssl3_get_record:wrong version number
* stopped the pause stream!
* Closing connection 0
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

Simply changing the loadbalancer port to 3579 does not work however. Since traefik.port is deprecated in 2.0, how does one specify the required port? Add an entrypoint apart from http & https in traefik.yml?

Have a look here: for the full reference, but yes you need to change the port. Also check if ombi receives TLS connection or not and update your config accordingly.

1 Like

Change the port via "" (did not work at first try) or via the entry points?
curl --verbose results in the regular ombi start page being shown.

Yep, that's right. Carry on! :wink:

1 Like

You're almost there!

Just a tip to remember:

  • "Entrypoint" (in Traefik) allows to define the public port to access traefik from the outside
  • Label associated to a docker container describes the private port that Traefik will use to contact this container.
1 Like

@dduportal not to confuse everyone, but I start thinking that "service" may be not the best name for what it represents in traefik. First, when dealing with kubernetes provider people confuse kubernetes and traefik services all the time, and secondly, commonly in IT when you are a service you provide the service. E.g. Windows Services are applications that are running in the background which other entities can call (if the service allow it). In kubernetes Service is a collection of endpoints other entities can use to reach an application, but traefik has it backwards, in traefik a service is the the "address" of an external entity that traefik itself calls. This probably one of the reason why quite a few people have trouble wrapping their heads around the concept.

I'm not suggesting to change the name, mind you, probably too late for that, just voicing some thoughts.


Okay, read the reference and gathered I do not need to add an additional entrypoint but rather need to specify the port traefik is forwarding the traffic to. However, I cannot for the life of me find another thing that I might need to do apart from The fact that the container responds to http requests makes me think that the container cannot respond to https requests.
When I remove the secure entrypoints and the https-redirect, everything works fine.

You might want to add the label


to specify the scheme for communicating with the backend ombi container, as I understand ombi is doing HTTPS right?

Also, if ombi provides a self signed certificate, you might want to skip the tls verify, or better: providing the certificate to Traefik so it can dial with ombi in TLS securely. Both options can be found on (clue: check [serversTransport] section :slight_smile: )

1 Like

Thanks, that simple line did the work. A few minutes ago I thought that ombi might not support HTTPS but apparently it does. So I need both entrypoints for the loadbalancer - any idea why nextcloud works with only the http-version? Because NC listens on ports 80/443 by default and ombi does not?
Anyway, I'm happy now! :slight_smile:

There is absolutely no link between entrypoint and loadbalancer. Take time to read and look at the diagram :slight_smile:

For ombi, you need 2 routers, one for HTTP traffic and another for HTTPS traffic. The first one is bound to the entrypoint for public HTTP (e.g. external port 80), and the other on to 443. But whatever "entry point" is used, the request should always be sent to the private port 3579 of you container, and only in HTTPS: both router share the same "Traefik Service", which defines the settings to communicate with the backend (e.g. the port and scheme).

It means that, for the "HTTPS" from the public port 443, Traefik will terminate TLS, and then initate a new client -> server HTTPS connection to the backend.

I'm not sure to understand your question about nextcloud? Traefik listens to ports 80 and 443, publishing these ports. If requests come to Traefik with the right hostname, then it's sent to the corresponding router.