Bad gateway on docker service

Hi,
I know there are a lot of posts with a quite similar subject, but I can't find a solution to my issue.
My traefik configuration is the following. It works fine for other docker services:

traefik:
    image: traefik:v2.10
    ports:
      - 80:80
      - 443:443
    container_name: reverse-proxy
    networks:
      - proxy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - letsencrypt:/letsencrypt
      - /var/log:/var/log
    command:
      - --api.dashboard=true
      - --log.level=INFO
      - --accesslog=true
      - --providers.docker.network=proxy
      - --providers.docker.exposedByDefault=false
      - --entrypoints.web.address=:80
      - --entrypoints.web.http.redirections.entrypoint.to=websecure
      - --entryPoints.web.http.redirections.entrypoint.scheme=https
      - --entrypoints.websecure.address=:443
      - --entrypoints.websecure.http.tls.certresolver=myresolver
      - --certificatesresolvers.myresolver.acme.email=mathieu.leclaire@iscpif.fr
      - --certificatesresolvers.myresolver.acme.tlschallenge=true
      - --certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json
    labels:
      - traefik.enable=true
      - traefik.http.routers.mydashboard.entrypoints=websecure
      - "traefik.http.routers.mydashboard.rule=Host(`traefik.$MY_DOMAIN`)"
      - traefik.http.routers.mydashboard.service=api@internal

I have a docker service exposed on port 9002 (final line of my Dockerfile is EXPOSE 9002) with the following definition in my docker-compose:

  mobiliquest:
    build:
      context: .
      dockerfile: mobiliquest/Dockerfile
    security_opt:
       - no-new-privileges:true
    container_name: mobiliquest
      # ports:
      # - 9002:9002
    volumes:
      - '/data/minio:/data'
    labels:
      - traefik.enable=true
      - traefik.http.routers.mobiliquest.entrypoints=websecure
      - traefik.http.routers.mobiliquest.service=mobiliquest
      - "traefik.http.routers.mobiliquest.rule=Host(`mobiliquest.$MY_DOMAIN`)"
      -  traefik.http.services.mobiliquest.loadbalancer.server.port=9002
    networks:
      - proxy

When I up the docker-compose, everything seems to be OK in logs and in traefik dashboard. But I can't reach the service:

curl -i http://172.22.0.2:9002
curl: (7) Failed to connect to 172.22.0.2 port 9002 after 0 ms: Connexion refusée

A docker ps gives:

CONTAINER ID   IMAGE                                      COMMAND                  CREATED          STATUS          PORTS                                      NAMES
c53692e659c6   mobiliscope-services_mobiliquest           "/bin/sh -c 'bash /v…"   10 minutes ago   Up 10 minutes   9002/tcp                                   mobiliquest

and the subdomain entry mobiliquest. has been properly added to my DNS conf.
I don't understand why traefik cannot reach the mobiliquest service and serve it.

Thanks for your help

Are you sure the service (mobiliquest) is running on port 9002 and has it properly exposed?
Also I usually the ports I want exposed in the 'exposed' section, just for good measure.

You are right, it does not run on port 9002.
If I run my server without docker on my machine, I can reach it.
curl -i http://127.0.0.1:9002 returns HTTP/1.1 200 OK
but it fails when I start it from a docker / docker-compose (without traefik). What do you think is missing in my Dockerfile / docker-compose?

services:
    mobiliquest:
      build:
        context: .
        dockerfile: mobiliquest/Dockerfile
      security_opt:
         - no-new-privileges:true
      container_name: mobiliquest
      ports:
        - 9002:9002
      volumes:
       - '/data/minio:/data'

FROM eed3si9n/sbt:latest as mobiliquest

USER root

RUN apk update && apk add bash git sudo npm

RUN npm install npm@9.3.1
RUN npm install -g n
RUN n 19.4.0

RUN node --version

RUN mkdir -p /var/mobiliquest
RUN adduser mobiliquest -g "" -D -h /var/mobiliquest/

COPY mobiliquest/run.sh /var/mobiliquest/run.sh
COPY mobiliquest/compile.sh /var/mobiliquest/compile.sh
RUN chmod +x /var/mobiliquest/run.sh
RUN chmod +x /var/mobiliquest/compile.sh

USER mobiliquest

RUN cd /var/mobiliquest && \
    git config --global http.sslVerify "false" && \
    git clone https://github.com/Geographie-cites/mobiliquest.git && \
    cd mobiliquest && \
    git checkout master

RUN bash /var/mobiliquest/compile.sh

ENTRYPOINT bash /var/mobiliquest/run.sh

If anything I think you are missing the following statement in your Dockerfile.

EXPOSE 9002

Also how are you accessing it locally (without docker)? Just localhost:9002 right?

Your Dockerfile also seems to be off if you ask me. Cloning from within a container is a bit odd thing to do. It would be better to copy the source into it. An easier solution would be to just copy the output of the build process into it (this is the easiest, unless you want to use multi-stage builds).

I used to use EXPOSE 9002 but it does not help. I get this (localhost returns the same):

curl -i http://127.0.0.1:9002
curl: (56) Recv failure: Connexion ré-initialisée par le correspondant

Concerning the cloning, I need to rebuild an application from fresh commits, that why I use it this way, I didn't know it wass odd :). I guess it is uncorrelated with my port issue, right ?

Depends, a 'odd' build makes your build process a bit more obscure which could lead to hidden bugs.
As for the argument of "I need to rebuild an application from fresh commits", what prevents you from doing a git pull and docker build <args> .? This would send the whole new source to the docker daemon which can then just run your build commands.

As for your problem, are you sure the container is properly starting? In other words, what is the output of docker ps?

Edit: Docker recommends the following format for an entrypoint:

ENTRYPOINT ["executable", "param1", "param2"]

If I only pull, my docker is not self content, I need to initialize the repo. How would you do that in an other way ?

docker ps returns:

CONTAINER ID   IMAGE                    COMMAND                  CREATED          STATUS          PORTS                                       NAMES
31a292113f2c   testdocker_mobiliquest   "/bin/sh -c 'bash /v…"   29 minutes ago   Up 29 minutes   0.0.0.0:9002->9002/tcp, :::9002->9002/tcp   mobiliquest

Normally the Dockerfile is in the repo with all the files you need to build the final image. Source, scripts etc.

Looking at your COMMAND section I think your application might just be in the bash shell instead of executing. You should be able to verify that your application is running my using docker logs <container_id>. This will give you the output of the container, and allows you to verify if the application is actually running.

The logs returns the correct outputs when the server is run. I inspected the docker bash and run a top, my server seems to be up:

    7     1 mobiliqu S    5721m  34%   3   0% java -XX:+UseContainerSupport -Dfile.encoding=UTF-8 -Xmx2048M -Xss2M -jar /opt/sbt/sbt/bin/sbt-launch.jar project server;run
  172   165 mobiliqu R     1588   0%   2   0% top
  165     0 mobiliqu S     2416   0%   1   0% /bin/bash
    1     0 mobiliqu S     2212   0%   1   0% bash /var/mobiliquest/run.sh

"Bad Gateway" is usually an issue with Docker network. You don’t show your full docker-compose.yml, we can’t see how it is declared.

It is because I tried to reproduced the issue at the docker level. It appears that the port mapping does not work at the docker level. That's why we do not think it is a traefik issue. Here is my traefik docker-compose anyway:

services:
  traefik:
    image: traefik:v2.10
    ports:
      - 80:80
      - 443:443
    container_name: reverse-proxy
    networks:
      - proxy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - letsencrypt:/letsencrypt
      - /var/log:/var/log
    command:
      - --api.dashboard=true
      - --log.level=INFO
      - --accesslog=true
      - --providers.docker.network=proxy
      - --providers.docker.exposedByDefault=false
      - --entrypoints.web.address=:80
      - --entrypoints.web.http.redirections.entrypoint.to=websecure
      - --entryPoints.web.http.redirections.entrypoint.scheme=https
      - --entrypoints.websecure.address=:443
      - --entrypoints.websecure.http.tls.certresolver=myresolver
      - --certificatesresolvers.myresolver.acme.email=mathieu.leclaire@iscpif.fr
      - --certificatesresolvers.myresolver.acme.tlschallenge=true
      - --certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json
    labels:
      - traefik.enable=true
      - traefik.http.routers.mydashboard.entrypoints=websecure
      - "traefik.http.routers.mydashboard.rule=Host(`traefik.$MY_DOMAIN`)"
      - traefik.http.routers.mydashboard.service=api@internal

  mobiliquest:
    build:
      context: .
      dockerfile: mobiliquest/Dockerfile
    security_opt:
       - no-new-privileges:true
    container_name: mobiliquest
      # ports:
      # - 9002:9002
    volumes:
      - '/data/minio:/data'
    labels:
      - traefik.enable=true
      - traefik.http.routers.mobiliquest.entrypoints=websecure
      - traefik.http.routers.mobiliquest.service=mobiliquest
      - "traefik.http.routers.mobiliquest.rule=Host(`mobiliquest.$MY_DOMAIN`)"
      -  traefik.http.services.mobiliquest.loadbalancer.server.port=9002
    networks:
      - proxy
networks:
  proxy:
    external: true
  internal:
    driver: bridge

volumes:
  letsencrypt:
    name: letsencrypt

What does Traefik dashboard tell you? What does Traefik debug log tell you?

Your target service runs on port 9002 container internally, outside containers you can only reach it via Traefik by the domain name from rule (not by IP) and with the standard TLS port 443.

You can probably get the Docker network IP of your target service via docker inspect, from inside your Traefik container you can try wget http://IP:9002 to test the internal connection.

The dashboard does not mention any error. Should I inspect something in particular ?
The debug logs when starting services is :

time="2023-06-02T08:38:02Z" level=info msg="Configuration loaded from flags."
time="2023-06-02T08:38:02Z" level=info msg="Traefik version 2.10.1 built on 2023-04-27T14:52:35Z"
time="2023-06-02T08:38:02Z" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"web\":{\"address\":\":80\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{\"redirections\":{\"entryPoint\":{\"to\":\"websecure\",\"scheme\":\"https\",\"permanent\":true,\"priority\":2147483646}}},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}},\"websecure\":{\"address\":\":443\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{\"tls\":{\"certResolver\":\"myresolver\"}},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}}},\"providers\":{\"providersThrottleDuration\":\"2s\",\"docker\":{\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"network\":\"proxy\",\"swarmModeRefreshSeconds\":\"15s\"}},\"api\":{\"dashboard\":true},\"log\":{\"level\":\"DEBUG\",\"format\":\"common\"},\"accessLog\":{\"format\":\"common\",\"filters\":{},\"fields\":{\"defaultMode\":\"keep\",\"headers\":{\"defaultMode\":\"drop\"}}},\"certificatesResolvers\":{\"myresolver\":{\"acme\":{\"email\":\"mathieu.leclaire@iscpif.fr\",\"caServer\":\"https://acme-v02.api.letsencrypt.org/directory\",\"storage\":\"/letsencrypt/acme.json\",\"keyType\":\"RSA4096\",\"certificatesDuration\":2160,\"tlsChallenge\":{}}}}}"
time="2023-06-02T08:38:02Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\n"
time="2023-06-02T08:38:02Z" level=info msg="Starting provider aggregator aggregator.ProviderAggregator"
time="2023-06-02T08:38:02Z" level=debug msg="Starting TCP Server" entryPointName=websecure
time="2023-06-02T08:38:02Z" level=debug msg="Starting TCP Server" entryPointName=web
time="2023-06-02T08:38:02Z" level=info msg="Starting provider *traefik.Provider"
time="2023-06-02T08:38:02Z" level=debug msg="*traefik.Provider provider configuration: {}"
time="2023-06-02T08:38:02Z" level=info msg="Starting provider *docker.Provider"
time="2023-06-02T08:38:02Z" level=debug msg="*docker.Provider provider configuration: {\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"network\":\"proxy\",\"swarmModeRefreshSeconds\":\"15s\"}"
time="2023-06-02T08:38:02Z" level=info msg="Starting provider *acme.Provider"
time="2023-06-02T08:38:02Z" level=debug msg="*acme.Provider provider configuration: {\"email\":\"mathieu.leclaire@iscpif.fr\",\"caServer\":\"https://acme-v02.api.letsencrypt.org/directory\",\"storage\":\"/letsencrypt/acme.json\",\"keyType\":\"RSA4096\",\"certificatesDuration\":2160,\"tlsChallenge\":{},\"ResolverName\":\"myresolver\",\"store\":{},\"TLSChallengeProvider\":{},\"HTTPChallengeProvider\":{}}"
time="2023-06-02T08:38:02Z" level=info msg="Starting provider *acme.ChallengeTLSALPN"
time="2023-06-02T08:38:02Z" level=debug msg="*acme.ChallengeTLSALPN provider configuration: {}"
time="2023-06-02T08:38:02Z" level=debug msg="Attempt to renew certificates \"720h0m0s\" before expiry and check every \"24h0m0s\"" ACME CA="https://acme-v02.api.letsencrypt.org/directory" providerName=myresolver.acme
time="2023-06-02T08:38:02Z" level=info msg="Testing certificate renew..." providerName=myresolver.acme ACME CA="https://acme-v02.api.letsencrypt.org/directory"
time="2023-06-02T08:38:02Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"web-to-websecure\":{\"entryPoints\":[\"web\"],\"middlewares\":[\"redirect-web-to-websecure\"],\"service\":\"noop@internal\",\"rule\":\"HostRegexp(`{host:.+}`)\",\"priority\":2147483646}},\"services\":{\"api\":{},\"dashboard\":{},\"noop\":{}},\"middlewares\":{\"redirect-web-to-websecure\":{\"redirectScheme\":{\"scheme\":\"https\",\"port\":\"443\",\"permanent\":true}}},\"models\":{\"websecure\":{\"tls\":{\"certResolver\":\"myresolver\"}}},\"serversTransports\":{\"default\":{\"maxIdleConnsPerHost\":200}}},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=internal
time="2023-06-02T08:38:02Z" level=debug msg="Configuration received: {\"http\":{},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=myresolver.acme
time="2023-06-02T08:38:02Z" level=debug msg="Provider connection established with docker 23.0.6 (API 1.42)" providerName=docker
time="2023-06-02T08:38:02Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"mobiliquest\":{\"entryPoints\":[\"websecure\"],\"service\":\"mobiliquest\",\"rule\":\"Host(`mobiliquest.ajmr.mobiliscope.com`)\"},\"mydashboard\":{\"entryPoints\":[\"websecure\"],\"middlewares\":[\"myauth\"],\"service\":\"api@internal\",\"rule\":\"Host(`traefik.ajmr.mobiliscope.com`)\"}},\"services\":{\"mobiliquest\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.22.0.2:9002\"}],\"passHostHeader\":true}},\"traefik-mobiliscope-services\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.22.0.3:80\"}],\"passHostHeader\":true}}},\"middlewares\":{\"myauth\":{\"basicAuth\":{\"users\":[\"admin:$2y$05$HlenhNuTSVHbdVUqSEjWTuFjFgOA.ekitdNy35ywHAgZe/pJN.dq.\"]}}}},\"tcp\":{},\"udp\":{}}" providerName=docker
time="2023-06-02T08:38:03Z" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default
time="2023-06-02T08:38:03Z" level=debug msg="Added outgoing tracing middleware noop@internal" middlewareName=tracing middlewareType=TracingForwarder routerName=web-to-websecure@internal entryPointName=web
time="2023-06-02T08:38:03Z" level=debug msg="Creating middleware" routerName=web-to-websecure@internal middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme entryPointName=web
time="2023-06-02T08:38:03Z" level=debug msg="Setting up redirection to https 443" entryPointName=web routerName=web-to-websecure@internal middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme
time="2023-06-02T08:38:03Z" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=web middlewareName=traefik-internal-recovery
time="2023-06-02T08:38:03Z" level=debug msg="Adding certificate for domain(s) ajmr.mobiliscope.com"
time="2023-06-02T08:38:03Z" level=debug msg="Adding certificate for domain(s) traefik.ajmr.mobiliscope.com"
time="2023-06-02T08:38:03Z" level=debug msg="Adding certificate for domain(s) minio.ajmr.mobiliscope.com"
time="2023-06-02T08:38:03Z" level=debug msg="Adding certificate for domain(s) minioadmin.ajmr.mobiliscope.com"
time="2023-06-02T08:38:03Z" level=debug msg="Adding certificate for domain(s) mobiliquest.ajmr.mobiliscope.com"
time="2023-06-02T08:38:03Z" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default
time="2023-06-02T08:38:03Z" level=debug msg="Added outgoing tracing middleware noop@internal" routerName=web-to-websecure@internal middlewareName=tracing middlewareType=TracingForwarder entryPointName=web
time="2023-06-02T08:38:03Z" level=debug msg="Creating middleware" entryPointName=web routerName=web-to-websecure@internal middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme
time="2023-06-02T08:38:03Z" level=debug msg="Setting up redirection to https 443" entryPointName=web routerName=web-to-websecure@internal middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme
time="2023-06-02T08:38:03Z" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=web
time="2023-06-02T08:38:03Z" level=debug msg="Creating middleware" middlewareType=Pipelining entryPointName=websecure routerName=mobiliquest@docker serviceName=mobiliquest middlewareName=pipelining
time="2023-06-02T08:38:03Z" level=debug msg="Creating load-balancer" entryPointName=websecure routerName=mobiliquest@docker serviceName=mobiliquest
time="2023-06-02T08:38:03Z" level=debug msg="Creating server 0 http://172.22.0.2:9002" routerName=mobiliquest@docker serviceName=mobiliquest serverName=0 entryPointName=websecure
time="2023-06-02T08:38:03Z" level=debug msg="child http://172.22.0.2:9002 now UP"
time="2023-06-02T08:38:03Z" level=debug msg="Propagating new UP status"
time="2023-06-02T08:38:03Z" level=debug msg="Added outgoing tracing middleware mobiliquest" middlewareType=TracingForwarder middlewareName=tracing entryPointName=websecure routerName=mobiliquest@docker
time="2023-06-02T08:38:03Z" level=debug msg="Added outgoing tracing middleware api@internal" middlewareType=TracingForwarder entryPointName=websecure routerName=mydashboard@docker middlewareName=tracing
time="2023-06-02T08:38:03Z" level=debug msg="Creating middleware" middlewareType=BasicAuth entryPointName=websecure routerName=mydashboard@docker middlewareName=myauth@docker
time="2023-06-02T08:38:03Z" level=debug msg="Adding tracing to middleware" entryPointName=websecure routerName=mydashboard@docker middlewareName=myauth@docker
time="2023-06-02T08:38:03Z" level=debug msg="Creating middleware" entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2023-06-02T08:38:03Z" level=debug msg="Adding route for mobiliquest.ajmr.mobiliscope.com with TLS options default" entryPointName=websecure
time="2023-06-02T08:38:03Z" level=debug msg="Adding route for traefik.ajmr.mobiliscope.com with TLS options default" entryPointName=websecure
time="2023-06-02T08:38:03Z" level=debug msg="Trying to challenge certificate for domain [mobiliquest.ajmr.mobiliscope.com] found in HostSNI rule" providerName=myresolver.acme ACME CA="https://acme-v02.api.letsencrypt.org/directory" routerName=mobiliquest@docker rule="Host(`mobiliquest.ajmr.mobiliscope.com`)"
time="2023-06-02T08:38:03Z" level=debug msg="Trying to challenge certificate for domain [traefik.ajmr.mobiliscope.com] found in HostSNI rule" rule="Host(`traefik.ajmr.mobiliscope.com`)" providerName=myresolver.acme ACME CA="https://acme-v02.api.letsencrypt.org/directory" routerName=mydashboard@docker
time="2023-06-02T08:38:03Z" level=debug msg="Looking for provided certificate(s) to validate [\"mobiliquest.ajmr.mobiliscope.com\"]..." providerName=myresolver.acme ACME CA="https://acme-v02.api.letsencrypt.org/directory" routerName=mobiliquest@docker rule="Host(`mobiliquest.ajmr.mobiliscope.com`)"
time="2023-06-02T08:38:03Z" level=debug msg="No ACME certificate generation required for domains [\"mobiliquest.ajmr.mobiliscope.com\"]." ACME CA="https://acme-v02.api.letsencrypt.org/directory" routerName=mobiliquest@docker rule="Host(`mobiliquest.ajmr.mobiliscope.com`)" providerName=myresolver.acme
time="2023-06-02T08:38:03Z" level=debug msg="Looking for provided certificate(s) to validate [\"traefik.ajmr.mobiliscope.com\"]..." ACME CA="https://acme-v02.api.letsencrypt.org/directory" routerName=mydashboard@docker rule="Host(`traefik.ajmr.mobiliscope.com`)" providerName=myresolver.acme
time="2023-06-02T08:38:03Z" level=debug msg="No ACME certificate generation required for domains [\"traefik.ajmr.mobiliscope.com\"]." providerName=myresolver.acme ACME CA="https://acme-v02.api.letsencrypt.org/directory" routerName=mydashboard@docker rule="Host(`traefik.ajmr.mobiliscope.com`)"

And for some reason I cannot bash the traefik docker (but I can bash the mobilquest one). Here is the error:

> docker exec -ti reverse-proxy /bin/bash
OCI runtime exec failed: exec failed: unable to start container process: exec: "/bin/bash": stat /bin/bash: no such file or directory: unknown

Yes, bash is an advanced shell (needs space, security implications), not included in all images

Use docker exec -it reverse-proxy sh instead.

OK, thanks.
So, from the reverse-proxy docker (with url picked up from dashboard):

/ # wget http://172.22.0.2:9002
Connecting to 172.22.0.2:9002 (172.22.0.2:9002)
wget: can't connect to remote host (172.22.0.2): Connection refused

Either the IP is wrong, the port is not open, they don’t share the same Docker Network or the network is broken.

Do you use Docker Swarm and overlay network, potentially with a vSwitch?

Nope. The point is I can't reach my service just starting it from docker (without neither traefik nor docker-compose) . I think the problem starts here. Don't you think ?

If you cannot reach the container just using simple port binds then Traefik won't be able to either.
I would suggest sorting out that issue first, and then add Traefik to the mix.

Does this work within the target container?

(Alternatively try with curl)