Bad gateway on Portainer

yannisant · November 6, 2022, 11:49pm

Hello,

I tried to find a solution on the community but I cannot get what I am doing wrong!
I am trying to run Portainer on docker swarm. The traefik gui loads fine and I can see the portainer service running.
Any ideas?

[global]
  checkNewVersion = true

# Enable the Dashboard
[api]
  dashboard = true

# Write out Traefik logs
[log]
  level = "DEBUG"
  filePath = "/traefik.log"

[entryPoints.http]
  address = ":80"

  [entryPoints.http.http.redirections.entryPoint]
    to = "https"
    scheme = "https"

[entryPoints.https]
  address = ":443"
  [entryPoints.https.http.tls]
    certResolver = "main"


# Let's Encrypt
[certificatesResolvers.main.acme]
  email = "user@gmail.com"
  storage = "acme.json"

  [certificatesResolvers.main.acme.dnsChallenge]
    provider = "route53"
  
# Docker Traefik provider
[providers.docker]
  endpoint = "unix:///var/run/docker.sock"
  swarmMode = true
  watch = true

[tls.options]
  [tls.options.default]
    minVersion = "VersionTLS13"
    sniStrict = true

  [tls.options.tls12]
    minVersion = "VersionTLS12"
    sinStrict = true
    cipherSuites = [
      "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
      "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
      "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
      "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
      "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305",
      "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"
    ]

version: '3.2'

services:
  agent:
    image: portainer/agent:latest
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /var/lib/docker/volumes:/var/lib/docker/volumes
    networks:
      - traefik_public
    deploy:
      mode: global
      placement:
        constraints: [node.platform.os == linux]

  portainer:
    image: portainer/portainer-ee:latest
    #command: -H tcp://tasks.agent:9001 --tlsskipverify
    
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /home/user/webapp/portainer/portainer_data:/data
    networks:
      - traefik_public
    deploy:
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.portainer.tls.certresolver=main"
        - "traefik.http.routers.portainer.rule=Host(`portainer.mydomain.com`)"
        - "traefik.http.services.portainer.loadbalancer.server.port=9000"
        - "traefik.http.routers.portainer.entrypoints=https"
        - "traefik.http.services.portainer.loadbalancer.server.scheme=https"
        - "traefik.http.routers.portainer.tls=true"
      mode: replicated
      placement:
        constraints: [node.role == manager]

networks:
  traefik_public:
    external: true

bluepuma77 · November 7, 2022, 9:14am

What does Traefik debug log show?

yannisant · November 7, 2022, 10:43am

level=error msg="Could not define the service name for the router: too many services" providerName=docker routerName=traefik-secure

msg="service \"portainer-agent\" error: port is missing" container=portainer-agent-wdrqj7dm5hma0yhzmrthefgz1 providerName=docker

level=debug msg="Configuration received: {\"http\":{\"routers\":{\"api\":{\"entryPoints\":[\"https\"],\"service\":\"api@internal\",\"rule\":\"Host(`traefik.example.com`)\",\"tls\":{\"certResolver\":\"main\"}},\"portainer\":{\"entryPoints\":[\"https\"],\"service\":\"portainer\",\"rule\":\"Host(`portainer.example.com`)\",\"tls\":{\"certResolver\":\"main\"}}},\"services\":{\"dummy\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.1.3:9999\"}],\"passHostHeader\":true}},\"portainer\":{\"loadBalancer\":{\"servers\":[{\"url\":\"https://10.0.1.34:9000\"}],\"passHostHeader\":true}},\"traefik\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.1.3:8080\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"udp\":{}}" providerName=docker

msg="Skipping unchanged configuration." providerName=docker

bluepuma77 · November 7, 2022, 11:09am

I would try to remove

traefik.http.services.portainer.loadbalancer.server.scheme=https

Those are the labels on my working portainer service (adapt to your setup):

      labels:
        - "traefik.enable=true"
        - "traefik.docker.network=proxy"
        - "traefik.http.routers.portainer.entrypoints=websecure"
        - "traefik.http.routers.portainer.rule=Host(`portainer.example.com`)"
        - "traefik.http.services.portainer.loadbalancer.server.port=9000"
        - "traefik.http.services.portainer.loadbalancer.passhostheader=true"

By the way, for security I would set

providers.docker.exposedByDefault: false

in your Traefik static config (in your case adapt to .toml), then Traefik will not try to expose the agent. Docs.

yannisant · November 7, 2022, 11:50am

Hello @bluepuma77 ,

thank you for your reply. I tried with my labels (removing the https scheme as instructed) and I also tried with your setup (adapted). Still getting Bad gateway...

I also get this in the logs:

error msg="Could not define the service name for the router: too many services" routerName=traefik-secure providerName=docker
debug msg="Filtering disabled container" providerName=docker container=portainer-agent-rxrad3e1k0mr9auxja1778gje
debug msg="Configuration received: {\"http\":{\"routers\":{\"api\":{\"entryPoints\":[\"https\"],\"service\":\"api@internal\",\"rule\":\"Host(`traefik.example.com`)\",\"tls\":{\"certResolver\":\"main\"}},\"portainer\":{\"entryPoints\":[\"https\"],\"service\":\"portainer\",\"rule\":\"Host(`portainer.example.com`)\"}},\"services\":{\"dummy\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.1.3:9999\"}],\"passHostHeader\":true}},\"portainer\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.1.7:9000\"}],\"passHostHeader\":true}},\"traefik\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.1.3:8080\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"udp\":{}}" providerName=docker
debug msg="Skipping unchanged configuration." providerName=docker

bluepuma77 · November 7, 2022, 6:09pm

Is this auto-generated?

Maybe check your Traefik config.

yannisant · November 7, 2022, 9:26pm

this is my traefik.yml,
the .env file contains only the AWS tokens for route53

version: "3.9"

services:
  app:
    image: traefik:latest
    env_file: /user/user/webapp/traefik/traefikv2.env
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /home/user/webapp/traefik:/etc/traefik
      - /home/user/webapp/traefik/traefik.log:/traefik.log
      - /home/user/webapp/traefik/acme.json:/acme.json
    networks:
      - traefik_public
    deploy:
      mode: global
      labels:
        - "traefik.enable=true"
        - "traefik.docker.network=traefik_public"
        - "traefik.http.routers.traefik.rule=Host(`traefik.example.com`)"
        - "traefik.http.routers.traefik.entrypoints=http"
        - "traefik.http.routers.traefik.entrypoints=https"       
        - "traefik.http.routers.traefik.tls=true"
        - "traefik.http.routers.traefik.tls.certresolver=main"
        - "traefik.http.routers.traefik.service=api@internal"
        - "traefik.http.services.traefik.loadbalancer.server.port=8080"
        - "traefik.http.routers.traefik.tls.domains[0].main=example.com"
        - "traefik.http.routers.traefik.tls.domains[0].sans=*.example.com"     
      placement:
        constraints: [node.role == manager]

networks:
  traefik_public:
    external: true

yannisant · November 7, 2022, 9:55pm

the file above was edited. @bluepuma77 was right I had done some config wrong in my file. After the edit I get :

502 Bad Gateway caused by: dial tcp 10.0.1.3:9000: connect: connection refused
...
Filtering disabled container" providerName=docker container=portainer-agent

in the logs which means that the agent and portainer are failing for some reason...

bluepuma77 · November 8, 2022, 7:54am

The portainer-agent should be filtered, that's ok, as it should not be exposed by Traefik.

It seems Traefik, portainer and portainer-agent all joint the same network, that's good.

Did you adapt the network in the labels?

        - "traefik.http.routers.portainer.entrypoints=https"

yannisant · November 8, 2022, 10:36am

I tried to add this line by line on the agent and on portainer:

- "traefik.docker.network=traefik_public"
- "traefik.http.routers.portainer.entrypoints=https"`

still bad gateway...

I also tried your labels from your post above (with and without the "traefik.docker.network=traefik_public" and the "traefik.http.routers.portainer.entrypoints=https"

the only way I have managed to login to portainer so far is by opening

ports:
      - "9443:9443"

in the portainer service file and navigating to: portainer.example.com:9443

I'm sharing my setup, maybe you can see something I'm missing.
Thank you!

yannisant · November 8, 2022, 1:31pm

Finally I deleted everything and started from scratch.
I followed this amazing guide and everything works like a charm!

thank you!

Topic		Replies	Views
Getting bad gateway intermitently when routing to portainer through traefik Traefik v3 (latest) docker-swarm , tcp	3	27	November 4, 2024
Dashboard gives me an error (Bad gateway) Traefik v2 docker-swarm , dashboard-api	4	4278	September 26, 2019
Traefik route to portainer does not work reliable Traefik v2 docker-swarm	2	1200	March 12, 2023
Docker Swam 502 bad gateway error Traefik v2 docker , docker-swarm	11	1559	February 27, 2020
Traefik portainer bad certificate Traefik v2 docker-swarm	15	3594	December 19, 2022

Bad gateway on Portainer

Related topics