Portainer docker behind Traefik / URL not working Gateway timeout error 504

ThierryIT · May 1, 2024, 2:47pm

services:
  traefik:
    image: traefik:v2.11.2
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    read_only: true
    mem_limit: 2G
    cpus: 0.75
    #depends_on:
      #- dockerproxy
    networks:
      - mynet
      #- socket-t
    #command:
     # - '--host=tcp://t-docker-socket-proxy:2375'
    ports:
      - 1180:80
      - 11443:443
      - 8087:8080
      - 1181:1181
      - 11444:11444
    environment:
      - CF_API_EMAIL=myemail
      - CF_DNS_API_TOKEN=nejxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxB-
      - TZ=Europe/Helsinki
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /mnt/user/appdata/docker/traefik/data/traefik.yml:/traefik.yml:ro
      - /mnt/user/appdata/docker/letsencrypt:/letsencrypt
      - /mnt/user/appdata/docker/traefik/data/dynamic_conf.yml:/dynamic_conf.yml:ro
      - /var/log/crowdsec/:/var/log/crowdsec

    
    labels:
      #- "com.centurylinklabs.watchtower.enable=false" # if watchtower would try to update the proxy, it would just stop
      - "traefik.enable=true"
      - "traefik.docker.network=mynet"  <<<<<<<<<<<<<<<<<<<<< add
      - "traefik.http.routers.traefik.entrypoints=http"
      - "traefik.http.routers.traefik.rule=Host(`traefik.domain.org`)"
      - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=https"
      - "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
      - "traefik.http.routers.traefik-secure.entrypoints=https"
      - "traefik.http.routers.traefik-secure.rule=Host(`traefik.domain.org`)"
      - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
      - "traefik.http.routers.traefik-secure.tls=true"
      - "traefik.http.routers.traefik-secure.tls.certresolver=dns-cloudflare"
      - "traefik.http.routers.traefik-secure.tls.domains[0].main=domain.org"
      - "traefik.http.routers.traefik-secure.tls.domains[0].sans=*.domain.org"
      - "traefik.http.routers.traefik-secure.service=api@internal"
      # middlewares
      - "traefik.http.middlewares.traefik-auth.basicauth.removeheader=true"
      - "traefik.http.middlewares.traefik-auth.basicauth.users=theboss:xxxxxxx" 
      # middlewares security headers
      - "traefik.http.middlewares.security-headers.headers.accesscontrolallowmethods=GET, OPTIONS, PUT"
      - "traefik.http.middlewares.security-headers.headers.accesscontrolmaxage=100"
      - "traefik.http.middlewares.security-headers.headers.addvaryheader=true"
      - "traefik.http.middlewares.security-headers.headers.hostsproxyheaders=X-Forwarded-Host"
      - "traefik.http.middlewares.security-headers.headers.sslredirect=true"
      - "traefik.http.middlewares.security-headers.headers.sslproxyheaders.X-Forwarded-Proto=https"
      - "traefik.http.middlewares.security-headers.headers.stsseconds=63072000"
      - "traefik.http.middlewares.security-headers.headers.stsincludesubdomains=true"
      - "traefik.http.middlewares.security-headers.headers.stspreload=true"
      - "traefik.http.middlewares.security-headers.headers.forcestsheader=true"
      - "traefik.http.middlewares.security-headers.headers.framedeny=true"
      - "traefik.http.middlewares.security-headers.headers.contenttypenosniff=true"
      - "traefik.http.middlewares.security-headers.headers.browserxssfilter=true"
      - "traefik.http.middlewares.security-headers.headers.referrerpolicy=same-origin"
      - "traefik.http.middlewares.security-headers.headers.featurepolicy=camera 'none'; geolocation 'none'; microphone 'none'; payment 'none'; usb 'none'; vr 'none';"
      - "traefik.http.middlewares.security-headers.headers.customresponseheaders.X-Robots-Tag=none,noarchive,nosnippet,notranslate,noimageindex"
  
networks:
  mynet:
    name: mynet
    external: true
  #socket-t:
    #driver: bridge
    #internal: true
    #attachable: false

my dynamic file:

portainer:
      entryPoints:
        - "https-external"
      rule: "Host(`portainer.domain.tld`)"
      middlewares:
        - https-redirectscheme
        #- authentik
      tls: {}
      service: portainer

portainer:
      loadBalancer:
        servers:
          - url: "http://192.xxx.xxx.xx:9443"
        passHostHeader: true

When trying https://portainer.domain.tld I do have the GW timeout ...

Thx

bluepuma77 · May 1, 2024, 6:45pm

We get Traefik compose and Portainer dynamic, but no Traefik static and no Portainer compose?

Gateway timeout is usually a Docker network not connected or using multiple Docker networks without specificity the right one with .docker.network=.

ThierryIT · May 2, 2024, 4:12am

traefik static:

api:
  dashboard: true
  insecure: true
  debug: true
entryPoints:
  http:
    address: ":80"
    forwardedHeaders:
      trustedIPs: 
        # Start of Clouflare public IP list for HTTP requests, remove this if you don't use it
        - 173.245.48.0/20
        - 103.21.244.0/22
        - 103.22.200.0/22
        - 103.31.4.0/22
        - 141.101.64.0/18
        - 108.162.192.0/18
        - 190.93.240.0/20
        - 188.114.96.0/20
        - 197.234.240.0/22
        - 198.41.128.0/17
        - 162.158.0.0/15
        - 104.16.0.0/13
        - 104.24.0.0/14
        - 172.64.0.0/13
        - 131.0.72.0/22
        - 2400:cb00::/32
        - 2606:4700::/32
        - 2803:f800::/32
        - 2405:b500::/32
        - 2405:8100::/32
        - 2a06:98c0::/29
        - 2c0f:f248::/32
        # End of Cloudlare public IP list
    http:
      middlewares:
        - crowdsec-bouncer@file
      redirections:
        entryPoint:
          to: https
          scheme: https
  https:
    address: ":443"
    forwardedHeaders:
      trustedIPs: 
        # Start of Clouflare public IP list for HTTP requests, remove this if you don't use it
        - 173.245.48.0/20
        - 103.21.244.0/22
        - 103.22.200.0/22
        - 103.31.4.0/22
        - 141.101.64.0/18
        - 108.162.192.0/18
        - 190.93.240.0/20
        - 188.114.96.0/20
        - 197.234.240.0/22
        - 198.41.128.0/17
        - 162.158.0.0/15
        - 104.16.0.0/13
        - 104.24.0.0/14
        - 172.64.0.0/13
        - 131.0.72.0/22
        - 2400:cb00::/32
        - 2606:4700::/32
        - 2803:f800::/32
        - 2405:b500::/32
        - 2405:8100::/32
        - 2a06:98c0::/29
        - 2c0f:f248::/32
        # End of Cloudlare public IP list
    http:
      middlewares: 
        - crowdsec-bouncer@file
  http-external:  # changement
    address: ":1181"
    http:
      middlewares:
        - crowdsec-bouncer@file
      redirections:
        entrypoint:
          to: https-external
          scheme: https
  https-external:
    address: ":11444"
    http:
      middlewares:
        - crowdsec-bouncer@file
serversTransport:
  insecureSkipVerify: true
providers:
  docker:
    network: mynet
    endpoint: "tcp://t-docker-socket-proxy:2375"
    exposedByDefault: false
  file:
    filename: /dynamic_conf.yml
    watch: true
certificatesResolvers:
  dns-cloudflare:
    acme:
      email: email
      storage: ./letsencrypt/acme.json
      dnsChallenge:
        provider: cloudflare
      caServer: https://acme-v02.api.letsencrypt.org/directory
        #disablePropagationCheck: true # uncomment this if you have issues pulling certificates through cloudflare, By setting this flag to true disables the need to wait for the propagation of the TXT record to all authoritative name servers.
experimental:
  plugins:
    traefik-get-real-ip:
      moduleName: "github.com/Paxxs/traefik-get-real-ip"
      version: "v1.0.3"
    crowdsec-bouncer-traefik-plugin:
      moduleName: "github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin"
      version: "v1.2.1"
log:
  level: "INFO"
  filePath: "/var/log/crowdsec/traefik.log"
accessLog:
  filePath: "/var/log/crowdsec/access.log"
bufferingSize: 50

ThierryIT · May 2, 2024, 5:49am

I am still working on it, and I have made some changes:

removed the docker portainer
add in the traefik compose file:

portainer:
    image: portainer/portainer-ee:2.20.2
    container_name: portainer
    command: -H unix:///var/run/docker.sock
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - portainer_data:/data
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=mynet"  <<<<<< same as traefik
      - "traefik.http.routers.portainer.rule=Host(`portainer.domain.tld`)"
      - "traefik.http.routers.portainer.entrypoints=https-external"
      - "traefik.http.services.portainer.loadbalancer.server.port=9443"
      - "traefik.http.routers.portainer.service=portainer"
      - "traefik.http.routers.portainer.tls.certresolver=dns-cloudflare"

Nothing has been change in both static and dynamic files from Traefik.
When checking open ports on the server, there is no 9443 port listenning ...

bluepuma77 · May 2, 2024, 7:56am

No, when using Traefik, you don’t want portainer to listen on the node, as that would enable to circumvent any Traefik security middlewares.

Traefik and portainer should both be connected to a Docker network, within all ports are reachable. Only Traefik should listen on external ports.

Check simple Traefik example and other post about portainer.

ThierryIT · May 3, 2024, 3:35pm

Still working on it ....
I had to correct error message from crowdsec and his bouncer and portainer ... No error messages anymore. And I have build a new fresh server too.
But if I try to reach any of my URL (authentik, web server etc,) I have a "forbidden".
I don't know if there is a link, but when checking the trafik logs:

time="2024-05-03T18:21:47+03:00" level=info msg="Starting provider *file.Provider"
time="2024-05-03T18:21:47+03:00" level=info msg="Starting provider *traefik.Provider"
time="2024-05-03T18:21:47+03:00" level=info msg="Starting provider *docker.Provider"
time="2024-05-03T18:21:47+03:00" level=info msg="Starting provider *acme.ChallengeTLSALPN"
time="2024-05-03T18:21:47+03:00" level=info msg="Starting provider *acme.Provider"
time="2024-05-03T18:21:47+03:00" level=info msg="Testing certificate renew..." ACME CA="https://acme-v02.api.letsencrypt.org/directory" providerName=dns-cloudflare.acme
time="2024-05-03T18:21:48+03:00" level=error msg="field not found, node: serviceaportainer" providerName=docker container=portainer-traefik-79013a885bfee61216c93409a2a09ebacd94a1e5c265e12be873d37197460666
time="2024-05-03T18:21:48+03:00" level=warning msg="Defaulting to first available network (&{\"kuma_mydomain\" \"172.26.0.2\" '\\x00' \"\" \"b481f47aaede76202759736d471e0d913fdfe3913d9973c59874ae5c6ca3e29b\"}) for container \"/kuma-uptime-kuma-1\"." serviceName=uptime-kuma container=uptime-kuma-kuma-35a40c0436566e93ad08e00c664b43c51002d027d5816f909ce21dacf92149c7 providerName=docker
time="2024-05-03T18:21:49+03:00" level=warning msg="No domain found in rule HostRegexp(`{subdomain:[a-z0-9-]+}.mydomain`) && PathPrefix(`/outpost.goauthentik.io/`), the TLS options applied for this router will depend on the SNI of each request" entryPointName=https-external routerName=authentik-rtr-outpost@docker
time="2024-05-03T18:21:49+03:00" level=error msg="the router authentik-rtr@docker uses a non-existent resolver: cloudflare"```

No idea of what it says ...

bluepuma77 · May 3, 2024, 4:51pm

The relevant ones are probably with

Topic		Replies	Views
Traefik+docker socket proxy / doesn't seems to work Traefik v2 docker	34	1937	April 8, 2024
504 Gateway Timeout for (only) one of my service Traefik v2 docker	3	1365	January 25, 2023
Traefik gateway timeout for every service Traefik v2 docker-swarm	2	4072	January 8, 2022
Portainer 502 after reboot Traefik v1 docker	1	1039	October 9, 2019
Gateway 504 error with services Traefik v2 docker-swarm	12	5691	May 16, 2022

Portainer docker behind Traefik / URL not working Gateway timeout error 504

Related topics