Traefik route to portainer does not work reliable

Hi,

I am just experimenting and learning with docker swarm and wanted to add traefik to the mix today.
I've already worked with traefik a bit in my normal docker-compose environment.

But now I am stuck with getting portainer to work with traefik in my docker swarm. I want to connect to portainer via portainer.example.at and I've already got it working 3 times but every time after a few minutes (max 5minutes) I get the Error "Gateway Timeout". I did not change anything before I got this error.

Connecting to portainer directly via the port 9000 works every time.

The route to portainer.example.at shows up in the traefik dashboard.
Traefik log:
10.0.0.2 - - [12/Mar/2023:16:29:30 +0000] "GET /favicon.ico HTTP/2.0" 504 15 "-" "-" 1868 "portainer@docker" "http://10.0.0.220:9000" 30000ms

docker inspect:
[
    {
        "Id": "bd55fbf8aa60680eb7ddf5a92e726c4b0028820f0dea23de3dd9c6840571fc8c",
        "Created": "2023-03-12T15:52:43.081835497Z",
        "Path": "/portainer",
        "Args": [
            "-H",
            "tcp://tasks.agent:9001",
            "--tlsskipverify"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 70783,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2023-03-12T15:52:47.990864182Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:a87d51c7a32408ba63e65888d023ad7d4328fd575d8a4c8bd8cffa495fcd055c",
        "ResolvConfPath": "/var/lib/docker/containers/bd55fbf8aa60680eb7ddf5a92e726c4b0028820f0dea23de3dd9c6840571fc8c/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/bd55fbf8aa60680eb7ddf5a92e726c4b0028820f0dea23de3dd9c6840571fc8c/hostname",
        "HostsPath": "/var/lib/docker/containers/bd55fbf8aa60680eb7ddf5a92e726c4b0028820f0dea23de3dd9c6840571fc8c/hosts",
        "LogPath": "/var/lib/docker/containers/bd55fbf8aa60680eb7ddf5a92e726c4b0028820f0dea23de3dd9c6840571fc8c/bd55fbf8aa60680eb7ddf5a92e726c4b0028820f0dea23de3dd9c6840571fc8c-json.log",
        "Name": "/test-stack_portainer.1.62bs6nizzycq0maq043odwd38",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "docker-default",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "default",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "ConsoleSize": [
                0,
                0
            ],
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "private",
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "Isolation": "default",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": null,
            "PidsLimit": null,
            "Ulimits": [],
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "Mounts": [
                {
                    "Type": "volume",
                    "Source": "test-stack_portainer_data",
                    "Target": "/data",
                    "VolumeOptions": {
                        "Labels": {
                            "com.docker.stack.namespace": "test-stack"
                        }
                    }
                }
            ],
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/1115220be6e32c9688fe5998056dc0ff53996d70bbed37be5ccd7e7f9faedf4e-init/diff:/var/lib/docker/overlay2/2d9d5e9b5de2a3ff5113c238bd3b00cb152722ec1fb66aef85e82b2850e2b60d/diff:/var/lib/docker/overlay2/fe814f7f1c1beec5d4ed1535a9cb19398c68363c0eed22baa9f36c33f6990d3d/diff:/var/lib/docker/overlay2/ef03fac10f6853e0b0bc578da7f6076ce91b5ee1251d99e4474454bfb09b06d3/diff:/var/lib/docker/overlay2/6f51e84a18019ad18fe5032dab806bb21022744ea0dc1a368bc235409615a0ec/diff",
                "MergedDir": "/var/lib/docker/overlay2/1115220be6e32c9688fe5998056dc0ff53996d70bbed37be5ccd7e7f9faedf4e/merged",
                "UpperDir": "/var/lib/docker/overlay2/1115220be6e32c9688fe5998056dc0ff53996d70bbed37be5ccd7e7f9faedf4e/diff",
                "WorkDir": "/var/lib/docker/overlay2/1115220be6e32c9688fe5998056dc0ff53996d70bbed37be5ccd7e7f9faedf4e/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "volume",
                "Name": "test-stack_portainer_data",
                "Source": "/var/lib/docker/volumes/test-stack_portainer_data/_data",
                "Destination": "/data",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            }
        ],
        "Config": {
            "Hostname": "bd55fbf8aa60",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "8000/tcp": {},
                "9000/tcp": {},
                "9443/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "-H",
                "tcp://tasks.agent:9001",
                "--tlsskipverify"
            ],
            "Image": "portainer/portainer-ce:2.17.1@sha256:9fa1ec78b4e29d83593cf9720674b72829c9cdc0db7083a962bc30e64e27f64e",
            "Volumes": {
                "/data": {}
            },
            "WorkingDir": "/",
            "Entrypoint": [
                "/portainer"
            ],
            "OnBuild": null,
            "Labels": {
                "com.docker.desktop.extension.api.version": ">= 0.2.2",
                "com.docker.desktop.extension.icon": "https://portainer-io-assets.sfo2.cdn.digitaloceanspaces.com/logos/portainer.png",
                "com.docker.extension.additional-urls": "[{\"title\":\"Website\",\"url\":\"https://www.portainer.io?utm_campaign=DockerCon&utm_source=DockerDesktop\"},{\"title\":\"Documentation\",\"url\":\"https://docs.portainer.io\"},{\"title\":\"Support\",\"url\":\"https://join.slack.com/t/portainer/shared_invite/zt-txh3ljab-52QHTyjCqbe5RibC2lcjKA\"}]",
                "com.docker.extension.detailed-description": "<p data-renderer-start-pos=\"226\">Portainer&rsquo;s Docker Desktop extension gives you access to all of Portainer&rsquo;s rich management functionality within your docker desktop experience.</p><h2 data-renderer-start-pos=\"374\">With Portainer you can:</h2><ul><li>See all your running containers</li><li>Easily view all of your container logs</li><li>Console into containers</li><li>Easily deploy your code into containers using a simple form</li><li>Turn your YAML into custom templates for easy reuse</li></ul><h2 data-renderer-start-pos=\"660\">About Portainer&nbsp;</h2><p data-renderer-start-pos=\"680\">Portainer is the worlds&rsquo; most popular universal container management platform with more than 650,000 active monthly users. Portainer can be used to manage Docker Standalone, Kubernetes, Docker Swarm and Nomad environments through a single common interface. It includes a simple GitOps automation engine and a Kube API.&nbsp;</p><p data-renderer-start-pos=\"1006\">Portainer Business Edition is our fully supported commercial grade product for business-wide use. It includes all the functionality that businesses need to manage containers at scale. Visit <a class=\"sc-jKJlTe dPfAtb\" href=\"http://portainer.io/\" title=\"http://Portainer.io\" data-renderer-mark=\"true\">Portainer.io</a> to learn more about Portainer Business and <a class=\"sc-jKJlTe dPfAtb\" href=\"http://portainer.io/take5?utm_campaign=DockerCon&amp;utm_source=Docker%20Desktop\" title=\"http://portainer.io/take5?utm_campaign=DockerCon&amp;utm_source=Docker%20Desktop\" data-renderer-mark=\"true\">get 5 free nodes.</a></p>",
                "com.docker.extension.publisher-url": "https://www.portainer.io",
                "com.docker.extension.screenshots": "[{\"alt\": \"screenshot one\", \"url\": \"https://portainer-io-assets.sfo2.digitaloceanspaces.com/screenshots/docker-extension-1.png\"},{\"alt\": \"screenshot two\", \"url\": \"https://portainer-io-assets.sfo2.digitaloceanspaces.com/screenshots/docker-extension-2.png\"},{\"alt\": \"screenshot three\", \"url\": \"https://portainer-io-assets.sfo2.digitaloceanspaces.com/screenshots/docker-extension-3.png\"},{\"alt\": \"screenshot four\", \"url\": \"https://portainer-io-assets.sfo2.digitaloceanspaces.com/screenshots/docker-extension-4.png\"},{\"alt\": \"screenshot five\", \"url\": \"https://portainer-io-assets.sfo2.digitaloceanspaces.com/screenshots/docker-extension-5.png\"},{\"alt\": \"screenshot six\", \"url\": \"https://portainer-io-assets.sfo2.digitaloceanspaces.com/screenshots/docker-extension-6.png\"},{\"alt\": \"screenshot seven\", \"url\": \"https://portainer-io-assets.sfo2.digitaloceanspaces.com/screenshots/docker-extension-7.png\"},{\"alt\": \"screenshot eight\", \"url\": \"https://portainer-io-assets.sfo2.digitaloceanspaces.com/screenshots/docker-extension-8.png\"},{\"alt\": \"screenshot nine\", \"url\": \"https://portainer-io-assets.sfo2.digitaloceanspaces.com/screenshots/docker-extension-9.png\"}]",
                "com.docker.stack.namespace": "test-stack",
                "com.docker.swarm.node.id": "fqqd3suoh51nqcyvu10tuv2ld",
                "com.docker.swarm.service.id": "w0kr2507oncmbrfy15x6dt9s1",
                "com.docker.swarm.service.name": "test-stack_portainer",
                "com.docker.swarm.task": "",
                "com.docker.swarm.task.id": "62bs6nizzycq0maq043odwd38",
                "com.docker.swarm.task.name": "test-stack_portainer.1.62bs6nizzycq0maq043odwd38",
                "io.portainer.server": "true",
                "org.opencontainers.image.description": "Docker container management made simple, with the world’s most popular GUI-based container management platform.",
                "org.opencontainers.image.title": "Portainer",
                "org.opencontainers.image.vendor": "Portainer.io"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "030ddf113abf453b9253eb3e226859b5dd47d2ba654e8a0bcef1eff6a1a144c2",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "8000/tcp": null,
                "9000/tcp": null,
                "9443/tcp": null
            },
            "SandboxKey": "/var/run/docker/netns/030ddf113abf",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "ingress": {
                    "IPAMConfig": {
                        "IPv4Address": "10.0.0.220"
                    },
                    "Links": null,
                    "Aliases": [
                        "bd55fbf8aa60"
                    ],
                    "NetworkID": "kb4q7ij85hodesad0hzjwk2ib",
                    "EndpointID": "d29d54e2a01a5611a58e7012dba30a465b99d0af242ee2bbee3084fcaaf78098",
                    "Gateway": "",
                    "IPAddress": "10.0.0.220",
                    "IPPrefixLen": 24,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:0a:00:00:dc",
                    "DriverOpts": null
                },
                "test-stack_agent_network": {
                    "IPAMConfig": {
                        "IPv4Address": "10.0.7.16"
                    },
                    "Links": null,
                    "Aliases": [
                        "bd55fbf8aa60"
                    ],
                    "NetworkID": "uguxy488aea4gc5wm3p3dd0ij",
                    "EndpointID": "6873ba3660f0c11ecf96133320c013a7deac995c48cb806971d5fbb4a00278ae",
                    "Gateway": "",
                    "IPAddress": "10.0.7.16",
                    "IPPrefixLen": 24,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:0a:00:07:10",
                    "DriverOpts": null
                },
                "test-stack_traefik-public": {
                    "IPAMConfig": {
                        "IPv4Address": "10.0.6.22"
                    },
                    "Links": null,
                    "Aliases": [
                        "bd55fbf8aa60"
                    ],
                    "NetworkID": "swo9decttvw9ti07a7owtujxq",
                    "EndpointID": "6f82edc239a0d83e4e285c21ddc6f50542e4d84f814a86fac9c4243f2030fea6",
                    "Gateway": "",
                    "IPAddress": "10.0.6.22",
                    "IPPrefixLen": 24,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:0a:00:06:16",
                    "DriverOpts": null
                }
            }
        }
    }
]
docker stack compose file
version: '3.9'

networks:
  traefik-public:
    driver: overlay
    attachable: true
  agent_network:
    driver: overlay
    attachable: true

volumes:
  traefik-public-certificates:
  portainer_data:

services:
  traefik:
    image: "traefik:v2.9.8"
    command:
      - --providers.docker                                                                      # Enable Docker in Traefik, so that it reads labels from Docker services
      - --providers.docker.constraints=Label(`traefik.constraint-label`, `traefik-public`)      # Add a constraint to only use services with the label "traefik.constraint-label=traefik-public"
      - --providers.docker.exposedbydefault=false                                               # Do not expose all Docker services, only the ones explicitly exposed
      - --providers.docker.swarmmode                                                            # Enable Docker Swarm mode
      - --entrypoints.http.address=:80                                                          # Create an entrypoint "http" listening on port 80
      - --entrypoints.https.address=:443                                                        # Create an entrypoint "https" listening on port 443
      - --entrypoints.http.http.redirections.entryPoint.to=https
      - --entrypoints.http.http.redirections.entrypoint.scheme=https
      - --certificatesresolvers.letsencrypt.acme.email=${EMAIL?Variable not set}                # Create the certificate resolver "le" for Let's Encrypt, uses the environment variable EMAIL
      - --certificatesresolvers.letsencrypt.acme.storage=/certificates/acme.json                # Store the Let's Encrypt certificates in the mounted volume
      - --certificatesresolvers.letsencrypt.acme.tlschallenge=true                              # Use the TLS Challenge for Let's Encrypt
      - --accesslog                                                                             # Enable the access log, with HTTP requests
      - --log                                                                                   # Enable the Traefik log, for configurations and errors          
      - --api.insecure=true
    ports:
      - "8080:8080"
      - "80:80"
      - "443:443"
    networks:
      - traefik-public
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro         # Add Docker as a mounted volume, so that Traefik can read the labels of other services
      - traefik-public-certificates:/certificates            # Mount the volume to store the certificates
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints:
          - node.role == manager                                              # Constraint to only run on a manager node
          - node.labels.traefik-public.traefik-public-certificates == true    # Constraint to only run on a node with this label
  whoami0:
    image: traefik/whoami
    networks:
      - traefik-public
    deploy:
      mode: replicated
      replicas: 1
      labels:
        - "traefik.enable=true"
        - "traefik.constraint-label=traefik-public"
        - "traefik.http.services.whoami0.loadbalancer.server.port=80"
        - "traefik.http.routers.whoami0.rule=Host(`whoami.example.at`)"
        - "traefik.http.routers.whoami0.tls=true"
        - "traefik.http.routers.whoami0.entrypoints=http,https"
        - "traefik.http.routers.whoami0.tls.certresolver=letsencrypt"
  portainer:
    image: portainer/portainer-ce:2.17.1
    command: -H tcp://tasks.agent:9001 --tlsskipverify
    ports:
      - "8000:8000"
      - "9443:9443"
      - "9000:9000"    
    networks:
      - traefik-public
      - agent_network
    volumes:
      - portainer_data:/data
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints:
          - node.role == manager
          - node.labels.traefik-public.traefik-public-certificates == true
      labels:
        - "traefik.enable=true"
        - "traefik.constraint-label=traefik-public"
        - "traefik.http.services.portainer.loadbalancer.server.port=9000"
        - "traefik.http.routers.portainer.rule=Host(`portainer.example.at`)"
        - "traefik.http.routers.portainer.tls=true"
        - "traefik.http.routers.portainer.entrypoints=http,https"
        - "traefik.http.routers.portainer.tls.certresolver=letsencrypt"
        - "traefik.port=9000"
  agent:
    image: portainer/agent:2.17.1
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /var/lib/docker/volumes:/var/lib/docker/volumes
    networks:
      #- traefik-public
      - agent_network
    deploy:
      mode: global
      placement:
        constraints:
          - node.platform.os == linux

Your target service is connected to multiple networks, you need to tell Traefik which network to use, for example in the static config:

--providers.docker.network=traefik-public

Make sure you give your network a fixed name, which is not changed by compose:

networks:
  traefik-public:
    driver: overlay
    attachable: true
    name: traefik-public

Hi @bluepuma77, thanks for the fast answer!

I've just tried it and I get the same phenomenon as before, it did not work at first then like 5 minutes after starting the stack it worked for 1 minute, and then I got the "Gateway Timeout" error again.

I've set a fix name for the network in the compose file and I've added

--providers.docker.network=traefik-public

to the command section of the traefik container.

I hope I did not mess up something here ^^

EDIT:
I think I got it working, just added
- "traefik.docker.network=traefik-public"
to the portainer service and that did the trick, :slight_smile: thanks for your help!