Getting "gateway timeout" since updating Docker to latest (potential bug?)

Traefik Traefik v2
1

I am author of a detailed manual in the German-speaking area (link to manual) for a Traefik-Crowdsec-Stack and the error had not yet occurred in the associated community.

I have read here the last entries about the error "gateway timeout" however the situations do not fit to mine.

The error has now occurred on several servers independently of each other.

On all servers (Ubnutu 22.04 LTS) I did an update of the current Docker version. Before the problem Docker 24.0.2 was installed. After updating to version 24.0.4, several services on my server fail with the error message "gateway timeout" and are no longer accessible.

My environment:
Ubuntu: 22.04 LTS
Docker: 24.0.2 respectively 24.0.4
Traefik: traefik:2.10
CrowdSec: crowdsec:latest
Bouncer: fbonalair/traefik-crowdsec-bouncer:latest

Various applications via docker-compose.yml which have their own network but are also properly connected to the traefik network and ran without problems until the update.

After updating from Docker 24.0.2 to Docker 24.0.4 the error occurs as already written above and in the log files something like this appears

time="2023-07-10T09:24:42+02:00" level=error msg="Error while dialing backend: dial tcp 172.16.63.254:22: connect: connection timed out"
time="2023-07-10T09:31:13+02:00" level=error msg="Error while dialing backend: dial tcp 172.16.63.254:22: connect: connection timed out"
time="2023-07-10T09:33:59+02:00" level=error msg="Error while dialing backend: dial tcp 172.16.63.254:22: connect: connection timed out"
time="2023-07-10T09:35:11+02:00" level=error msg="Error while dialing backend: dial tcp 172.16.63.254:22: connect: connection timed out"
time="2023-07-10T09:36:26+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73 serviceName=gitlab-ssh providerName=docker
time="2023-07-10T09:36:26+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73 providerName=docker serviceName=gitlab-ce
time="2023-07-10T09:36:26+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.251\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-registry\"." serviceName=gitlab-registry providerName=docker container=gitlab-registry-gitlab-8551d1f2da6eb9f8b92b1fc23f346095db4743c15b37a64bc070085faabe683d
time="2023-07-10T09:36:26+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." providerName=docker container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73 serviceName=gitlab-ssh
time="2023-07-10T09:36:26+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73 serviceName=gitlab-ce providerName=docker
time="2023-07-10T09:36:26+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.251\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-registry\"." providerName=docker container=gitlab-registry-gitlab-8551d1f2da6eb9f8b92b1fc23f346095db4743c15b37a64bc070085faabe683d serviceName=gitlab-registry
time="2023-07-10T09:36:30+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." providerName=docker container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73 serviceName=gitlab-ssh
time="2023-07-10T09:36:30+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." serviceName=gitlab-ce providerName=docker container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73
time="2023-07-10T09:36:30+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.251\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-registry\"." providerName=docker serviceName=gitlab-registry container=gitlab-registry-gitlab-8551d1f2da6eb9f8b92b1fc23f346095db4743c15b37a64bc070085faabe683d
time="2023-07-10T09:36:53+02:00" level=error msg="Error while dialing backend: dial tcp 172.16.63.254:22: connect: connection timed out"
time="2023-07-10T09:37:00+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." providerName=docker container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73 serviceName=gitlab-ssh
time="2023-07-10T09:37:00+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." providerName=docker container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73 serviceName=gitlab-ce
time="2023-07-10T09:37:00+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.251\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-registry\"." serviceName=gitlab-registry providerName=docker container=gitlab-registry-gitlab-8551d1f2da6eb9f8b92b1fc23f346095db4743c15b37a64bc070085faabe683d
time="2023-07-10T09:37:01+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." providerName=docker container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73 serviceName=gitlab-ssh
time="2023-07-10T09:37:01+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." serviceName=gitlab-ce providerName=docker container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73
time="2023-07-10T09:37:01+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.251\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-registry\"." providerName=docker container=gitlab-registry-gitlab-8551d1f2da6eb9f8b92b1fc23f346095db4743c15b37a64bc070085faabe683d serviceName=gitlab-registry
time="2023-07-10T09:38:09+02:00" level=error msg="Error while dialing backend: dial tcp 172.16.63.254:22: connect: connection timed out"
[...]
time="2023-07-10T10:55:36+02:00" level=error msg="Error while dialing backend: dial tcp 172.16.63.254:22: connect: connection timed out"
[...]
time="2023-07-10T12:25:37+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." serviceName=gitlab-ssh providerName=docker container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73
time="2023-07-10T12:25:37+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." providerName=docker container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73 serviceName=gitlab-ce
time="2023-07-10T12:25:37+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.251\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-registry\"." providerName=docker container=gitlab-registry-gitlab-8551d1f2da6eb9f8b92b1fc23f346095db4743c15b37a64bc070085faabe683d serviceName=gitlab-registry
time="2023-07-10T12:25:37+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." providerName=docker serviceName=gitlab-ssh container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73
time="2023-07-10T12:25:37+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." providerName=docker container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73 serviceName=gitlab-ce
time="2023-07-10T12:25:37+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.251\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-registry\"." container=gitlab-registry-gitlab-8551d1f2da6eb9f8b92b1fc23f346095db4743c15b37a64bc070085faabe683d serviceName=gitlab-registry providerName=docker
time="2023-07-10T12:25:38+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." serviceName=gitlab-ssh providerName=docker container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73
time="2023-07-10T12:25:38+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." providerName=docker container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73 serviceName=gitlab-ce
time="2023-07-10T12:25:38+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.251\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-registry\"." providerName=docker container=gitlab-registry-gitlab-8551d1f2da6eb9f8b92b1fc23f346095db4743c15b37a64bc070085faabe683d serviceName=gitlab-registry
time="2023-07-10T12:26:08+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." providerName=docker container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73 serviceName=gitlab-ssh
time="2023-07-10T12:26:08+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." providerName=docker container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73 serviceName=gitlab-ce
time="2023-07-10T12:26:08+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.251\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-registry\"." providerName=docker container=gitlab-registry-gitlab-8551d1f2da6eb9f8b92b1fc23f346095db4743c15b37a64bc070085faabe683d serviceName=gitlab-registry
time="2023-07-10T12:26:08+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." providerName=docker container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73 serviceName=gitlab-ssh
time="2023-07-10T12:26:08+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." providerName=docker container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73 serviceName=gitlab-ce
time="2023-07-10T12:26:08+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.251\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-registry\"." providerName=docker container=gitlab-registry-gitlab-8551d1f2da6eb9f8b92b1fc23f346095db4743c15b37a64bc070085faabe683d serviceName=gitlab-registry
time="2023-07-10T12:26:10+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." providerName=docker container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73 serviceName=gitlab-ssh
time="2023-07-10T12:26:10+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." providerName=docker container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73 serviceName=gitlab-ce
time="2023-07-10T12:26:10+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.251\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-registry\"." serviceName=gitlab-registry container=gitlab-registry-gitlab-8551d1f2da6eb9f8b92b1fc23f346095db4743c15b37a64bc070085faabe683d providerName=docker
time="2023-07-10T12:26:58+02:00" level=error msg="Error while dialing backend: dial tcp 172.16.63.254:22: connect: connection timed out"
time="2023-07-10T12:27:00+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." providerName=docker container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73 serviceName=gitlab-ssh
time="2023-07-10T12:27:00+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." providerName=docker container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73 serviceName=gitlab-ce
time="2023-07-10T12:27:00+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.251\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-registry\"." providerName=docker container=gitlab-registry-gitlab-8551d1f2da6eb9f8b92b1fc23f346095db4743c15b37a64bc070085faabe683d serviceName=gitlab-registry
time="2023-07-10T12:27:01+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73 serviceName=gitlab-ssh providerName=docker
time="2023-07-10T12:27:01+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." serviceName=gitlab-ce providerName=docker container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73
time="2023-07-10T12:27:01+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.251\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-registry\"." providerName=docker container=gitlab-registry-gitlab-8551d1f2da6eb9f8b92b1fc23f346095db4743c15b37a64bc070085faabe683d serviceName=gitlab-registry
time="2023-07-10T12:27:31+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." providerName=docker serviceName=gitlab-ssh container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73
time="2023-07-10T12:27:31+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." providerName=docker container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73 serviceName=gitlab-ce
time="2023-07-10T12:27:31+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.251\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-registry\"." container=gitlab-registry-gitlab-8551d1f2da6eb9f8b92b1fc23f346095db4743c15b37a64bc070085faabe683d serviceName=gitlab-registry providerName=docker
time="2023-07-10T12:27:32+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." serviceName=gitlab-ssh providerName=docker container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73
time="2023-07-10T12:27:32+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." providerName=docker container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73 serviceName=gitlab-ce
time="2023-07-10T12:27:32+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.251\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-registry\"." container=gitlab-registry-gitlab-8551d1f2da6eb9f8b92b1fc23f346095db4743c15b37a64bc070085faabe683d serviceName=gitlab-registry providerName=docker
time="2023-07-10T12:31:21+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." providerName=docker container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73 serviceName=gitlab-ssh
time="2023-07-10T12:31:21+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." serviceName=gitlab-ce providerName=docker container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73
time="2023-07-10T12:31:21+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.251\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-registry\"." serviceName=gitlab-registry providerName=docker container=gitlab-registry-gitlab-8551d1f2da6eb9f8b92b1fc23f346095db4743c15b37a64bc070085faabe683d
time="2023-07-10T12:31:21+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." providerName=docker container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73 serviceName=gitlab-ssh
time="2023-07-10T12:31:21+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." serviceName=gitlab-ce providerName=docker container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73
time="2023-07-10T12:31:21+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.251\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-registry\"." serviceName=gitlab-registry container=gitlab-registry-gitlab-8551d1f2da6eb9f8b92b1fc23f346095db4743c15b37a64bc070085faabe683d providerName=docker
time="2023-07-10T12:31:26+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73 serviceName=gitlab-ssh providerName=docker
time="2023-07-10T12:31:26+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." serviceName=gitlab-ce providerName=docker container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73
time="2023-07-10T12:31:26+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.251\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-registry\"." container=gitlab-registry-gitlab-8551d1f2da6eb9f8b92b1fc23f346095db4743c15b37a64bc070085faabe683d serviceName=gitlab-registry providerName=docker
time="2023-07-10T12:31:56+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." serviceName=gitlab-ssh providerName=docker container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73
time="2023-07-10T12:31:56+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73 serviceName=gitlab-ce providerName=docker
time="2023-07-10T12:31:56+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.251\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-registry\"." providerName=docker container=gitlab-registry-gitlab-8551d1f2da6eb9f8b92b1fc23f346095db4743c15b37a64bc070085faabe683d serviceName=gitlab-registry
time="2023-07-10T12:31:57+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." serviceName=gitlab-ssh providerName=docker container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73
time="2023-07-10T12:31:57+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.254\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-ce\"." providerName=docker container=gitlab-ce-gitlab-b4cc4eb40ebb02a3be3b5d77f9884c8e5884dfbf9435772365f71d5f95790e73 serviceName=gitlab-ce
time="2023-07-10T12:31:57+02:00" level=warning msg="Defaulting to first available network (&{\"gitlab\" \"172.16.63.251\" '\\x00' \"\" \"fd87fdc77397f9f44a4057939d8ba1c60ca5eb9615e54b8d0a4bc02533a82e0c\"}) for container \"/gitlab-registry\"." providerName=docker container=gitlab-registry-gitlab-8551d1f2da6eb9f8b92b1fc23f346095db4743c15b37a64bc070085faabe683d serviceName=gitlab-registry

I have tried different approaches. What is noticeable is that after downgrading to Docker 24.0.2 everything runs as it should again - at least until now.

I would like to comply with the request of the developers and ask here first before I open a bug-ticket. Unfortunately I don't know or can't differentiate if the problem is with Docker or with Traefik. Does anyone have another idea?

1 Like
2

You want us to read your 100 pages of manual and have us put all the little pieces together? :laughing:

(But I really like that you eyplain every single step and why it’s done that way :clap:)

Don’t you have a simple Traefik example (including complete files) with static and dynamic config, and docker-compose.yml? :slight_smile:

Gateway timeout is usually an issue with the Docker network:

  1. target service is connected to multiple networks and provider.docker.network is not set or incorrect
  2. Docker network is established within docker-compose.yml and name is not set, then the name is automatically extended by the project name and doesn’t match anymore
  3. Docker overlay network is used and the MTU is not adapted to a smaller value that fits inside a VLAN used

PS: Why do you use a plugin to "get the real IP"? HTTP provides it in header "x-real-ip" and for TCP you could use standard ProxyProtocol which is supported by Traefik on entrypoint and service.

2 Likes
3

Thank you very much for your answer. You are right of course it makes it easier if I write the configuration here :smiley:

Here is my setup GitHub - Psycho0verload/traefik-crowdsec-stack
The only difference to the manual is that IPv6 is still included.

In my traefik.yml the network is defined:

And here is the configuration I use in the docker-compose.yml. At the top of the log it was Gitlab. So here is the configuration of Gitlab:

version: "3.9"
services:
  gitlab-ce:
    configs:
      - source: gitlab
        target: /omnibus_config.rb
    container_name: ${SERVICE_GITLAB_CE_CONTAINER_NAME:-gitlab-ce}
    depends_on:
      gitlab-postgres:
        condition: service_healthy
    environment:
      GITLAB_OMNIBUS_CONFIG: "from_file('/omnibus_config.rb')"
    hostname: ${SERVICE_GITLAB_CE_HOSTNAME}
    image: gitlab/gitlab-ce:${SERVICE_GITLAB_CE_VERSION:-latest}
    labels:
      traefik.docker.network: proxy
      traefik.enable: "true"
      traefik.http.routers.gitlab-ce.entrypoints: websecure
      traefik.http.routers.gitlab-ce.rule: Host(${SERVICE_GITLAB_CE_TRAEFIK_HOST})
      traefik.http.routers.gitlab-ce.service: gitlab-ce
      traefik.http.routers.gitlab-ce.tls: "true"
      traefik.http.routers.gitlab-ce.tls.certresolver: http_resolver
      traefik.http.services.gitlab-ce.loadbalancer.server.port: "80"
      traefik.tcp.routers.gitlab-ssh.entrypoints: git-ssh
      traefik.tcp.routers.gitlab-ssh.rule: HostSNI(`*`)
      traefik.tcp.routers.gitlab-ssh.service: gitlab-ssh
      traefik.tcp.services.gitlab-ssh.loadbalancer.server.port: "22"
    networks:
      gitlab:
        ipv4_address: ${SERVICE_GITLAB_CE_NETWORK_IPV4}
        ipv6_address: ${SERVICE_GITLAB_CE_NETWORK_IPV6}
      proxy: null
    restart: unless-stopped
    secrets:
    - gitlab_root_password
    - postgres_password
    shm_size: '256m'
    volumes:
    - ./data/config:/etc/gitlab
    - ./data/data:/var/opt/gitlab
#   - ./data/backup:/opt/gitlab/embedded/bin/gitaly-backup
    - /var/log/gitlab:/var/log/gitlab

  gitlab-postgres:
    container_name: ${SERVICE_GITLAB_DATABASE_CONTAINER_NAME:-gitlab-database}
    environment:
      DEBUG: "false"
      POSTGRES_USER: gitlab
      POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
      POSTGRES_DB: gitlab
    hostname: ${SERVICE_GITLAB_DATABASE_HOSTNAME:-gitlab-database}
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}"]
      start_period: 20s
      interval: 30s
      retries: 5
      timeout: 5s
    image: postgres:${SERVICE_GITLAB_DATABASE_VERSION:-15}
    networks:
      gitlab:
        ipv4_address: ${SERVICE_GITLAB_DATABASE_NETWORK_IPV4}
        ipv6_address: ${SERVICE_GITLAB_DATABASE_NETWORK_IPV6}
    restart: unless-stopped
    secrets:
    - postgres_password
    volumes:
      - ./data/database:/var/lib/postgresql/data

  gitlab-registry:
    container_name: ${SERVICE_GITLAB_REGISTRY_CONTAINER_NAME:-gitlab-registry}
    environment:
      REGISTRY_AUTH_TOKEN_ISSUER: "gitlab-issuer"
      REGISTRY_AUTH_TOKEN_REALM: https://git.domain.de/jwt/auth
      REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE: /certs/gitlab-registry.crt
      REGISTRY_AUTH_TOKEN_SERVICE: container_registry
      REGISTRY_LOG_LEVEL: debug
      REGISTRY_STORAGE_DELETE_ENABLED: "true"
      REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /registry
    hostname: ${SERVICE_GITLAB_REGISTRY_HOSTNAMENAME:-gitlab-registry}
    image: registry:latest
    labels:
      traefik.docker.network: proxy
      traefik.enable: "true"
      traefik.http.routers.gitlab-registry.entrypoints: websecure
      traefik.http.routers.gitlab-registry.rule: Host(`registry.domain.de`)
      traefik.http.routers.gitlab-registry.service: gitlab-registry
      traefik.http.routers.gitlab-registry.tls: "true"
      traefik.http.routers.gitlab-registry.tls.certresolver: http_resolver
      traefik.http.services.gitlab-registry.loadbalancer.server.port: "5000"
    networks:
      gitlab:
        ipv4_address: ${SERVICE_GITLAB_REGISTRY_NETWORK_IPV4}
        ipv6_address: ${SERVICE_GITLAB_REGISTRY_NETWORK_IPV6}
      proxy: null
    restart: unless-stopped
    volumes:
      - ./data/registry/data:/registry
      - ./data/registry/certs:/certs

networks:
  proxy:
    external: true
  gitlab:
    name: ${NETWORK_GITLAB_NAME}
    driver: bridge
    ipam:
      config:
      - subnet: ${NETWORK_GITLAB_IPV4_SUBNET}
      - subnet: ${NETWORK_GITLAB_IPV6_SUBNET}
    attachable: true
    enable_ipv6: true
secrets:
  gitlab_root_password:
    file: ${SECRETSDIR}/gitlab/root
  postgres_password:
    file: ${SECRETSDIR}/postgres/gitlab/password

configs:
  gitlab:
    file: ./gitlab.rb

With the configuration also no network is created which I have not defined

NETWORK ID     NAME                                        DRIVER    SCOPE
64bc500e4b5d   bridge                                      bridge    local
7800d82b9e1e   crowdsec                                    bridge    local
713e1539ec48   gitlab                                      bridge    local
00d272097fa4   host                                        host      local
29e8b564e5c3   none                                        null      local
2c6bba012786   proxy                                       bridge    local
4

I just tested an even simpler Traefik example with a new Hetzner VM running Ubuntu latest (using apt update && apt -y upgrade), Docker latest (using curl -fsSL https://get.docker.com | sh -) and Traefik latest (v2.10.3).

# docker-compose.yml
version: '3'

services:
  traefik:
    image: traefik:v2.10.3
    ports:
      - 80:80
    networks:
      - proxy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    command:
      - --log.level=INFO
      - --accesslog=true
      - --providers.docker.network=proxy
      - --providers.docker.exposedByDefault=false
      - --entrypoints.web.address=:80

  whoami:
    image: traefik/whoami:v1.9
    networks:
      - proxy
    labels:
      - traefik.enable=true
      - traefik.http.routers.mywhoami.rule=Method(`GET`)
      - traefik.http.services.mywhoami.loadbalancer.server.port=80

networks:
  proxy:
    name: proxy

Works for me using curl http://<IP> and curl http://<IP>/data?size=100000.

5

Your setup does not match the problem I described. It currently boils down to me that there are problems with it when you use multiple networks

6

You are right, I added some dummy networks:

docker-compose.yml
version: '3'

services:
  traefik:
    image: traefik:v2.10.3
    ports:
      - 80:80
    networks:
      - dummy1
      - proxy
      - dummy3
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    command:
      - --api.dashboard=true
      - --log.level=DEBUG
      - --accesslog=true
      - --providers.docker.network=proxy
      - --providers.docker.exposedByDefault=false
      - --entrypoints.web.address=:80

  whoami:
    image: traefik/whoami:v1.9
    networks:
      - dummy2
      - proxy
      - dummy4
    labels:
      - traefik.enable=true
      - traefik.http.routers.mywhoami.rule=Method(`GET`)
      - traefik.http.services.mywhoami.loadbalancer.server.port=80

networks:
  dummy1:
  dummy2:
  proxy:
    name: proxy
  dummy3:
  dummy4:

Here comes the challenge: I saw the Gateway Timeout only once, have never seen it before. But never again after multiple restarts. I sadly didn't try again the first time, because I wanted to enable debug.

docker compose up
[+] Running 7/5
 ✔ Network root_dummy1       Created                                                                                                                                                0.1s
 ✔ Network proxy             Created                                                                                                                                                0.1s
 ✔ Network root_dummy3       Created                                                                                                                                                0.1s
 ✔ Network root_dummy2       Created                                                                                                                                                0.1s
 ✔ Network root_dummy4       Created                                                                                                                                                0.1s
 ✔ Container root-whoami-1   Created                                                                                                                                                0.1s
 ✔ Container root-traefik-1  Created                                                                                                                                                0.1s
Attaching to root-traefik-1, root-whoami-1
root-whoami-1   | 2023/07/11 07:26:25 Starting up on port 80
root-traefik-1  | time="2023-07-11T07:26:25Z" level=info msg="Configuration loaded from flags."
root-traefik-1  | time="2023-07-11T07:26:25Z" level=info msg="Traefik version 2.10.3 built on 2023-06-19T16:18:54Z"
root-traefik-1  | time="2023-07-11T07:26:25Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\n"
root-traefik-1  | time="2023-07-11T07:26:25Z" level=info msg="Starting provider aggregator aggregator.ProviderAggregator"
root-traefik-1  | time="2023-07-11T07:26:25Z" level=info msg="Starting provider *traefik.Provider"
root-traefik-1  | time="2023-07-11T07:26:26Z" level=info msg="Starting provider *docker.Provider"
root-traefik-1  | time="2023-07-11T07:26:26Z" level=info msg="Starting provider *acme.ChallengeTLSALPN"
root-traefik-1  | 1.2.3.4 - - [11/Jul/2023:07:26:31 +0000] "GET / HTTP/1.1" 504 15 "-" "-" 1 "mywhoami@docker" "http://172.20.0.2:80" 30002ms
1 Like
7

Tried again on a new server, could not reproduce. So if it's a bug it's gonna be hard to find.

1 Like
8

Thank you very much for your effort.
I have tried again on one of my servers. I wanted to exclude that you always have the traefik network in the first place. Since then with the update but also no problems.

Again yesterday the whole server was paralyzed

9

I can no longer reconstruct it. Meanwhile, the update is on it and everything works as it should

10

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.