Gateway timeout for one file-provider

I use traefik on OMV and have different container and file-provider, which works well. But one file-provider leads to "Gateway Timeout". The traefik.log says:

time="2024-04-22T12:01:50+02:00" level=debug msg="'504 Gateway Timeout' caused by: dial tcp 192.168.105.161:80: i/o timeout"

Here is my compose-file:

services:
  traefik:
    container_name: traefik
    image: traefik:latest
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /srv/dev-disk-by-uuid-xxx/appdata/traefik/data/traefik.yml:/traefik.yml:ro
      - /srv/dev-disk-by-uuid-xxx/appdata/traefik/data/acme_letsencrypt.json:/acme_letsencrypt.json
      - /srv/dev-disk-by-uuid-xxx/appdata/traefik/data/conf:/conf
      - /srv/dev-disk-by-uuid-xxx/appdata/traefik/logs:/logs
    labels:
      - "com.centurylinklabs.watchtower.enable=true"
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.entrypoints=https"
      - "traefik.http.routers.traefik.rule=Host(`traefik.xxxx.de`)"
      - "traefik.http.routers.traefik.middlewares=traefikAuth@file,default@file"
      - "traefik.http.routers.traefik.tls=true"
      - "traefik.http.routers.traefik.tls.certresolver=http"
      - "traefik.http.routers.traefik.service=api@internal"
      - "traefik.http.services.traefik.loadbalancer.server.port=80"
      - "traefik.http.services.traefik.loadbalancer.sticky.cookie.httpOnly=true"
      - "traefik.http.services.traefik.loadbalancer.sticky.cookie.secure=true"
      - "traefik.docker.network=proxy" 
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    networks:
      proxy:
    hostname: traefik
    ports:
      - "880:80"
      - "8083:8080"
      - "443:443/tcp"
      - "443:443/udp"

Here my traefik.yml:

api:
  dashboard: true
experimental:
  http3: true
certificatesResolvers:
  http:
    acme:
      email: "xx@xxx.de"                 
      storage: "acme_letsencrypt.json"       
      httpChallenge:
        entryPoint: http
entryPoints:
  http:
    address: ":80" 
    http:
      redirections: 
        entryPoint:
          to: "https" 
          scheme: "https"
  https:
    address: ":443" 
    http:
    http3:
      advertisedPort: "443"
global:
  checknewversion: true 
  sendanonymoususage: false 
providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
    network: "proxy"
  file:
    directory: "/conf" 
    watch: true
  providersThrottleDuration: 10
log:
  level: "DEBUG"
  filePath: "/logs/traefik.log"
accessLog:
  filePath: "/logs/access.log"

Here the conf/dynamic_conf.yml:

tls:
  options:
    default:
      minVersion: VersionTLS12
      cipherSuites:
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
        - TLS_AES_128_GCM_SHA256
        - TLS_AES_256_GCM_SHA384
        - TLS_CHACHA20_POLY1305_SHA256
      curvePreferences:
        - CurveP521
        - CurveP384
      sniStrict: true
http:
  middlewares:
    traefikAuth:
      basicAuth:
        users:
          - "xxx@xxx"
    default:
      chain:
        middlewares:
          - default-security-headers
          - gzip
    secHeaders:
      chain:
        middlewares:
          - default-security-headers
          - gzip
    default-security-headers:
      headers:
        browserXssFilter: true
        contentTypeNosniff: true
        forceSTSHeader: true
        frameDeny: true
        stsIncludeSubdomains: true
        stsPreload: true
        stsSeconds: 31536000
        customFrameOptionsValue: "SAMEORIGIN"
    gzip:
      compress: {}

And the conf/vuduo2.yml, the configuration for the file-provider:

http:
  routers:
    vuduo2:
      entryPoints:
        - "https"
        - "http"
      rule: "Host (`vuduo2.xxx.de`)" 
      middlewares:
        - https-redirect
      tls:
        certResolver: http
      service: vuduo2

  services:
    vuduo2:
      loadBalancer:
        servers:
          - url: "http://192.168.105.161:80"
        passHostHeader: true

  middlewares:
    https-redirect:
      redirectScheme:
        scheme: https

Has anyone an idea, where the problem is? It worked at first configuration, but after updates in OMV it does not. I have a similar file-configuration for pi-hole on a raspberrypi with a different IP, which still works.

Thanks in advance

Matthias

It seems you use Traefik in Docker with Docker network on an OpenMediaVault server.

Are you sure the target service IP is still correct? Networking in Traefik container and on server are correct? Seems like a gateway IP is wrong.

My network topology is:
Main router (Fritzbox): 192.168.178.1
NAS with OMV: 192.168.178.57
Docker network proxy: 172.19.0.x
openwrt-router for managing VLANs: 192.168.178.64 -> 192.168.1.1
VLAN-Network 105: 192.168.105.1/24

There are static routes in main-router:
192.168.1.0 -> 192.168.178.64
192.168.105.0 -> 192.168.178.64

From any computer in the main network 192.168.178.x the target ip 192.168.105.161 is reachable. And is was also reachable via traefik at first. But after updates it will not work anymore.

Matthias

Try docker network inspect proxy.

Here is the result of "docker network inspect proxy":

[
    {
        "Name": "proxy",
        "Id": "cb7e869c11a533dac8de3e34e38c2c44cb3737c7061562293b490ff319368bc1",
        "Created": "2023-05-08T15:20:03.294770832+02:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.19.0.0/16",
                    "Gateway": "172.19.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "0a73b50e212187d4a221e82dcbd8e1065d9003f2321a3a1fb6d676fa615b59d3": {
                "Name": "php8-fpm",
                "EndpointID": "4ec22b3811d16f3dc24fdb4f08c63976df0d3b008384563845b3fa2030694d89",
                "MacAddress": "02:42:ac:13:00:0a",
                "IPv4Address": "172.19.0.10/16",
                "IPv6Address": ""
            },
            "1da0ae05e4611297bbf5cfbfa257175519e561890c30159cfa3fd201e7c238c0": {
                "Name": "syncthing",
                "EndpointID": "0d69ed6c739f232f9c37b24e357b0c9ba8801b4ab56f45071d1924518b652cde",
                "MacAddress": "02:42:ac:13:00:04",
                "IPv4Address": "172.19.0.4/16",
                "IPv6Address": ""
            },
            "39c6b062f709f6ca0475158aaab47df32626f21e8fb7155cc4d5026c7e9e140f": {
                "Name": "nginx",
                "EndpointID": "3038a319d48f31d462bb49e53d14b3b8c83be66bed3bac95b1d26775ddb66fbb",
                "MacAddress": "02:42:ac:13:00:07",
                "IPv4Address": "172.19.0.7/16",
                "IPv6Address": ""
            },
            "403e400cf8996684c90f59b10b835e69c13e0f3357ef96dacc2a981eb47ba0c9": {
                "Name": "vaultwarden",
                "EndpointID": "c98a7afce13a946ec297332d03c0e8830d64ce94629dbd9249a72282b49a52c3",
                "MacAddress": "02:42:ac:13:00:02",
                "IPv4Address": "172.19.0.2/16",
                "IPv6Address": ""
            },
            "4c3d6ea9994d5e01821fc483b783293b02d6bbce76608eac721db1ac387348d1": {
                "Name": "pigallery2",
                "EndpointID": "c2a4fc6be967cf3063642650cd637dedf3e2443e39a571e0f11a6adb425a527b",
                "MacAddress": "02:42:ac:13:00:05",
                "IPv4Address": "172.19.0.5/16",
                "IPv6Address": ""
            },
            "631ec49bcd312833d0c1a3df132d43350830268266125e1e89ff89952ecc0de9": {
                "Name": "recipes-nginx_recipes-1",
                "EndpointID": "eb82af8a74016ff2597fa9dd277d0f8f52b0f1015db3ccefd2f2d34ed6463127",
                "MacAddress": "02:42:ac:13:00:03",
                "IPv4Address": "172.19.0.3/16",
                "IPv6Address": ""
            },
            "9c35a90a0638349d92e9e0f43cb774eccc127ffeade28b2b26c6b54674e140fc": {
                "Name": "traefik",
                "EndpointID": "034bee04e65fb24d968bde9f10174e36e5b5beef75e6e284f45ad2fe40116df0",
                "MacAddress": "02:42:ac:13:00:0b",
                "IPv4Address": "172.19.0.11/16",
                "IPv6Address": ""
            },
            "af4f91257b70be2de026bbc1033b6cd2ff9b6a029f66e01e279aaa3293af4499": {
                "Name": "ejabberd",
                "EndpointID": "9c0aebc8aa42ab4706e6b9d5e118d08acc9b07bcef6044b71f4a8f8c8f9f0475",
                "MacAddress": "02:42:ac:13:00:09",
                "IPv4Address": "172.19.0.9/16",
                "IPv6Address": ""
            },
            "cbda9622e90abca831931fb31699d998889e7b8fa9d2919d245cbd06a5a61111": {
                "Name": "calibre-web",
                "EndpointID": "7f87b10ca8b8d929dfb5f437b58d702c0e9444f76051e9bee80718baf001a4c8",
                "MacAddress": "02:42:ac:13:00:0c",
                "IPv4Address": "172.19.0.12/16",
                "IPv6Address": ""
            },
            "f7d5a67baa33732a85afb4984ecb74085d8e7aa0a2fa2e4da614807ff2672aac": {
                "Name": "fhem",
                "EndpointID": "68846190828e33e049c816fb9050ab9f5f4758144a7f44769378f71ed0d37a13",
                "MacAddress": "02:42:ac:13:00:06",
                "IPv4Address": "172.19.0.6/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "proxy",
            "com.docker.compose.project": "traefik",
            "com.docker.compose.version": "2.17.3"
        }
    }
]

Matthias

Can you connect from within the Traefik container? Something like

docker exec -it traefik ping 192.168.105.161

You can also try wget, that’s included, curl is not.

No, ping and wget cannot reach ip 192.168.105.161 - that seems to be the cause of my problems.

How can I reach the openwrt VLAN from docker-container? Again, it worked before...

Matthias

This seems more like a Docker problem, maybe head over to their forum.

OK, thank you for your help.

Matthias