Long response time from Traefik (10 seconds too long - 2x 5 second freezes)

Traefik Traefik v2
1

I'm having really long response times from my Traefik, it looks like it's 10 seconds too long (I see 2x 5 second freezes).

I'm using wildcard certificate from Let's Encrypt that I handle outside of Traefik.

The container runs on a macvlan network (vlan-service-network).

Here's the docker-compose.yaml:

version: "3"
services:
  traefik:
    image: traefik:v2.9
    container_name: traefik
    networks:
      vlan-service-network:
        ipv4_address: 10.2.0.131
      default:
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./traefik.yaml:/etc/traefik/traefik.yaml
      - ./tls.yaml:/etc/traefik/tls.yaml
      - ./ssl/:/certs/
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.service=api@internal"
  whoami:
    image: containous/whoami
    container_name: whoami
    labels:
      - "traefik.enable=true"
networks:
  default:
    driver: bridge
  vlan-service-network:
    external: true

tls.yaml:

tls:
  stores:
    default:
      defaultCertificate:
        certFile: /certs/mydomain.com/fullchain.pem
        keyFile: /certs/mydomain.com/privkey.pem

traefik.yaml:

log:
  level: DEBUG

api:
  insecure: false
  dashboard: true

entryPoints:
  web:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
  websecure:
    address: ":443"
    http:
      tls: true
providers:
  file:
    filename: /etc/traefik/tls.yaml
  docker:
    endpoint: unix:///var/run/docker.sock
    watch: true
    exposedByDefault: false
    defaultRule: "HostRegexp(`{{ index .Labels \"com.docker.compose.service\"}}.mydomain.com`)"

I used time and moreutils' ts for the following call. Please notice the 5s delays at 03:12:26 and 03:12:31

$ time curl -svL http://whoami.mydomain.com |& ts '[%Y-%m-%d %H:%M:%.S]'
[2023-03-09 03:12:20.775811] *   Trying 10.2.0.131:80...
[2023-03-09 03:12:20.776229] * Connected to whoami.mydomain.com (10.2.0.131) port 80 (#0)
[2023-03-09 03:12:20.776238] > GET / HTTP/1.1
[2023-03-09 03:12:20.776245] > Host: whoami.mydomain.com
[2023-03-09 03:12:20.776252] > User-Agent: curl/7.86.0
[2023-03-09 03:12:20.776258] > Accept: */*
[2023-03-09 03:12:20.776265] >
[2023-03-09 03:12:26.096582] * Mark bundle as not supporting multiuse
[2023-03-09 03:12:26.096890] < HTTP/1.1 301 Moved Permanently
[2023-03-09 03:12:26.096942] < Location: https://whoami.mydomain.com/
[2023-03-09 03:12:26.096983] < Date: Thu, 09 Mar 2023 02:12:26 GMT
[2023-03-09 03:12:26.097200] < Content-Length: 17
[2023-03-09 03:12:26.097394] < Content-Type: text/plain; charset=utf-8
[2023-03-09 03:12:26.097484] <
[2023-03-09 03:12:26.097530] * Ignoring the response-body
[2023-03-09 03:12:26.097571] { [17 bytes data]
[2023-03-09 03:12:26.097610] * Connection #0 to host whoami.mydomain.com left intact
[2023-03-09 03:12:26.097649] * Clear auth, redirects to port from 80 to 443
[2023-03-09 03:12:26.097686] * Issue another request to this URL: 'https://whoami.mydomain.com/'
[2023-03-09 03:12:26.104533] *   Trying 10.2.0.131:443...
[2023-03-09 03:12:26.108736] * Connected to whoami.mydomain.com (10.2.0.131) port 443 (#1)
[2023-03-09 03:12:26.108904] * ALPN: offers h2
[2023-03-09 03:12:26.108915] * ALPN: offers http/1.1
[2023-03-09 03:12:26.111731] *  CAfile: /etc/ssl/cert.pem
[2023-03-09 03:12:26.111761] *  CApath: none
[2023-03-09 03:12:26.112060] * (304) (OUT), TLS handshake, Client hello (1):
[2023-03-09 03:12:26.112111] } [325 bytes data]
[2023-03-09 03:12:31.669751] * (304) (IN), TLS handshake, Server hello (2):
[2023-03-09 03:12:31.669809] { [122 bytes data]
[2023-03-09 03:12:31.669818] * (304) (IN), TLS handshake, Unknown (8):
[2023-03-09 03:12:31.669854] { [15 bytes data]
[2023-03-09 03:12:31.669870] * (304) (IN), TLS handshake, Certificate (11):
[2023-03-09 03:12:31.669954] { [4030 bytes data]
[2023-03-09 03:12:31.670697] * (304) (IN), TLS handshake, CERT verify (15):
[2023-03-09 03:12:31.670711] { [264 bytes data]
[2023-03-09 03:12:31.670861] * (304) (IN), TLS handshake, Finished (20):
[2023-03-09 03:12:31.670882] { [36 bytes data]
[2023-03-09 03:12:31.670894] * (304) (OUT), TLS handshake, Finished (20):
[2023-03-09 03:12:31.670903] } [36 bytes data]
[2023-03-09 03:12:31.670910] * SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
[2023-03-09 03:12:31.670917] * ALPN: server accepted h2
[2023-03-09 03:12:31.670925] * Server certificate:
[2023-03-09 03:12:31.670932] *  subject: CN=*.mydomain.com
[2023-03-09 03:12:31.670939] *  start date: Jan 15 17:44:15 2023 GMT
[2023-03-09 03:12:31.670945] *  expire date: Apr 15 17:44:14 2023 GMT
[2023-03-09 03:12:31.670952] *  subjectAltName: host "whoami.mydomain.com" matched cert's "*.mydomain.com"
[2023-03-09 03:12:31.670959] *  issuer: C=US; O=Let's Encrypt; CN=R3
[2023-03-09 03:12:31.670966] *  SSL certificate verify ok.
[2023-03-09 03:12:31.670974] * Using HTTP2, server supports multiplexing
[2023-03-09 03:12:31.670981] * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
[2023-03-09 03:12:31.670989] * h2h3 [:method: GET]
[2023-03-09 03:12:31.670995] * h2h3 [:path: /]
[2023-03-09 03:12:31.671002] * h2h3 [:scheme: https]
[2023-03-09 03:12:31.671009] * h2h3 [:authority: whoami.mydomain.com]
[2023-03-09 03:12:31.671016] * h2h3 [user-agent: curl/7.86.0]
[2023-03-09 03:12:31.671023] * h2h3 [accept: */*]
[2023-03-09 03:12:31.671030] * Using Stream ID: 1 (easy handle 0x145813600)
[2023-03-09 03:12:31.671394] > GET / HTTP/2
[2023-03-09 03:12:31.671418] > Host: whoami.mydomain.com
[2023-03-09 03:12:31.671427] > user-agent: curl/7.86.0
[2023-03-09 03:12:31.671435] > accept: */*
[2023-03-09 03:12:31.671443] >
[2023-03-09 03:12:31.676028] * Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
[2023-03-09 03:12:31.679922] < HTTP/2 200
[2023-03-09 03:12:31.679992] < content-type: text/plain; charset=utf-8
[2023-03-09 03:12:31.680014] < date: Thu, 09 Mar 2023 02:12:31 GMT
[2023-03-09 03:12:31.680033] < content-length: 363
[2023-03-09 03:12:31.680052] <
[2023-03-09 03:12:31.680078] { [363 bytes data]
[2023-03-09 03:12:31.680097] * Connection #1 to host whoami.mydomain.com left intact
[2023-03-09 03:12:31.680116] Hostname: a05f0195a181
[2023-03-09 03:12:31.680135] IP: 127.0.0.1
[2023-03-09 03:12:31.680155] IP: 172.20.0.5
[2023-03-09 03:12:31.680174] RemoteAddr: 172.20.0.6:51452
[2023-03-09 03:12:31.680192] GET / HTTP/1.1
[2023-03-09 03:12:31.680211] Host: whoami.mydomain.com
[2023-03-09 03:12:31.680229] User-Agent: curl/7.86.0
[2023-03-09 03:12:31.680248] Accept: */*
[2023-03-09 03:12:31.680266] Accept-Encoding: gzip
[2023-03-09 03:12:31.680285] X-Forwarded-For: 10.0.0.101
[2023-03-09 03:12:31.680303] X-Forwarded-Host: whoami.mydomain.com
[2023-03-09 03:12:31.680320] X-Forwarded-Port: 443
[2023-03-09 03:12:31.680338] X-Forwarded-Proto: https
[2023-03-09 03:12:31.680357] X-Forwarded-Server: 6978e800d4c9
[2023-03-09 03:12:31.680376] X-Real-Ip: 10.0.0.101
[2023-03-09 03:12:31.680396]

real    0m10.926s
user    0m0.021s
sys 0m0.014s

Here's the log from traefik's container:

time="2023-03-09T02:11:30Z" level=info msg="Configuration loaded from file: /etc/traefik/traefik.yaml"
time="2023-03-09T02:11:30Z" level=info msg="Traefik version 2.9.8 built on 2023-02-15T15:23:25Z"
time="2023-03-09T02:11:30Z" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"web\":{\"address\":\":80\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{\"redirections\":{\"entryPoint\":{\"to\":\"websecure\",\"scheme\":\"https\",\"permanent\":true,\"priority\":2147483646}}},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}},\"websecure\":{\"address\":\":443\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{\"tls\":{}},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}}},\"providers\":{\"providersThrottleDuration\":\"2s\",\"docker\":{\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"HostRegexp(`{{ index .Labels \\\"com.docker.compose.service\\\"}}.mydomain.com`)\",\"swarmModeRefreshSeconds\":\"15s\"},\"file\":{\"watch\":true,\"filename\":\"/etc/traefik/tls.yaml\"}},\"api\":{\"dashboard\":true},\"log\":{\"level\":\"DEBUG\",\"format\":\"common\"}}"
time="2023-03-09T02:11:30Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\n"
time="2023-03-09T02:11:30Z" level=info msg="Starting provider aggregator aggregator.ProviderAggregator"
time="2023-03-09T02:11:30Z" level=debug msg="Starting TCP Server" entryPointName=web
time="2023-03-09T02:11:30Z" level=debug msg="Starting TCP Server" entryPointName=websecure
time="2023-03-09T02:11:30Z" level=info msg="Starting provider *file.Provider"
time="2023-03-09T02:11:30Z" level=debug msg="*file.Provider provider configuration: {\"watch\":true,\"filename\":\"/etc/traefik/tls.yaml\"}"
time="2023-03-09T02:11:30Z" level=info msg="Starting provider *traefik.Provider"
time="2023-03-09T02:11:30Z" level=debug msg="*traefik.Provider provider configuration: {}"
time="2023-03-09T02:11:30Z" level=info msg="Starting provider *docker.Provider"
time="2023-03-09T02:11:30Z" level=debug msg="*docker.Provider provider configuration: {\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"HostRegexp(`{{ index .Labels \\\"com.docker.compose.service\\\"}}.mydomain.com`)\",\"swarmModeRefreshSeconds\":\"15s\"}"
time="2023-03-09T02:11:30Z" level=info msg="Starting provider *acme.ChallengeTLSALPN"
time="2023-03-09T02:11:30Z" level=debug msg="*acme.ChallengeTLSALPN provider configuration: {}"
time="2023-03-09T02:11:30Z" level=debug msg="Configuration received: {\"http\":{},\"tcp\":{},\"udp\":{},\"tls\":{\"stores\":{\"default\":{}}}}" providerName=file
time="2023-03-09T02:11:30Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"web-to-websecure\":{\"entryPoints\":[\"web\"],\"middlewares\":[\"redirect-web-to-websecure\"],\"service\":\"noop@internal\",\"rule\":\"HostRegexp(`{host:.+}`)\",\"priority\":2147483646}},\"services\":{\"api\":{},\"dashboard\":{},\"noop\":{}},\"middlewares\":{\"redirect-web-to-websecure\":{\"redirectScheme\":{\"scheme\":\"https\",\"port\":\"443\",\"permanent\":true}}},\"models\":{\"websecure\":{\"tls\":{}}},\"serversTransports\":{\"default\":{\"maxIdleConnsPerHost\":200}}},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=internal
time="2023-03-09T02:11:30Z" level=debug msg="Added outgoing tracing middleware noop@internal" entryPointName=web middlewareName=tracing middlewareType=TracingForwarder routerName=web-to-websecure@internal
time="2023-03-09T02:11:30Z" level=debug msg="Creating middleware" entryPointName=web routerName=web-to-websecure@internal middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme
time="2023-03-09T02:11:30Z" level=debug msg="Setting up redirection to https 443" entryPointName=web routerName=web-to-websecure@internal middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme
time="2023-03-09T02:11:30Z" level=debug msg="Creating middleware" entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2023-03-09T02:11:30Z" level=debug msg="Provider connection established with docker 23.0.1 (API 1.42)" providerName=docker
time="2023-03-09T02:11:30Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"traefik\":{\"service\":\"api@internal\",\"rule\":\"HostRegexp(`traefik.mydomain.com`)\"},\"whoami-services\":{\"service\":\"whoami-services\",\"rule\":\"HostRegexp(`whoami.mydomain.com`)\"}},\"services\":{\"traefik-services\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.20.0.6:80\"}],\"passHostHeader\":true}},\"whoami-services\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.20.0.5:80\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"udp\":{}}" providerName=docker
time="2023-03-09T02:11:30Z" level=debug msg="No entryPoint defined for this router, using the default one(s) instead: [web websecure]" routerName=traefik
time="2023-03-09T02:11:30Z" level=debug msg="No entryPoint defined for this router, using the default one(s) instead: [web websecure]" routerName=whoami-services
time="2023-03-09T02:11:30Z" level=debug msg="Added outgoing tracing middleware api@internal" middlewareName=tracing entryPointName=web routerName=traefik@docker middlewareType=TracingForwarder
time="2023-03-09T02:11:30Z" level=debug msg="Creating middleware" middlewareType=Pipelining entryPointName=web routerName=whoami-services@docker serviceName=whoami-services middlewareName=pipelining
time="2023-03-09T02:11:30Z" level=debug msg="Creating load-balancer" serviceName=whoami-services entryPointName=web routerName=whoami-services@docker
time="2023-03-09T02:11:30Z" level=debug msg="Creating server 0 http://172.20.0.5:80" entryPointName=web routerName=whoami-services@docker serviceName=whoami-services serverName=0
time="2023-03-09T02:11:30Z" level=debug msg="child http://172.20.0.5:80 now UP"
time="2023-03-09T02:11:30Z" level=debug msg="Propagating new UP status"
time="2023-03-09T02:11:30Z" level=debug msg="Added outgoing tracing middleware whoami-services" entryPointName=web routerName=whoami-services@docker middlewareName=tracing middlewareType=TracingForwarder
time="2023-03-09T02:11:30Z" level=debug msg="Added outgoing tracing middleware noop@internal" routerName=web-to-websecure@internal middlewareName=tracing middlewareType=TracingForwarder entryPointName=web
time="2023-03-09T02:11:30Z" level=debug msg="Creating middleware" routerName=web-to-websecure@internal middlewareType=RedirectScheme middlewareName=redirect-web-to-websecure@internal entryPointName=web
time="2023-03-09T02:11:30Z" level=debug msg="Setting up redirection to https 443" middlewareType=RedirectScheme middlewareName=redirect-web-to-websecure@internal entryPointName=web routerName=web-to-websecure@internal
time="2023-03-09T02:11:30Z" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=web
time="2023-03-09T02:11:30Z" level=debug msg="Added outgoing tracing middleware api@internal" entryPointName=websecure routerName=websecure-traefik@docker middlewareType=TracingForwarder middlewareName=tracing
time="2023-03-09T02:11:30Z" level=debug msg="Creating middleware" middlewareName=pipelining middlewareType=Pipelining entryPointName=websecure routerName=websecure-whoami-services@docker serviceName=whoami-services
time="2023-03-09T02:11:30Z" level=debug msg="Creating load-balancer" routerName=websecure-whoami-services@docker serviceName=whoami-services entryPointName=websecure
time="2023-03-09T02:11:30Z" level=debug msg="Creating server 0 http://172.20.0.5:80" entryPointName=websecure routerName=websecure-whoami-services@docker serviceName=whoami-services serverName=0
time="2023-03-09T02:11:30Z" level=debug msg="child http://172.20.0.5:80 now UP"
time="2023-03-09T02:11:30Z" level=debug msg="Propagating new UP status"
time="2023-03-09T02:11:30Z" level=debug msg="Added outgoing tracing middleware whoami-services" middlewareType=TracingForwarder entryPointName=websecure routerName=websecure-whoami-services@docker middlewareName=tracing
time="2023-03-09T02:11:30Z" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=websecure middlewareName=traefik-internal-recovery
time="2023-03-09T02:11:30Z" level=warning msg="No domain found in rule HostRegexp(`traefik.mydomain.com`), the TLS options applied for this router will depend on the SNI of each request" entryPointName=websecure routerName=websecure-traefik@docker
time="2023-03-09T02:11:30Z" level=warning msg="No domain found in rule HostRegexp(`whoami.mydomain.com`), the TLS options applied for this router will depend on the SNI of each request" entryPointName=websecure routerName=websecure-whoami-services@docker



time="2023-03-09T02:12:31Z" level=debug msg="Serving default certificate for request: \"whoami.mydomain.com\""

I'm seeing the same issue when accessing the dashboard (traefik.mydomain.com).

I'm pretty sure, I'm doing something wrong, can someone point me in the right direction?

2

Hi @shalak. Thanks for your interest in Traefik!

According to your posted configuration, I assume you're talking about Traefik Proxy and not Traefik Hub?
Is that correct?

At the moment, your post is labeled as category Traefik Hub.

3

That is correct, sorry, must've miss-clicked the category. I'm moving the topic to proper place.

4

Hi @shalak I would suggest to start debugging with a simpler configuration for now.

I would start with just one network.
Once I am sure, that is working like expected, I would add the second network.

By doing so, it is easier to debug and figure out potential issues.

1 Like
5

Huh, with only one network it works immediately... How can I debug what's wrong with two networks? Even with DEBUG, the only log I see during the call is the time="2023-03-09T02:12:31Z" level=debug msg="Serving default certificate for request: \"whoami.mydomain.com\""...

Edit:
I tried binding the entryPoints explicitly to the IP like so:

entryPoints:
  web:
    address: "10.2.0.131:80"
# ...
  websecure:
    address: "10.2.0.131:443"

That didn't change anything :confused:

EDIT2:

The first 5s freeze is between sending the request and getting the * Mark bundle as not supporting multiuse - it only happens when I request an HTTP endpoint.

The second freeze is between Client & Server Hello - during the TLS handshake (and it happens with both HTTP and HTTPS endpoints)

EDIT3:

I deployed a docker run --name swiss-army-knife --net=container:traefik -itd leodotcloud/swiss-army-knife and executed curl -vL -H Host:whoami.mydomain.com 10.2.0.131 from inside the swiss-army-knife - it works correctly.

Any ideas?

6

Huh, I believe I confirmed that the issue doesn't come from Traefik, but maybe from my networking? :frowning:

Until now, I've been making requests from 10.0.0.101, which is routed to 10.2.0.0/24 on my mikrotik router. Here's the tcpdump -n -i eth1 from swiss-army-knife (eth1 is the 10.2.0.131).

16:47:56.563373 IP 10.0.0.101.52115 > 10.2.0.131.80: Flags [S], seq 2644962693, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 437066641 ecr 0,sackOK,eol], length 0
16:48:01.640651 ARP, Request who-has 10.2.0.131 tell 10.2.0.1, length 46
16:48:01.640706 ARP, Reply 10.2.0.131 is-at 02:42:0a:02:00:83, length 28
16:48:01.897144 IP 10.0.0.101.52115 > 10.2.0.131.80: Flags [P.], seq 2644962694:2644962778, ack 3159032917, win 2058, options [nop,nop,TS val 437071978 ecr 3483486319], length 84: HTTP: GET / HTTP/1.1
16:48:01.903222 IP 10.0.0.101.52115 > 10.2.0.131.80: Flags [.], ack 191, win 2055, options [nop,nop,TS val 437071984 ecr 3483491656], length 0
16:48:01.909407 IP 10.0.0.101.52117 > 10.2.0.131.443: Flags [S], seq 1783932268, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 2215034105 ecr 0,sackOK,eol], length 0
16:48:07.320120 IP 10.0.0.101.52117 > 10.2.0.131.443: Flags [P.], seq 1783932269:1783932599, ack 3031798770, win 2058, options [nop,nop,TS val 2215039443 ecr 3483491665], length 330
16:48:07.338959 IP 10.0.0.101.52117 > 10.2.0.131.443: Flags [.], ack 2897, win 2013, options [nop,nop,TS val 2215039534 ecr 3483497091], length 0
16:48:07.338960 IP 10.0.0.101.52117 > 10.2.0.131.443: Flags [.], ack 2897, win 2013, options [nop,nop,TS val 2215039534 ecr 3483497091,nop,nop,sack 1 {4345:4567}], length 0
16:48:07.338960 IP 10.0.0.101.52117 > 10.2.0.131.443: Flags [.], ack 4567, win 1987, options [nop,nop,TS val 2215039534 ecr 3483497091], length 0
16:48:07.341223 IP 10.0.0.101.52117 > 10.2.0.131.443: Flags [.], ack 4567, win 2048, options [nop,nop,TS val 2215039537 ecr 3483497091], length 0
16:48:07.341223 IP 10.0.0.101.52117 > 10.2.0.131.443: Flags [P.], seq 330:336, ack 4567, win 2048, options [nop,nop,TS val 2215039537 ecr 3483497091], length 6
16:48:07.341223 IP 10.0.0.101.52117 > 10.2.0.131.443: Flags [P.], seq 336:394, ack 4567, win 2048, options [nop,nop,TS val 2215039537 ecr 3483497091], length 58
16:48:07.342222 IP 10.0.0.101.52117 > 10.2.0.131.443: Flags [P.], seq 394:440, ack 4567, win 2048, options [nop,nop,TS val 2215039537 ecr 3483497091], length 46
16:48:07.342222 IP 10.0.0.101.52117 > 10.2.0.131.443: Flags [P.], seq 440:489, ack 4567, win 2048, options [nop,nop,TS val 2215039537 ecr 3483497091], length 49
16:48:07.342222 IP 10.0.0.101.52117 > 10.2.0.131.443: Flags [P.], seq 489:524, ack 4567, win 2048, options [nop,nop,TS val 2215039537 ecr 3483497091], length 35
16:48:07.342356 IP 10.0.0.101.52117 > 10.2.0.131.443: Flags [P.], seq 524:590, ack 4567, win 2048, options [nop,nop,TS val 2215039537 ecr 3483497091], length 66
16:48:07.344377 IP 10.0.0.101.52117 > 10.2.0.131.443: Flags [.], ack 4628, win 2047, options [nop,nop,TS val 2215039540 ecr 3483497098], length 0
16:48:07.344473 IP 10.0.0.101.52117 > 10.2.0.131.443: Flags [P.], seq 590:621, ack 4628, win 2048, options [nop,nop,TS val 2215039540 ecr 3483497098], length 31
16:48:07.346213 IP 10.0.0.101.52117 > 10.2.0.131.443: Flags [.], ack 4663, win 2047, options [nop,nop,TS val 2215039541 ecr 3483497099], length 0
16:48:07.346213 IP 10.0.0.101.52117 > 10.2.0.131.443: Flags [.], ack 4694, win 2047, options [nop,nop,TS val 2215039541 ecr 3483497099], length 0
16:48:07.351684 IP 10.0.0.101.52117 > 10.2.0.131.443: Flags [.], ack 4775, win 2046, options [nop,nop,TS val 2215039548 ecr 3483497104], length 0
16:48:07.352907 IP 10.0.0.101.52117 > 10.2.0.131.443: Flags [.], ack 5169, win 2041, options [nop,nop,TS val 2215039548 ecr 3483497105], length 0
16:48:07.352907 IP 10.0.0.101.52117 > 10.2.0.131.443: Flags [P.], seq 621:645, ack 5169, win 2048, options [nop,nop,TS val 2215039548 ecr 3483497105], length 24
16:48:07.353872 IP 10.0.0.101.52115 > 10.2.0.131.80: Flags [F.], seq 84, ack 191, win 2055, options [nop,nop,TS val 437077434 ecr 3483491656], length 0
16:48:07.353872 IP 10.0.0.101.52117 > 10.2.0.131.443: Flags [F.], seq 645, ack 5169, win 2048, options [nop,nop,TS val 2215039549 ecr 3483497105], length 0
16:48:07.355849 IP 10.0.0.101.52117 > 10.2.0.131.443: Flags [R], seq 1783932914, win 0, length 0
16:48:07.357009 IP 10.0.0.101.52115 > 10.2.0.131.80: Flags [.], ack 192, win 2055, options [nop,nop,TS val 437077437 ecr 3483497110], length 0

Now, I added a host that is connected directly to 10.2.0.0/24 network, and the curl request works immediately. And the tcpdump looks like this:

16:47:35.846253 IP 10.2.0.4.55308 > 10.2.0.131.80: Flags [S], seq 3532360627, win 64240, options [mss 1460,sackOK,TS val 2155616014 ecr 0,nop,wscale 7], length 0
16:47:35.846525 IP 10.2.0.131.80 > 10.2.0.4.55308: Flags [S.], seq 1237468008, ack 3532360628, win 65160, options [mss 1460,sackOK,TS val 3322942501 ecr 2155616014,nop,wscale 7], length 0
16:47:35.847208 IP 10.2.0.4.55308 > 10.2.0.131.80: Flags [.], ack 1, win 502, options [nop,nop,TS val 2155616015 ecr 3322942501], length 0
16:47:35.847636 IP 10.2.0.4.55308 > 10.2.0.131.80: Flags [P.], seq 1:85, ack 1, win 502, options [nop,nop,TS val 2155616015 ecr 3322942501], length 84: HTTP: GET / HTTP/1.1
16:47:35.847749 IP 10.2.0.131.80 > 10.2.0.4.55308: Flags [.], ack 85, win 509, options [nop,nop,TS val 3322942502 ecr 2155616015], length 0
16:47:35.850258 IP 10.2.0.131.80 > 10.2.0.4.55308: Flags [P.], seq 1:191, ack 85, win 509, options [nop,nop,TS val 3322942505 ecr 2155616015], length 190: HTTP: HTTP/1.1 301 Moved Permanently
16:47:35.850816 IP 10.2.0.4.55308 > 10.2.0.131.80: Flags [.], ack 191, win 501, options [nop,nop,TS val 2155616019 ecr 3322942505], length 0
16:47:35.853922 IP 10.2.0.4.57034 > 10.2.0.131.443: Flags [S], seq 1697357511, win 64240, options [mss 1460,sackOK,TS val 2155616022 ecr 0,nop,wscale 7], length 0
16:47:35.854048 IP 10.2.0.131.443 > 10.2.0.4.57034: Flags [S.], seq 4009257224, ack 1697357512, win 65160, options [mss 1460,sackOK,TS val 3322942509 ecr 2155616022,nop,wscale 7], length 0
16:47:35.854539 IP 10.2.0.4.57034 > 10.2.0.131.443: Flags [.], ack 1, win 502, options [nop,nop,TS val 2155616023 ecr 3322942509], length 0
16:47:35.857476 IP 10.2.0.4.57034 > 10.2.0.131.443: Flags [P.], seq 1:518, ack 1, win 502, options [nop,nop,TS val 2155616025 ecr 3322942509], length 517
16:47:35.857601 IP 10.2.0.131.443 > 10.2.0.4.57034: Flags [.], ack 518, win 506, options [nop,nop,TS val 3322942512 ecr 2155616025], length 0
16:47:35.868687 IP 10.2.0.131.443 > 10.2.0.4.57034: Flags [P.], seq 1:4719, ack 518, win 506, options [nop,nop,TS val 3322942523 ecr 2155616025], length 4718
16:47:35.869360 IP 10.2.0.4.57034 > 10.2.0.131.443: Flags [.], ack 4719, win 485, options [nop,nop,TS val 2155616037 ecr 3322942523], length 0
16:47:35.872830 IP 10.2.0.4.57034 > 10.2.0.131.443: Flags [P.], seq 518:582, ack 4719, win 485, options [nop,nop,TS val 2155616041 ecr 3322942523], length 64
16:47:35.873633 IP 10.2.0.131.443 > 10.2.0.4.57034: Flags [P.], seq 4719:4780, ack 582, win 506, options [nop,nop,TS val 3322942528 ecr 2155616041], length 61
16:47:35.873715 IP 10.2.0.4.57034 > 10.2.0.131.443: Flags [P.], seq 582:628, ack 4719, win 485, options [nop,nop,TS val 2155616042 ecr 3322942523], length 46
16:47:35.873715 IP 10.2.0.4.57034 > 10.2.0.131.443: Flags [P.], seq 628:677, ack 4719, win 485, options [nop,nop,TS val 2155616042 ecr 3322942523], length 49
16:47:35.873917 IP 10.2.0.4.57034 > 10.2.0.131.443: Flags [P.], seq 677:712, ack 4719, win 485, options [nop,nop,TS val 2155616042 ecr 3322942523], length 35
16:47:35.873918 IP 10.2.0.4.57034 > 10.2.0.131.443: Flags [P.], seq 712:778, ack 4719, win 485, options [nop,nop,TS val 2155616042 ecr 3322942523], length 66
16:47:35.873959 IP 10.2.0.131.443 > 10.2.0.4.57034: Flags [.], ack 677, win 506, options [nop,nop,TS val 3322942529 ecr 2155616042], length 0
16:47:35.874138 IP 10.2.0.131.443 > 10.2.0.4.57034: Flags [P.], seq 4780:4815, ack 778, win 506, options [nop,nop,TS val 3322942529 ecr 2155616042], length 35
16:47:35.874240 IP 10.2.0.4.57034 > 10.2.0.131.443: Flags [.], ack 4780, win 485, options [nop,nop,TS val 2155616042 ecr 3322942528], length 0
16:47:35.874597 IP 10.2.0.131.443 > 10.2.0.4.57034: Flags [P.], seq 4815:4846, ack 778, win 506, options [nop,nop,TS val 3322942529 ecr 2155616042], length 31
16:47:35.874709 IP 10.2.0.4.57034 > 10.2.0.131.443: Flags [.], ack 4815, win 501, options [nop,nop,TS val 2155616043 ecr 3322942529], length 0
16:47:35.874709 IP 10.2.0.4.57034 > 10.2.0.131.443: Flags [P.], seq 778:809, ack 4815, win 501, options [nop,nop,TS val 2155616043 ecr 3322942529], length 31
16:47:35.875133 IP 10.2.0.4.57034 > 10.2.0.131.443: Flags [.], ack 4846, win 501, options [nop,nop,TS val 2155616043 ecr 3322942529], length 0
16:47:35.879636 IP 10.2.0.131.443 > 10.2.0.4.57034: Flags [P.], seq 4846:4928, ack 809, win 506, options [nop,nop,TS val 3322942534 ecr 2155616043], length 82
16:47:35.879918 IP 10.2.0.131.443 > 10.2.0.4.57034: Flags [P.], seq 4928:5318, ack 809, win 506, options [nop,nop,TS val 3322942534 ecr 2155616043], length 390
16:47:35.880151 IP 10.2.0.4.57034 > 10.2.0.131.443: Flags [.], ack 4928, win 501, options [nop,nop,TS val 2155616048 ecr 3322942534], length 0
16:47:35.880595 IP 10.2.0.4.57034 > 10.2.0.131.443: Flags [.], ack 5318, win 501, options [nop,nop,TS val 2155616049 ecr 3322942534], length 0
16:47:35.881050 IP 10.2.0.4.55308 > 10.2.0.131.80: Flags [F.], seq 85, ack 191, win 501, options [nop,nop,TS val 2155616049 ecr 3322942505], length 0
16:47:35.881050 IP 10.2.0.4.57034 > 10.2.0.131.443: Flags [P.], seq 809:833, ack 5318, win 501, options [nop,nop,TS val 2155616049 ecr 3322942534], length 24
16:47:35.881215 IP 10.2.0.4.57034 > 10.2.0.131.443: Flags [F.], seq 833, ack 5318, win 501, options [nop,nop,TS val 2155616049 ecr 3322942534], length 0
16:47:35.881373 IP 10.2.0.131.443 > 10.2.0.4.57034: Flags [P.], seq 5318:5342, ack 834, win 506, options [nop,nop,TS val 3322942536 ecr 2155616049], length 24
16:47:35.881457 IP 10.2.0.131.80 > 10.2.0.4.55308: Flags [F.], seq 191, ack 86, win 509, options [nop,nop,TS val 3322942536 ecr 2155616049], length 0
16:47:35.881523 IP 10.2.0.131.443 > 10.2.0.4.57034: Flags [F.], seq 5342, ack 834, win 506, options [nop,nop,TS val 3322942536 ecr 2155616049], length 0
16:47:35.882054 IP 10.2.0.4.57034 > 10.2.0.131.443: Flags [R], seq 1697358345, win 0, length 0
16:47:35.882054 IP 10.2.0.4.55308 > 10.2.0.131.80: Flags [.], ack 192, win 501, options [nop,nop,TS val 2155616050 ecr 3322942536], length 0
16:47:35.882054 IP 10.2.0.4.57034 > 10.2.0.131.443: Flags [R], seq 1697358345, win 0, length 0

Here's the pcap view from wireshark from 10.0.0.101's perspective:

image
image1920×761 299 KB

Unfortunately, I do not get why the TCP retransmission is happening :frowning: This is what takes the 5 seconds.

EDIT: please notice, that it takes 5 seconds for router to send out the ARP who-has - why is it taking so long? Maybe the issue is on Traefik side after all (either the problem is with the container, or with docker host networking? But pings are coming through without any delays...)

EDIT2: I've set two services in the 10.2.0.0/24 network (also using the macvlan docker network):

  • a plain WWW server - connects correctly from 10.0.0.0/24
  • a Nginx Proxy Manager - suffers from the same issue as Traefik

EDIT3: I can confirm that the issue is unrelated to macvlan interface - when I used a docker host network, but in 10.2.0.0/24, the issue still occurs.

EDIT4: What's even weirder is if I do the following on the host that has both networks:

  • time curl -vLH Host:whoami.mydomain.com --interface 10.0.0.4 http://10.2.0.131
  • time curl -vLH Host:whoami.mydomain.com --interface 10.2.0.4 http://10.2.0.131
    It works correctly. However, the moment I disconnect the 10.2.0.4 interface, the call via 10.0.0.4 suffers from the issue at hand :crazy_face:

It's making my head spin :confused:

7

I updated my comment above with more troubleshooting info. At this point I'm desperate and looks like I'll have to drop my dreams of having a DMZ in my network :sob: