Intermittent 400 Bad Request responses

Traefik Traefik v1
1

We have an ASP.NET Core 2.1 Web API deployed as a docker service.
A Traefik service is set up to route the requests coming from our IIS server to the correct web API docker service.

One of our services (and only on this service), does not receive some requests from time to time.
The client and IIS logs would receive a 400 Bad Request response sometimes.

In our test, we have the same request with changing query parameter.
We execute this request 300 times, and sometimes we would receive a 400 Bad Request response for 1/300 request.

We are using version 1.7

YAML Config

  traefik:
    image: traefik
    hostname: traefik
    command:
      - "--docker"
      - "--docker.swarmMode"
      - "--docker.domain=xxxx.com"
      - "--docker.watch"
      - "--loglevel=DEBUG"
      - "--constraints=tag==uat"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    ports:
      - target: 80
        published: 3500
        mode: host
    deploy:
      replicas: 1
      placement:
        constraints: [node.labels.traefikhost==true]
      update_config:
        parallelism: 1
        delay: 15s
      restart_policy:
        condition: on-failure
    networks:
      - webgateway
      - app_swarm


  account:
    image: 53.125.25.125:5000/dsc/app.account.restservice:2.1.1078
    ports:
      - "80"
    env_file:
     - ./common.env
    networks:
      - app_swarm
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints: [node.labels.function == worker-manager]
      labels:
      - traefik.frontend.rule=PathPrefixStrip:/AccountJp
      - traerik.frontend2.rule=PathPrefixStrip:/Account
      - traefik.docker.network=app_swarm
      - traefik.port=80
      - traefik.enable=true
      - traefik.tags=uat
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 120s
    depends_on:
      - traefik

We have an IIS server that routes all of the API requests to our Traefik docker service (app_traefik).

Log output in DEBUG level

app_traefik.1.pq75bqlbodnz@LAEDC0000C55    | time="2020-09-11T03:32:20Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/api/Account/Login\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"zzzzzz=6\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"application/json, text/plain, */*\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"Keep-Alive\"],\"Content-Length\":[\"73\"],\"Content-Type\":[\"application/json\"],\"Max-Forwards\":[\"10\"],\"Postman-Token\":[\"7d53e0f9-a256-4a94-b00c-ad1342c5cc7a\"],\"Pragma\":[\"no-cache\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36\"],\"X-Arr-Log-Id\":[\"779262c2-c311-40b4-971b-e9c4cccbd889\"],\"X-Forwarded-For\":[\"112.110.128.34, 112.110.136.126:53273\"],\"X-Forwarded-Prefix\":[\"/Account\"],\"X-Original-Url\":[\"/AccountService/api/Account/Login?zzzzzz=6\"]},\"ContentLength\":73,\"TransferEncoding\":null,\"Host\":\"56.143.43.35:3500\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"56.143.5.95:56368\",\"RequestURI\":\"/api/Account/Login?zzzzzz=6\",\"TLS\":null}"
app_traefik.1.pq75bqlbodnz@LAEDC0000C55    | time="2020-09-11T03:32:20Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/api/Account/Login\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"zzzzzz=6\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"application/json, text/plain, */*\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"Keep-Alive\"],\"Content-Length\":[\"73\"],\"Content-Type\":[\"application/json\"],\"Max-Forwards\":[\"10\"],\"Postman-Token\":[\"7d53e0f9-a256-4a94-b00c-ad1342c5cc7a\"],\"Pragma\":[\"no-cache\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36\"],\"X-Arr-Log-Id\":[\"779262c2-c311-40b4-971b-e9c4cccbd889\"],\"X-Forwarded-For\":[\"112.110.128.34, 112.110.136.126:53273\"],\"X-Forwarded-Prefix\":[\"/Account\"],\"X-Original-Url\":[\"/AccountService/api/Account/Login?zzzzzz=6\"]},\"ContentLength\":73,\"TransferEncoding\":null,\"Host\":\"56.143.43.35:3500\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"56.143.5.95:56368\",\"RequestURI\":\"/api/Account/Login?zzzzzz=6\",\"TLS\":null}" ForwardURL="http://10.0.3.124:80"
app_traefik.1.pq75bqlbodnz@LAEDC0000C55    | time="2020-09-11T03:32:20Z" level=debug msg="vulcand/oxy/forward: begin ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.0.3.124:80\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"application/json, text/plain, */*\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"Keep-Alive\"],\"Content-Length\":[\"73\"],\"Content-Type\":[\"application/json\"],\"Max-Forwards\":[\"10\"],\"Postman-Token\":[\"7d53e0f9-a256-4a94-b00c-ad1342c5cc7a\"],\"Pragma\":[\"no-cache\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36\"],\"X-Arr-Log-Id\":[\"779262c2-c311-40b4-971b-e9c4cccbd889\"],\"X-Forwarded-For\":[\"112.110.128.34, 112.110.136.126:53273\"],\"X-Forwarded-Prefix\":[\"/Account\"],\"X-Original-Url\":[\"/AccountService/api/Account/Login?zzzzzz=6\"]},\"ContentLength\":73,\"TransferEncoding\":null,\"Host\":\"56.143.43.35:3500\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"56.143.5.95:56368\",\"RequestURI\":\"/api/Account/Login?zzzzzz=6\",\"TLS\":null}"
app_traefik.1.pq75bqlbodnz@LAEDC0000C55    | time="2020-09-11T03:32:20Z" level=debug msg="vulcand/oxy/forward/http: begin ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.0.3.124:80\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"application/json, text/plain, */*\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"Keep-Alive\"],\"Content-Length\":[\"73\"],\"Content-Type\":[\"application/json\"],\"Max-Forwards\":[\"10\"],\"Postman-Token\":[\"7d53e0f9-a256-4a94-b00c-ad1342c5cc7a\"],\"Pragma\":[\"no-cache\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36\"],\"X-Arr-Log-Id\":[\"779262c2-c311-40b4-971b-e9c4cccbd889\"],\"X-Forwarded-For\":[\"112.110.128.34, 112.110.136.126:53273\"],\"X-Forwarded-Prefix\":[\"/Account\"],\"X-Original-Url\":[\"/AccountService/api/Account/Login?zzzzzz=6\"]},\"ContentLength\":73,\"TransferEncoding\":null,\"Host\":\"56.143.43.35:3500\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"56.143.5.95:56368\",\"RequestURI\":\"/api/Account/Login?zzzzzz=6\",\"TLS\":null}"
app_traefik.1.pq75bqlbodnz@LAEDC0000C55    | time="2020-09-11T03:32:20Z" level=debug msg="vulcand/oxy/forward/http: completed ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.0.3.124:80\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"application/json, text/plain, */*\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"Keep-Alive\"],\"Content-Length\":[\"73\"],\"Content-Type\":[\"application/json\"],\"Max-Forwards\":[\"10\"],\"Postman-Token\":[\"7d53e0f9-a256-4a94-b00c-ad1342c5cc7a\"],\"Pragma\":[\"no-cache\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36\"],\"X-Arr-Log-Id\":[\"779262c2-c311-40b4-971b-e9c4cccbd889\"],\"X-Forwarded-For\":[\"112.110.128.34, 112.110.136.126:53273\"],\"X-Forwarded-Prefix\":[\"/Account\"],\"X-Original-Url\":[\"/AccountService/api/Account/Login?zzzzzz=6\"]},\"ContentLength\":73,\"TransferEncoding\":null,\"Host\":\"56.143.43.35:3500\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"56.143.5.95:56368\",\"RequestURI\":\"/api/Account/Login?zzzzzz=6\",\"TLS\":null}"
app_traefik.1.pq75bqlbodnz@LAEDC0000C55    | time="2020-09-11T03:32:20Z" level=debug msg="vulcand/oxy/forward: completed ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.0.3.124:80\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"application/json, text/plain, */*\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"Keep-Alive\"],\"Content-Length\":[\"73\"],\"Content-Type\":[\"application/json\"],\"Max-Forwards\":[\"10\"],\"Postman-Token\":[\"7d53e0f9-a256-4a94-b00c-ad1342c5cc7a\"],\"Pragma\":[\"no-cache\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36\"],\"X-Arr-Log-Id\":[\"779262c2-c311-40b4-971b-e9c4cccbd889\"],\"X-Forwarded-For\":[\"112.110.128.34, 112.110.136.126:53273\"],\"X-Forwarded-Prefix\":[\"/Account\"],\"X-Original-Url\":[\"/AccountService/api/Account/Login?zzzzzz=6\"]},\"ContentLength\":73,\"TransferEncoding\":null,\"Host\":\"56.143.43.35:3500\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"56.143.5.95:56368\",\"RequestURI\":\"/api/Account/Login?zzzzzz=6\",\"TLS\":null}"
app_traefik.1.pq75bqlbodnz@LAEDC0000C55    | time="2020-09-11T03:32:20Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/api/Account/Login\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"zzzzzz=6\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"application/json, text/plain, */*\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"Keep-Alive\"],\"Content-Length\":[\"73\"],\"Content-Type\":[\"application/json\"],\"Max-Forwards\":[\"10\"],\"Postman-Token\":[\"7d53e0f9-a256-4a94-b00c-ad1342c5cc7a\"],\"Pragma\":[\"no-cache\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36\"],\"X-Arr-Log-Id\":[\"779262c2-c311-40b4-971b-e9c4cccbd889\"],\"X-Forwarded-For\":[\"112.110.128.34, 112.110.136.126:53273\"],\"X-Forwarded-Prefix\":[\"/Account\"],\"X-Original-Url\":[\"/AccountService/api/Account/Login?zzzzzz=6\"]},\"ContentLength\":73,\"TransferEncoding\":null,\"Host\":\"56.143.43.35:3500\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"56.143.5.95:56368\",\"RequestURI\":\"/api/Account/Login?zzzzzz=6\",\"TLS\":null}"
app_traefik.1.pq75bqlbodnz@LAEDC0000C55    | time="2020-09-11T03:32:23Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/api/Account/Login\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"zzzzzz=7\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"application/json, text/plain, */*\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"Keep-Alive\"],\"Content-Length\":[\"73\"],\"Content-Type\":[\"application/json\"],\"Max-Forwards\":[\"10\"],\"Postman-Token\":[\"108d2735-e1b6-434e-8a72-a39e90ce1bd9\"],\"Pragma\":[\"no-cache\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36\"],\"X-Arr-Log-Id\":[\"67068c90-ff47-4d13-8e5c-bf5bda40c8fe\"],\"X-Forwarded-For\":[\"112.110.128.34, 112.110.136.126:37228\"],\"X-Forwarded-Prefix\":[\"/Account\"],\"X-Original-Url\":[\"/AccountService/api/Account/Login?zzzzzz=7\"]},\"ContentLength\":73,\"TransferEncoding\":null,\"Host\":\"56.143.43.35:3500\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"56.143.5.95:56370\",\"RequestURI\":\"/api/Account/Login?zzzzzz=7\",\"TLS\":null}"
app_traefik.1.pq75bqlbodnz@LAEDC0000C55    | time="2020-09-11T03:32:23Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/api/Account/Login\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"zzzzzz=7\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"application/json, text/plain, */*\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"Keep-Alive\"],\"Content-Length\":[\"73\"],\"Content-Type\":[\"application/json\"],\"Max-Forwards\":[\"10\"],\"Postman-Token\":[\"108d2735-e1b6-434e-8a72-a39e90ce1bd9\"],\"Pragma\":[\"no-cache\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36\"],\"X-Arr-Log-Id\":[\"67068c90-ff47-4d13-8e5c-bf5bda40c8fe\"],\"X-Forwarded-For\":[\"112.110.128.34, 112.110.136.126:37228\"],\"X-Forwarded-Prefix\":[\"/Account\"],\"X-Original-Url\":[\"/AccountService/api/Account/Login?zzzzzz=7\"]},\"ContentLength\":73,\"TransferEncoding\":null,\"Host\":\"56.143.43.35:3500\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"56.143.5.95:56370\",\"RequestURI\":\"/api/Account/Login?zzzzzz=7\",\"TLS\":null}" ForwardURL="http://10.0.3.124:80"
app_traefik.1.pq75bqlbodnz@LAEDC0000C55    | time="2020-09-11T03:32:23Z" level=debug msg="vulcand/oxy/forward: begin ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.0.3.124:80\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"application/json, text/plain, */*\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"Keep-Alive\"],\"Content-Length\":[\"73\"],\"Content-Type\":[\"application/json\"],\"Max-Forwards\":[\"10\"],\"Postman-Token\":[\"108d2735-e1b6-434e-8a72-a39e90ce1bd9\"],\"Pragma\":[\"no-cache\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36\"],\"X-Arr-Log-Id\":[\"67068c90-ff47-4d13-8e5c-bf5bda40c8fe\"],\"X-Forwarded-For\":[\"112.110.128.34, 112.110.136.126:37228\"],\"X-Forwarded-Prefix\":[\"/Account\"],\"X-Original-Url\":[\"/AccountService/api/Account/Login?zzzzzz=7\"]},\"ContentLength\":73,\"TransferEncoding\":null,\"Host\":\"56.143.43.35:3500\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"56.143.5.95:56370\",\"RequestURI\":\"/api/Account/Login?zzzzzz=7\",\"TLS\":null}"
app_traefik.1.pq75bqlbodnz@LAEDC0000C55    | time="2020-09-11T03:32:23Z" level=debug msg="vulcand/oxy/forward/http: begin ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.0.3.124:80\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"application/json, text/plain, */*\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"Keep-Alive\"],\"Content-Length\":[\"73\"],\"Content-Type\":[\"application/json\"],\"Max-Forwards\":[\"10\"],\"Postman-Token\":[\"108d2735-e1b6-434e-8a72-a39e90ce1bd9\"],\"Pragma\":[\"no-cache\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36\"],\"X-Arr-Log-Id\":[\"67068c90-ff47-4d13-8e5c-bf5bda40c8fe\"],\"X-Forwarded-For\":[\"112.110.128.34, 112.110.136.126:37228\"],\"X-Forwarded-Prefix\":[\"/Account\"],\"X-Original-Url\":[\"/AccountService/api/Account/Login?zzzzzz=7\"]},\"ContentLength\":73,\"TransferEncoding\":null,\"Host\":\"56.143.43.35:3500\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"56.143.5.95:56370\",\"RequestURI\":\"/api/Account/Login?zzzzzz=7\",\"TLS\":null}"
app_traefik.1.pq75bqlbodnz@LAEDC0000C55    | time="2020-09-11T03:32:23Z" level=debug msg="vulcand/oxy/forward/http: completed ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.0.3.124:80\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"application/json, text/plain, */*\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"Keep-Alive\"],\"Content-Length\":[\"73\"],\"Content-Type\":[\"application/json\"],\"Max-Forwards\":[\"10\"],\"Postman-Token\":[\"108d2735-e1b6-434e-8a72-a39e90ce1bd9\"],\"Pragma\":[\"no-cache\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36\"],\"X-Arr-Log-Id\":[\"67068c90-ff47-4d13-8e5c-bf5bda40c8fe\"],\"X-Forwarded-For\":[\"112.110.128.34, 112.110.136.126:37228\"],\"X-Forwarded-Prefix\":[\"/Account\"],\"X-Original-Url\":[\"/AccountService/api/Account/Login?zzzzzz=7\"]},\"ContentLength\":73,\"TransferEncoding\":null,\"Host\":\"56.143.43.35:3500\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"56.143.5.95:56370\",\"RequestURI\":\"/api/Account/Login?zzzzzz=7\",\"TLS\":null}"
app_traefik.1.pq75bqlbodnz@LAEDC0000C55    | time="2020-09-11T03:32:23Z" level=debug msg="vulcand/oxy/forward: completed ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.0.3.124:80\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"application/json, text/plain, */*\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"Keep-Alive\"],\"Content-Length\":[\"73\"],\"Content-Type\":[\"application/json\"],\"Max-Forwards\":[\"10\"],\"Postman-Token\":[\"108d2735-e1b6-434e-8a72-a39e90ce1bd9\"],\"Pragma\":[\"no-cache\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36\"],\"X-Arr-Log-Id\":[\"67068c90-ff47-4d13-8e5c-bf5bda40c8fe\"],\"X-Forwarded-For\":[\"112.110.128.34, 112.110.136.126:37228\"],\"X-Forwarded-Prefix\":[\"/Account\"],\"X-Original-Url\":[\"/AccountService/api/Account/Login?zzzzzz=7\"]},\"ContentLength\":73,\"TransferEncoding\":null,\"Host\":\"56.143.43.35:3500\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"56.143.5.95:56370\",\"RequestURI\":\"/api/Account/Login?zzzzzz=7\",\"TLS\":null}"
app_traefik.1.pq75bqlbodnz@LAEDC0000C55    | time="2020-09-11T03:32:23Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/api/Account/Login\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"zzzzzz=7\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"application/json, text/plain, */*\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"Keep-Alive\"],\"Content-Length\":[\"73\"],\"Content-Type\":[\"application/json\"],\"Max-Forwards\":[\"10\"],\"Postman-Token\":[\"108d2735-e1b6-434e-8a72-a39e90ce1bd9\"],\"Pragma\":[\"no-cache\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36\"],\"X-Arr-Log-Id\":[\"67068c90-ff47-4d13-8e5c-bf5bda40c8fe\"],\"X-Forwarded-For\":[\"112.110.128.34, 112.110.136.126:37228\"],\"X-Forwarded-Prefix\":[\"/Account\"],\"X-Original-Url\":[\"/AccountService/api/Account/Login?zzzzzz=7\"]},\"ContentLength\":73,\"TransferEncoding\":null,\"Host\":\"56.143.43.35:3500\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"56.143.5.95:56370\",\"RequestURI\":\"/api/Account/Login?zzzzzz=7\",\"TLS\":null}"

Both requests /api/Account/Login?zzzzzz=6 and /api/Account/Login?zzzzzz=7 were completed in Traefik logs.

However, based on our ASP.NET Core middleware logs, only /api/Account/Login?zzzzzz=7 was seen entering our API service.

Both IIS and the client received a 400 Bad Request response for /api/Account/Login?zzzzzz=6

docker stats reveals normal cpu and memory usage

2

Does the request take too long and maybe Traefik closes the request? Can you measure the request time then maybe? I've had this same issue and needed to add a Buffering middleware

3

Oh the request are not taking that long.
To verify this, we created a new API that does nothing.
After our test, some requests to this new API is still returning as 400 Bad Request response.

How do you know by the way that Traefik closed a request? Based from our Traefik logs, it started and completed forwarding the missing request.

4

Interesting, so this proves it is not application related. I had the same issue as well and in 1.7 with 2 of 10 ASP services. Restarting Traefik sometimes resolved this, we also increased performance on the Swarm manager, and finally, we've also seen Docker running out of IP addresses for their particular stack network