Hi,
with our newest application we get some strange behaviour. This application is a Java application in a customized openliberty container with traefik 1.7.26 in front. When sending a request with a body of over ~10Mib we get a 500 http response. With bodies smaller than this, the openliberty container does its job and returns the correct result.
Also could we rule out the application itself. We've got correct results when we accessed the container direct (ports via docker).
time="2020-09-01T06:45:43Z" level=debug msg="Upstream ResponseWriter of type *pipelining.writerWithoutCloseNotify does not implement http.CloseNotifier. Returning dummy channel."
time="2020-09-01T06:45:43Z" level=debug msg="'500 Internal Server Error' caused by: connection error: FLOW_CONTROL_ERROR"
time="2020-09-01T06:45:43Z" level=debug msg="vulcand/oxy/forward/http: Round trip: https://CONTAINER_IP:9443, code: 500, Length: 21, duration: 379.396108ms tls:version: 303, tls:resume:false, tls:csuite:cca8, tls:server:myhost.example.com"
time="2020-09-01T06:45:43Z" level=debug msg="vulcand/oxy/forward/http: completed ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"https\",\"Opaque\":\"\",\"User\":null,\"Host\":\"CONTAINER_IP:9443\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Authorization\":[\"Bearer SECRET\"],\"Connection\":[\"keep-alive\"],\"Content-Length\":[\"13594708\"],\"Content-Type\":[\"multipart/form-data; boundary=--------------------------821139872036740021867662\"],\"User-Agent\":[\"PostmanRuntime/7.26.3\"]},\"ContentLength\":13594708,\"TransferEncoding\":null,\"Host\":\"myhost.example.com:443\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RequestURI\":\"/rest/endpoint/\",\"TLS\":null}"
time="2020-09-01T06:45:43Z" level=debug msg="vulcand/oxy/forward: completed ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"https\",\"Opaque\":\"\",\"User\":null,\"Host\":\"CONTAINER_IP:9443\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate,br\"],\"Authorization\":[\"Bearer SECRET\"],\"Connection\":[\"keep-alive\"],\"Content-Length\":[\"13594708\"],\"Content-Type\":[\"multipart/form-data; boundary=--------------------------821139872036740021867662\"],\"User-Agent\":[\"PostmanRuntime/7.26.3\"]},\"ContentLength\":13594708,\"TransferEncoding\":null,\"Host\":\"myhost.example.com:443\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RequestURI\":\"/rest/endpoint/\",\"TLS\":null}"
time="2020-09-01T06:45:43Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/rest/endpoint/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Authorization\":[\"Bearer SECRET\"],\"Connection\":[\"keep-alive\"],\"Content-Length\":[\"13594708\"],\"Content-Type\":[\"multipart/form-data; boundary=--------------------------821139872036740021867662\"],\"User-Agent\":[\"PostmanRuntime/7.26.3\"]},\"ContentLength\":13594708,\"TransferEncoding\":null,\"Host\":\"myhost.example.com:443\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RequestURI\":\"/rest/endpoint/\",\"TLS\":null}"
time="2020-09-01T06:45:43Z" level=debug msg="vulcand/oxy/buffer: completed ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/rest/endpoint/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Authorization\":[\"Bearer SECRET\"],\"Connection\":[\"keep-alive\"],\"Content-Length\":[\"13594708\"],\"Content-Type\":[\"multipart/form-data; boundary=--------------------------821139872036740021867662\"],\"User-Agent\":[\"PostmanRuntime/7.26.3\"]},\"ContentLength\":13594708,\"TransferEncoding\":null,\"Host\":\"myhost.example.com:443\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RequestURI\":\"/rest/endpoint/\",\"TLS\":null}"
Docker compose file of our application (without environment and volumes):
services:
mysupercoolrestservice:
image: mysupercoolrestservice:latest
restart: unless-stopped
networks:
- webgateway
labels:
traefik.domain: "${PROJECT_FQDN}"
traefik.port: "9443"
traefik.protocol: "https"
traefik.weight: "10"
traefik.frontend.rule: "Host:${PROJECT_FQDN};PathPrefix:/rest/endpoint/, /rest/endpoint2/"
traefik.docker.network: "webgateway"
traefik.enable: "true"
traefik.backend.buffering.maxRequestBodyBytes: 52428800
traefik.backend.buffering.memRequestBodyBytes: 2097152
traefik.backend.buffering.maxResponseBodyBytes: 52428800
traefik.backend.buffering.memResponseBodyBytes: 2097152
traefik.backend.buffering.retryExpression: "IsNetworkError() && Attempts() <= 2"
...
Traefik configuration (all config done with this traefik.toml):
logLevel = "DEBUG"
InsecureSkipVerify = true
defaultEntryPoints = ["https"]
[docker]
domain = "myhost.example.com"
watch = true
exposedbydefault = false
[entryPoints]
[entryPoints.http]
address = ":80"
compress = true
[entryPoints.http.redirect]
entryPoint = "https"
[entryPoints.https]
address = ":443"
compress = true
[entryPoints.https.tls]
minVersion = "VersionTLS11"
cipherSuites = [
"TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305",
"TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305",
"TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
"TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
"TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256",
"TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256",
"TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA",
"TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA",
"TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA",
"TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA",
]
[entryPoints.https.tls.defaultCertificate]
certFile = "/certs/myhost.example.com.cer"
keyFile = "/certs/myhost.example.com.key"
[api]
[metrics]
[metrics.prometheus]
I've run out of ideas on what component may cause such an error.
kind regards