Internel Server Error v2.1.0

Traefik Traefik v2 (latest)
1

Edit: I upgraded to v2.1.1 from v2.1.0 so something in that release must have fixed it.



We are getting a 500 Internal Server Error when trying to login to any of our web applications. We are able to access the login page fine, but as soon as I hit submit, I receive the 500 error.

We are using v2.1.0 in this cluster while our prod cluster is running version v2.0.5 and handles the request just fine.

Traefik debug log for the failing request

time="2019-12-20T14:38:04Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/UserProfile/Login\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Accept-Language\":[\"en-US,en;q=0.9\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"keep-alive\"],\"Content-Length\":[\"76\"],\"Content-Type\":[\"application/x-www-form-urlencoded\"],\"Cookie\":[\"ASP.NET_SessionId=5zydbsoxedc4512ds1hwjo3t; _ga=GA1.2.901034761.1576678397; _gid=GA1.2.1633676375.1576775652\"],\"Dnt\":[\"1\"],\"Origin\":[\"http://app.example.com\"],\"Pragma\":[\"no-cache\"],\"Referer\":[\"http://app.example.com/UserProfile/Login\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36\"],\"X-Forwarded-Host\":[\"app.example.com\"],\"X-Forwarded-Port\":[\"80\"],\"X-Forwarded-Proto\":[\"http\"],\"X-Forwarded-Server\":[\"traefik-644c566bc6-7wfpt\"],\"X-Real-Ip\":[\"10.120.14.14\"]},\"ContentLength\":76,\"TransferEncoding\":null,\"Host\":\"app.example.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.120.14.14:46520\",\"RequestURI\":\"/UserProfile/Login\",\"TLS\":null}"

time="2019-12-20T14:38:04Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/UserProfile/Login\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Accept-Language\":[\"en-US,en;q=0.9\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"keep-alive\"],\"Content-Length\":[\"76\"],\"Content-Type\":[\"application/x-www-form-urlencoded\"],\"Cookie\":[\"ASP.NET_SessionId=5zydbsoxedc4512ds1hwjo3t; _ga=GA1.2.901034761.1576678397; _gid=GA1.2.1633676375.1576775652\"],\"Dnt\":[\"1\"],\"Origin\":[\"http://app.example.com\"],\"Pragma\":[\"no-cache\"],\"Referer\":[\"http://app.example.com/UserProfile/Login\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36\"],\"X-Forwarded-Host\":[\"app.example.com\"],\"X-Forwarded-Port\":[\"80\"],\"X-Forwarded-Proto\":[\"http\"],\"X-Forwarded-Server\":[\"traefik-644c566bc6-7wfpt\"],\"X-Real-Ip\":[\"10.120.14.14\"]},\"ContentLength\":76,\"TransferEncoding\":null,\"Host\":\"app.example.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.120.14.14:46520\",\"RequestURI\":\"/UserProfile/Login\",\"TLS\":null}" ForwardURL="http://10.42.2.19:80"

time="2019-12-20T14:38:04Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/UserProfile/Login\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Accept-Language\":[\"en-US,en;q=0.9\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"keep-alive\"],\"Content-Length\":[\"76\"],\"Content-Type\":[\"application/x-www-form-urlencoded\"],\"Cookie\":[\"ASP.NET_SessionId=5zydbsoxedc4512ds1hwjo3t; _ga=GA1.2.901034761.1576678397; _gid=GA1.2.1633676375.1576775652\"],\"Dnt\":[\"1\"],\"Origin\":[\"http://app.example.com\"],\"Pragma\":[\"no-cache\"],\"Referer\":[\"http://app.example.com/UserProfile/Login\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36\"],\"X-Forwarded-Host\":[\"app.example.com\"],\"X-Forwarded-Port\":[\"80\"],\"X-Forwarded-Proto\":[\"http\"],\"X-Forwarded-Server\":[\"traefik-644c566bc6-7wfpt\"],\"X-Real-Ip\":[\"10.120.14.14\"]},\"ContentLength\":76,\"TransferEncoding\":null,\"Host\":\"app.example.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.120.14.14:46520\",\"RequestURI\":\"/UserProfile/Login\",\"TLS\":null}"

10.120.14.14 - - [20/Dec/2019:14:38:04 +0000] "POST /UserProfile/Login HTTP/1.1" 500 22 "-" "-" 87728 "app-7b9ac3a3c653d7b56128@kubernetescrd" "http://10.42.2.19:80" 0ms

CURL output for a login request

curl -v -X POST -F 'Login=me;Password=mine;btnLogin=Sign In' https://app.example.com/UserProfile/Login
* About to connect() to app.example.com port 443 (#0)
*   Trying 1.2.3.4...
* Connected to app.example.com (1.2.3.4) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*       subject: CN=app.example.com
*       start date: Dec 19 12:41:56 2019 GMT
*       expire date: Mar 18 12:41:56 2020 GMT
*       common name: app.example.com
*       issuer: CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US
> POST /UserProfile/Login HTTP/1.1
> User-Agent: curl/7.29.0
> Host: app.example.com
> Accept: */*
> Content-Length: 210
> Expect: 100-continue
> Content-Type: multipart/form-data; boundary=----------------------------a40e7c84fa98
> 
< HTTP/1.1 500 Internal Server Error
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Fri, 20 Dec 2019 14:29:48 GMT
< Content-Length: 22
< Connection: close
< 
Internal Server Error
* Closing connection 0

IngressRoute Definition

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: app
  namespace: app
spec:
  entryPoints:
    - http
  routes:
    - match: Host(`app.example.com`)
      kind: Rule
      services:
        - name: app
          port: 80