500 errors with HTTP/1.1

Traefik Traefik v2
1

Hello,

We have a websocket that has to be opened between our webapplication and our backend.
The network goes naturally through Traefik and is redirected to our backend.

Even if it seems to work quite well we are facing a lot of 500 errors on Traefik regarding those websockets...

In fact, it seems that all websocket opened (around 220k per 24 hours) finished with a 500 error at the end...

Does anyone faces the same issue or have an idea how to fix it ?

Thanks in advance for your help.

Grég

2

Hey @djbryce , can you attach some logs + the Traefik static config and dynamic config for that particular router / application?

That will help anyone looking at this thread for generic 500 errors to filter if it can be related to their use case or not :slight_smile:

3

Hello @douglasdtm yes, you are right!

Here are some more informations.

One log with 500 errors:

{
   "ClientAddr":"10.32.36.6:57636",
   "ClientHost":"100.64.176.31",
   "ClientPort":"57636",
   "ClientUsername":"-",
   "DownstreamContentSize":0,
   "DownstreamStatus":500,
   "Duration":60365026731,
   "OriginContentSize":0,
   "OriginDuration":60364505633,
   "OriginStatus":500,
   "Overhead":521098,
   "RequestAddr":"xxxxx-prod.gce.kube.xxx.ch",
   "RequestContentSize":0,
   "RequestCount":9329536,
   "RequestHost":"xxxx-prod.gce.kube.xxxx.ch",
   "RequestMethod":"GET",
   "RequestPath":"/api/livechat/de",
   "RequestPort":"-",
   "RequestProtocol":"HTTP/1.1",
   "RequestScheme":"http",
   "RetryAttempts":0,
   "RouterName":"xxx@file",
   "ServiceAddr":"xxx-api-prod.gce-prod.svc.cluster.local",
   "ServiceName":"xxxx@file",
   "ServiceURL":{
      "Scheme":"http",
      "Opaque":"",
      "User":null,
      "Host":"xxxx-api-prod.gce-prod.svc.cluster.local",
      "Path":"",
      "RawPath":"",
      "ForceQuery":false,
      "RawQuery":"",
      "Fragment":"",
      "RawFragment":""
   },
   "StartLocal":"2022-05-31T14:13:09.123680848+02:00",
   "StartUTC":"2022-05-31T12:13:09.123680848Z",
   "entryPointName":"default",
   "level":"info",
   "msg":"",
   "request_Accept-Encoding":"gzip, deflate, br",
   "request_Accept-Language":"de,de-DE;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6",
   "request_Cache-Control":"no-cache",
   "request_Connection":"upgrade",
   "request_Origin":"https://xxxxxxx.xxxxxxx.ch",
   "request_Pragma":"no-cache",
   "request_Sec-Websocket-Extensions":"permessage-deflate; client_max_window_bits",
   "request_Sec-Websocket-Key":"fW4vbC/EBdPIjfbT4InPxw==",
   "request_Sec-Websocket-Version":"13",
   "request_Upgrade":"websocket",
   "request_User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Safari/537.36 Edg/101.0.1210.53",
   "request_X-Forwarded-For":"100.64.176.31",
   "request_X-Forwarded-Host":"xxxxx-prod.gce.kube.xxxxx.ch",
   "request_X-Forwarded-Port":"443",
   "request_X-Forwarded-Proto":"https",
   "request_X-Forwarded-Scheme":"https",
   "request_X-Forwarded-Server":"xxxxxx-prod-84d97cfdd-hs92w",
   "request_X-Original-Forwarded-For":"83.150.38.10",
   "request_X-Real-Ip":"100.64.176.31",
   "request_X-Request-Id":"43d4bf1f4527a60bd82fda65abbbb406",
   "request_X-Scheme":"https",
   "time":"2022-05-31T14:14:09+02:00"
}

The Traefik config file:

global:
  checkNewVersion: false
  sendAnonymousUsage: false
serversTransport:
  insecureSkipVerify: true
  forwardingTimeouts:
    idleConnTimeout: 30s
log:
  format: json
accessLog:
  format: json
  fields:
    defaultMode: keep
    headers:
      defaultMode: keep
      names:
          Authorization: drop
          cookie: drop
          x-correlation-id: keep
entryPoints:
  default:
    address: ':8080'
    forwardedHeaders:
      insecure: true
  metrics:
    address: ':8082'
providers:
  file:
    directory: /opt/app/config
    watch: true
metrics:
  prometheus:
    entryPoint: metrics

A middleware in use:

http:
  middlewares:
    addApi:
      addprefix:
        prefix: /api
    maintenance:
      addprefix:
        prefix: /maintenance
    nullRouteReplace:
      replacepathregex:
        regex: ^.*
        replacement: /
    limit:
      buffering:
        maxRequestBodyBytes: 8000000

The services config:

http:
  services:
    auth:
      loadBalancer:
        servers:
          - url: 'http://authentication-api-prod.gce-prod.svc.cluster.local'
    mobileGW:
      loadBalancer:
        servers:
          - url: 'http://xxx-rest-api-prod.gce-xxx.svc.cluster.local'
    extranetAssure:
      loadBalancer:
        servers:
          - url: 'http://xxxx-prod.gce-xxx.svc.cluster.local'
    cms:
      loadBalancer:
        passHostHeader: false
        servers:
          - url: 'http://cms-agent-xxx.xx.xxx'
    assuresApi:
      loadBalancer:
        servers:
          - url: 'http://xxx-api-prod.xxx-prod.svc.cluster.local'

The router config:

#@ load("@ytt:data", "data")
#@ def routers():
---
http:
  routers:
    #@ if/end data.values.maintenance:
    authWeb:
      rule: PathPrefix(`/auth/web`)
      middlewares:
        - maintenance
      service: extranetAssure
    auth:
      rule: PathPrefix(`/auth`)
      service: auth
    apiIntrospectionDeny:
      rule: Path(`/mobile-gw/swagger-ui.html`) || PathPrefix(`/api/swagger`) || ( Method(`GET`) && PathPrefix(`/graphql`) )
      priority: 10
      middlewares:
        - nullRouteReplace
      service: mobileGW
    mobileGW:
      rule: PathPrefix(`/mobile-gw`)
      service: mobileGW
      priority: 2
      middlewares:
        - limit
    assuresApi:
      rule: PathPrefix(`/api`) || PathPrefix(`/graphql`)
      service: assuresApi
      priority: 3
      middlewares:
        - limit
    cms:
      rule: PathPrefix(`/content`) || PathPrefix(`/etc`)
      service: cms
    extranetAssure:
      rule: PathPrefix(`/`)
      service: extranetAssure
      priority: 1
      #@ if/end data.values.maintenance:
      middlewares:
        - maintenance
#@ end

Let me know if I can add some more missing information

Thanks