Failure from Traefik moving grpc

Traefik Traefik v2
1

I have the following setup:
.Net client -> Traefik (kubernetes) -> .Net grpc server

I started see failures, and found this in the logs:

2024/02/12 21: 54: 24 transport.go: 3047: protocol error: received *http2.GoAwayFrame before a SETTINGS frame
{
    "level": "debug",
    "msg": "'500 Internal Server Error' caused by: connection error: PROTOCOL_ERROR",
    "time": "2024-02-12T21:54:24Z"
}

I then built a smaller .net server with the exact same proto (all build parameters the same between the two), and that works. I am wondering where I should be looking to diagnose this?

Traefik v2.10.6

Thanks,

2

What are you trying to achieve? Use Traefik for debugging? Enable Traefik log in debug level, and access log in JSON format.

3

I am trying to find out where to go for diagnosing the issue. Traefik is in debug, what I posted above is all I get. All I get from those logs which actually show the invocation is a 500 error - consistent with the error from Traefik above.

4

Debug says error 500, so it‘s probably sent from the target server. Enable access log in JSON will tell you if the error was sent by target service.

5

I doubt that. If I create a Kubernetes service that goes to my grpc backend directly, it works fine so it seems to me it is Traefik causing the issue.

Is there a way beyond the debug logs I can see what causes Traefik to fail?

6

Check for OriginStatus and DownstreamStatus.

7

Both 500.

{
    "ClientAddr": "192.168.65.3:51494",
    "ClientHost": "192.168.65.3",
    "ClientPort": "51494",
    "ClientUsername": "-",
    "DownstreamContentSize": 21,
    "DownstreamStatus": 500,
    "Duration": 3172960,
    "OriginContentSize": 21,
    "OriginDuration": 3038462,
    "OriginStatus": 500,
    "Overhead": 134498,
    "RequestAddr": "bar.mydomain.com",
    "RequestContentSize": 22,
    "RequestCount": 27570,
    "RequestHost": "bar.mydomain.com",
    "RequestMethod": "POST",
    "RequestPath": "/UpdateService/UpdateStateVector",
    "RequestPort": "-",
    "RequestProtocol": "HTTP/2.0",
    "RequestScheme": "https",
    "RetryAttempts": 0,
    "RouterName": "bar-state-grpc-route-d3da5a381d9d86fa66df@kubernetescrd",
    "ServiceAddr": "10.1.6.197:8081",
    "ServiceName": "bar-state-grpc-route-d3da5a381d9d86fa66df@kubernetescrd",
    "ServiceURL": {
        "Scheme": "h2c",
        "Opaque": "",
        "User": null,
        "Host": "10.1.6.197:8081",
        "Path": "",
        "RawPath": "",
        "OmitHost": false,
        "ForceQuery": false,
        "RawQuery": "",
        "Fragment": "",
        "RawFragment": ""
    },
    "StartLocal": "2024-02-12T21:54:24.078718161Z",
    "StartUTC": "2024-02-12T21:54:24.078718161Z",
    "TLSCipher": "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
    "TLSVersion": "1.2",
    "entryPointName": "websecure",
    "level": "info",
    "msg": "",
    "time": "2024-02-12T21:54:24Z"
}
8

That means the 500 is sent from your target service and just "returned" by Traefik to the client.

Look into your target service log, why it returns status 500.

You could also search this forum for "gRPC", I think there were discussions about setting the protocol between Traefik and target service, like force http 1.0 or http/2.