Properly annotating Ingress to serve gRPC

We've setup traefik with helm chart version 1.61.0 (because automatic encryption wasn't working with the newest version - see That's a bit beside the point.

  enabled: true
  enforced: false

We've been able to get Let's Encrypt working, and properly access via https and server some html. Cool.

I'm now trying to properly annotate a gRPC service's ingress so that I can successfully request... my data with a python gRPC client. (I've port forwarded to the pod and made the same client request and it servers properly so I know it's something with my traefik configuration...)

This is what my ingress looks like now:

kind: Ingress
  annotations: h2c "true" traefik
    traefik.protocol: h2c
  creationTimestamp: "2019-07-04T08:59:33Z"
  generation: 2
  name: dataservice
  namespace: <my namespace>
  resourceVersion: "21175139"
  selfLink: /apis/extensions/v1beta1/namespaces/<my namespace>/ingresses/dataservice
  uid: 0aa6f4ee-9e3a-11e9-9bb9-9e2c1485c626
  - host: dataservice.api.<my domain>.io
      - backend:
          serviceName: dataservice
          servicePort: 80
        path: /
  loadBalancer: {}

I get a redirect from my request like this... i'm trying to figure out whether I need to add another specific header to my client request or if i've got the wrong idea with my ingress annotations etc etc...

grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
        status = StatusCode.CANCELLED
        details = "Received http2 header with status: 308"

This might be relevant, I just need traefik to forward the request it get's to the backend, but it doesn't appear to work that way...


The annotation traefik.protocol: h2c is not a valid annotation, the right annotation is h2c "true" is not a supported Traefik annotation.

Could give the Traefik log?

1 Like

We've moved along quite a bit, in a few moments i'll share the current set of configuration and also the logs. Thanks!

The basic issue is that we've got a properly working frontend, running https, but when we send requests with our gRPC client to the frontend, it doesn't seem to get through to our gRPC backend...

FYI, the h2c protocol is made for http/2 without TLS.

So once we properly annotated our backend service with h2c, AND AND AND (importantly) fed the proper front end certificate into our python gRPC client then we got through! It'll take a few more cycles but i'll come back with more details.

Ok so here's how we got it to work:

On our gRPC client we had to include the certificate from the traefik served frontend (https secured by acme).

This was the important second argument to grpc.secure_channel --> grpc.ssl_channel_credentials()

import os
import grpc
import echo_pb2
import echo_pb2_grpc
import requests
import time
# import ssl

domain = ''
protected_url = f"https://{domain}/"

channel = grpc.secure_channel(f'{domain}:443', grpc.ssl_channel_credentials())
stub = echo_pb2_grpc.EchoServiceStub(channel)

### build up authentication token here...

res = stub.Echo(echo_pb2.envelope(message="Test 1 2 3 Test"), metadata=[token])
apiVersion: extensions/v1beta1
kind: Ingress
  name: {{ .Values.applicationservicename }}
  annotations: traefik h2c X-Userinfo, X-Id-Token, X-Access-Token, Authorization forward{{ .Values.ingress.authentication.config_token }}
  - hosts: 
    - {{ }}
  - host: {{ }}
      - backend:
          serviceName: {{ .Values.applicationservicename }}
          servicePort: 80
        path: /