Unable to resolve grpc request- "Received http2 header with status: 500"

Hi,

We are currently running Traefik v1.7 with K8s version 1.21 and are looking to migrate to Traefik v2.7 with corresponding K8s version 1.22.

The Kubernetes Ingress YAML definition for one of the gRPC services is as follows:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/protocol: h2c
    kubernetes.io/ingress.class: traefik-grpc
  labels:
    app: appname
    chart: chart-version
  name: test-internal-ingress
  namespace: prodns
spec:
  rules:
  - host: xyz.com
    http:
      paths:
      - backend:
          service:
            name: int-headless
            port:
                number: 7080
        pathType: ImplementationSpecific

However, when I try to access this service using , we can see the following error in the logs from the service:

begin ServeHttp on request" Request="{"Method":"POST","URL":{"Scheme":"","Opaque":"","User":null,"Host":"","Path":"/api.app-name/Query","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""},"Proto":"HTTP/2.0","ProtoMajor":2,"ProtoMinor":0,"Header":{"Accept-Encoding":["identity,gzip"],"Content-Type":["application/grpc"],"Grpc-Accept-Encoding":["identity,deflate,gzip"],"Grpc-Timeout":["4S"],"Te":["trailers"],"User-Agent":["grpc-python/1.23.0 grpc-c/7.0.0 (manylinux; chttp2; gangnam)"],"X-Forwarded-Host":["xyz.com"],"X-Forwarded-Port":["80"],"X-Forwarded-Proto":["http"],"X-Forwarded-Server":["trp-prod-traefik-evr-data-prod-746545b6fd-bptwq"],"X-Real-Ip":["10.1xx.xxx.xx"]},"ContentLength":-1,"TransferEncoding":null,"Host":"xyz.com","Form":null,"PostForm":null,"MultipartForm":null,"Trailer":null,"RemoteAddr":"10.151.248.46:48507","RequestURI":"/api.app-name/Query","TLS":null}"
trp-prod-traefik-evr-data-prod-746545b6fd-bptwq: time="2022-07-12T10:13:56Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" ForwardURL="http://10.15x.x.x:9080" Request="{"Method":"POST","URL":{"Scheme":"","Opaque":"","User":null,"Host":"","Path":"/api.app-name/Query","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""},"Proto":"HTTP/2.0","ProtoMajor":2,"ProtoMinor":0,"Header":{"Accept-Encoding":["identity,gzip"],"Content-Type":["application/grpc"],"Grpc-Accept-Encoding":["identity,deflate,gzip"],"Grpc-Timeout":["4S"],"Te":["trailers"],"User-Agent":["grpc-python/1.23.0 grpc-c/7.0.0 (manylinux; chttp2; gangnam)"],"X-Forwarded-Host":["xyz.com"],"X-Forwarded-Port":["80"],"X-Forwarded-Proto":["http"],"X-Forwarded-Server":["trp-prod-traefik-evr-data-prod-746545b6fd-bptwq"],"X-Real-Ip":["10.1xx.xxx.xx"]},"ContentLength":-1,"TransferEncoding":null,"Host":"xyz.com","Form":null,"PostForm":null,"MultipartForm":null,"Trailer":null,"RemoteAddr":"10.15x.xxx.xx:48507","RequestURI":"/api.app-name/Query","TLS":null}"
trp-prod-traefik-evr-data-prod-746545b6fd-bptwq: time="2022-07-12T10:13:56Z" level=debug msg="'500 Internal Server Error' caused by: net/http: HTTP/1.x transport connection broken: malformed HTTP response "x00x00x06x04x00x00x00x00x00x00x03x00x00x03xe8""

I've tried connecting to the service directly and can confirm that my client connects successfully.