End to End TLS in gRPC on Kubernetes

I have setup k3s node with deployed a gRPC service instance. No idea why the gRPC service not work properly in Fraefik with TLS.

Tested in below:

gRPC client -> Fraefik(v1.7.3) -> gRPC service: not working, gRPC service are able to receive client request.
gRPC client -> gRPC service: working fine, no any issue.

Also, tried two cases on Ingress annotation

Case 1 - using annotation "ingress.kubernetes.io/protocol: h2c"
It got below error on gRPC service

ssl_transport_security.cc:1238] Handshake failed with fatal error SSL_ERROR_SSL: error:1408F10B:SSL routines:ssl3_get_record:**wrong version number.**

Case 2 - using annotation "ingress.kubernetes.io/protocol: https"
It got below error on gRPC service

1 ssl_transport_security.cc:1238] Handshake failed with fatal error SSL_ERROR_SSL: error:14094412:SSL routines:ssl3_read_bytes:sslv3 **alert bad certificate.**

Anyone could advice for above issue?

Below is the Kubernetes ingress setting

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: grpcTest
  namespace: default
  annotations:
    kubernetes.io/ingress.class: traefik
    ingress.kubernetes.io/protocol: https
spec:
  rules:
  - host: grpc.k3s
    http:
      paths:
      - backend:
          serviceName: grpc-service
          servicePort: https
  tls:
   - secretName: k3s.cred

Your gRPC service needs to provide a certificate that contains the IP SAN, as Traefik connects to dynamic backends via IP, not hostname.

This is because PKI relies on DNS and IP for a basis.

Having a backend with a certificate for example.com will not work in this manner.

Also, h2c does not use TLS: https://http2.github.io/http2-spec/#versioning