HTTP/2 (h2c) annotations on Kubernetes Ingress not being applied

Traefik Traefik v2 (latest)
1

Hi,

We are currently running Traefik v1.7 and are looking to migrate to v2.1. In addition to the migration, we are also looking to move some gRPC services which currently haven't been going receiving their requests through Traefik to the new version.

The Kubernetes Ingress YAML definition for one of the gRPC services is as follows:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/protocol: h2c
    kubernetes.io/ingress.class: traefik
  name: foobar-service-grpc
  namespace: foobar-ci
spec:
  rules:
  - host: traefik-grpc-test.foobar.com
    http:
      paths:
      - backend:
          serviceName: foobar-service
          servicePort: pub
        path: /

However, when I try to access this service using HTTP/2, I can see the following error in the logs from the service:

{
  "@timestamp":"2020-02-07T11:49:16.921+1300",
  "message":"Transport failed",
  "priority":"INFO",
  "path":"io.grpc.netty.NettyServerTransport.connections",
  "thread":"grpc-default-worker-ELG-1-1",
  "stack_trace":"io.netty.handler.codec.http2.Http2Exception: Unexpected HTTP/1.x request: POST /grpc.health.v1.Health/Check
...

Which seems to imply that Traefik is downgrading the connection from HTTP/2 to HTTP 1.x.

I've tried connecting to the service directly and can confirm that my client connects successfully.

I've had a look through the documentation and it seems like the way to configure Traefik to use HTTP/2 for the backend is to set the "url" value to something starting with "h2c://...".

What is the correct annotation to do this with the Kubernetes Ingress? Is it even an annotation or something else?

Thank you

2

Hello,

The annotations on Ingress are currently not supported, they will be supported in the next version (v2.2)

1 Like
3

Thanks a lot for your work, I've been waiting for this feature to come. Even considered building a custom Ingress controller to convert Ingresses to IngressRoutes ...

I have one question though (sorry for hijacking the thread) :
How will the kubernetes namespace for middlewares be chosen ? Same as Ingress' namespace ? Other way ? I guess it'll have to change the traefik namespace, as is done in the documentation example ("prefix@kubernetes-crd", by the way, the doc writes "kuberntes-crd", is it a typo ?), so how is the kubernetes namespace decided ?

4

You are mixing to different thing:

To convert Ingress to IngressRoute you use our tool: https://github.com/containous/traefik-migration

The namespace from k8s in IngressRoute respect the classical namespace scope.

5

I don't think so, so I might not have explained clearly enough. Let me rephrase :

When I am/will be using Ingress (and not IngressRoute) and reference Middleware I created with the "Kind: Middleware" CRD that are not in the same kubernetes namespace than the Ingress is in, how should I reference it ?

I understand I have two namespace changes to do : first a Traefik Namespace, since I am using the Ingress provider and want the IngressRoute CRD provider, hence the "@kubernetescrd" suffix but then comes the problem of referencing the Middleware in its Kubernetes Namespace.

Let's say for example I have a Middleware named "foo" in the kubernetes namespace "bar" and an Ingress (the native Kubernetes Object, not the Traefik IngressRoute) in the namespace "default" that uses that middleware "foo"

What should be the value for annotation traefik.ingress.kubernetes.io/router.middlewares ?
Should it simply be foo@kubernetescrd ? bar/foo@kubernetes-crd? something else ? I couldn't find easily the answer in the traefik code (I am not familiar with the codebase) nor in the documentation you linked.

About the conversion : I know of the traefik-migration tool, and I planned to reuse it, but I have some automatically-generated Ingresses in my cluster, and as far as I could tell, the migration tool does not automatically convert these as the ingresses are generated.

6

I guess your are using the master branch because the v2.2 does not currently exist.

So in this case:

  • the provider namespace for the k8s CRD provider is kubernetescrd
  • in the ingress annotations, the provider namespace for middleware is mandatory (because the middleware cannot be created from Ingress annotations.)