Traefik and gRPC sending to the wrong port using https/h2c

Traefik Traefik v2
1

We have an application listening on gRPC and local testing works fine. This application has an http port for healthchecks and a gRPC port for application data.

The pod yaml:

(sic)
metadata:
  name: myapp
  namespace: default
(sic)
    ports:
    - containerPort: 8080
      protocol: TCP
    - containerPort: 6565
      protocol: TCP
(sic)

The service yaml:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: myapp
  name: myapp
  namespace: default
spec:
  ports:
  - name: app-svc-port
    port: 80
    protocol: TCP
    targetPort: 8080
  - name: app-grpc-port
    port: 6565
    protocol: TCP
    targetPort: 6565
  selector:
    app: myapp
  type: ClusterIP

The IngressRoute yaml for web access:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  labels:
    app: myapp
  name: myapp
  namespace: default
spec:
  routes:
  - kind: Rule
    match: Host(`myapp.example.com`)
    services:
    - kind: Service
      name: myapp
      port: 80
  tls:
    secretName: public-cert

The IngressRoute yaml for gRPC access:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  labels:
    app: myapp
  name: myapp
  namespace: default
spec:
  routes:
  - kind: Rule
    match: Host(`myapp-grpc.example.com`)
    services:
    - kind: Service
      name: myapp
      port: 6565
  tls:
    secretName: public-cert

Doing a query over the browser to myapp-grpc.example.com, I get an html page that is being served from the 8080 port of the pod, not the gRPC endpoint that should be responding.

HTTP/2.0 200 OK
content-type: application/vnd.spring-boot.actuator.v3+json;q=0.8
date: Thu, 27 Feb 2020 21:51:50 GMT
content-length: 15
X-Firefox-Spdy: h2

What I am expecting to see is something similar to this:

ÿÿÿ :aUnexpected HTTP/1.x request: GET /actuator/health

I tried making a different service for each of the two ports of the pod, but the traffic still ends up being sent to port 8080 on the pod rather than 6565.

Traefik version is latest:

time="2020-02-27T21:36:02Z" level=info msg="Traefik version 2.1.4 built on 2020-02-06T17:10:06Z"

as I found this issue: https://github.com/containous/traefik/issues/3131 which was merged when 2.1.3 was release and I assume is inside 2.1.4. When using the endpoint with a gRPC client, I get this response:

evans: failed to run REPL mode: failed to instantiate a new spec: failed to instantiate the spec from proto files: rpc error: code = Unimplemented desc = Not Found: HTTP status code 404; transport: received the unexpected content-type "text/plain; charset=utf-8"
2

Hello,

to disable the content type detection, you have to add the contentType middleware.

https://docs.traefik.io/v2.1/middlewares/contenttype/

3

Thanks for the tip, this definitely seems to be on the right direction. (However, it still doesn't explain why traefik is sending to the wrong port)

I made a middleware document:

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: content-type-disable
  namespace: kube-system

spec:
  contentType:
    autoDetect: false

But the traefik dashboard does not detect the type or any information

Screen Shot 2020-02-28 at 10.04.48
Screen Shot 2020-02-28 at 10.04.481302×1462 41.4 KB

When I add this middleware to an IngressRoute (modifying the one above):

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  labels:
    app: myapp
  name: myapp
  namespace: default
spec:
  routes:
  - kind: Rule
    match: Host(`myapp-grpc.example.com`)
    middlewares:
    - name: content-type-disable
      namespace: kube-system
    services:
    - kind: Service
      name: myapp
      port: 6565
  tls:
    secretName: public-cert

Both the middlware and the ingressroute show errors on the dashboard:

Screen Shot 2020-02-28 at 10.10.36
Screen Shot 2020-02-28 at 10.10.361278×752 56.2 KB

I can't find any source that says this middleware is not supported or a way to set the type manually

1 Like
4

Bump, I'm having the same issue. Any updates?
Thanks a bunch!

I would post screen shots, but they look identical to OP's.
Using : https://docs.traefik.io/middlewares/contenttype/#configuration-examples

---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: autodetect
spec:
  contentType:
    autoDetect: false
5

Are we doing something wrong @ldez :confused:
Thanks again!

6

@ldez :sweat_smile: any chance you got a chance to repro this? simply declaring the crd shows up blank in my dashboard.

image

It appears that the docker provider does recognize the middleware :thinking:

image

version: "3.3"

services:

  traefik:
    image: "traefik:v2.2"
    container_name: "traefik"
    command:
      #- "--log.level=DEBUG"
      - "--api.insecure=true"
      - "--api.dashboard=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
    ports:
      - "80:80"
      - "8080:8080"

    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"

  whoami:
    image: "containous/whoami"
    container_name: "simple-service"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami.rule=Host(`whoami.localhost`)"
      - "traefik.http.routers.whoami.entrypoints=web"
      - "traefik.http.middlewares.autodetect.contenttype.autodetect=false"
7

I have created an issue about this topic as there are no responses orno progress being made: https://github.com/containous/traefik/issues/7005

1 Like
8

Seems this was the way to go, the change that should fix this is merged in master now :slight_smile:

9

For future readers that were stuck on this for an insanely long time as @johnpekcan and I were, these are the fundemental things that must be understood to fully grasp gRPC and traefik:

  1. gRPC that is insecure on the backend will not convert itself to h2c without explicitly notating it with the serverscheme annotatiion on your service, or scheme in service in your IngressRoute.
  2. If your application uses a multi-faceted port to support both HTTP and gRPC you will (currently) have to make separate ingress rules and one will have to match gRPC traffic and others will have to match everything else explicitly.
  3. If you want the easy way out just use an IngressRouteTCP for 2.
  4. The contenttype middleware is not relevant to gRPC whatsoever.
  5. Here is our current Ingress+ingressroute for argocd to accomodate both (generalized):
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.priority: "10"
    traefik.ingress.kubernetes.io/router.tls: "true"
  name: argocd-server
spec:
  rules:
  - host: sub.domain.tld
    http:
      paths:
      - backend:
          serviceName: argocd-server
          servicePort: 80
        path: /
  tls:
  - hosts:
    - sub.domain.tld
    secretName: sub-domain-tld
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: argocd-server-grpc
spec:
  entryPoints:
    - websecure
  routes:
    - kind: Rule
      match: Host(`sub.domain.tld`) && Headers(`Content-Type`, `application/grpc`)
      priority: 11
      services:
        - name: argocd-server
          port: 80
          weight: 10
          scheme: h2c
  tls:
    secretName: sub-domain-tld

This allows both gRPC and regular HTTP/2 calls. The above can be accomplished in a single IngressRoute but cannot be accomplished in a single ingress as per path annotations do not exist nor do Headers matching. I have created a feature request for adding matching: