How to set up Traefik v2 with gRPC and docker

Traefik Traefik v2
1

Tried posting this to stackoverflow with to no avail. (I'm new to traefik so apologies if I missed something obvious).

I have followed the instructions for the TLS challenge and read through the grpc guide for traefik but I can't figure out how to put them together.

Currently I have the traefik dashboard at my domain working, and I could get a http server working, but I can't get the grpc service to be reachable. It shows up under HTTP Services in the dashboard, but when I attempt to hit the endpoint with a request it just times out saying it's unreachable. If I access it directly (via port mapping in docker-compose) it works, so I know this must be traefik related and not firewall rule related.

my docker-compose (I don't have a TOML file):

traefik:
    image: "traefik:v2.0.0"
    container_name: traefik
    command:
      - --entrypoints.web.address=:80
      - --entrypoints.websecure.address=:443
      - --entrypoints.grpc.address=:8090
      - --providers.docker
      - --api
      # Lets Encrypt Resolvers
      - --certificatesresolvers.leresolver.acme.email=${EMAIL}
      - --certificatesresolvers.leresolver.acme.storage=/etc/acme/cert.json
      - --certificatesresolvers.leresolver.acme.tlschallenge=true
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "/etc/acme/:/etc/acme/"
    labels:
      # Dashboard
      - "traefik.http.routers.traefik.rule=Host(`traefik.${DOMAIN}`)"
      - "traefik.http.routers.traefik.service=api@internal"
      - "traefik.http.routers.traefik.tls.certresolver=leresolver"
      - "traefik.http.routers.traefik.entrypoints=websecure"

      # Auth
      - "traefik.http.routers.traefik.middlewares=authtraefik"
      - "traefik.http.middlewares.authtraefik.basicauth.users=admin:xxx"

      # global redirect to https
      - "traefik.http.routers.http-catchall.rule=hostregexp(`{host:.+}`)"
      - "traefik.http.routers.http-catchall.entrypoints=web"
      - "traefik.http.routers.http-catchall.middlewares=redirect-to-https"

      # middleware redirect
      - "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"
    ports:
      - 80:80
      - 443:443
      - 8090:8090
    networks:
      - internal
      - proxied

  grpc_server:
    image: xxx
    container_name: grpc_server
    labels:
      - "traefik.http.routers.grpc_server.rule=Host(`grpc.${DOMAIN}`)"
      - "traefik.http.routers.grpc_server.entrypoints=grpc"
      - "traefik.http.routers.grpc_server.tls=true"
      - "traefik.http.routers.grpc_server.tls.certresolver=leresolver"
    expose:
      - 8090
   # The grpc server works when I uncomment the following line and remove the mapping from traefik.
   # ports: 
   #  - 8090:8090

I don't need the layer from traefik to grpc to be encrypted which is why I haven't set up the self signed cert as per the grpc example. My grpc service is running in insecure mode and works when not behind traefik.

Anything obvious that I missed?

2

I wonder if there is some test gRPC setup that would help reproducing issues grpc_server containers is obviously private, so one cannot up your configuration. Also not clear what clients could be used fro grpc testing. Would be nice if there were a grpc client/server container based repo, that we could use to reproduce such type of problems. (Same goes about websocket issues)

3

+1 with @zespri!

Also:

=> if you expect grpc to be unencrypted, shouldn't you disable tls (and the resolver) from the labels of the service?

4

Perhaps naively, but I expect it to be encrypted until traefik and then insecure between traefik and the container. I'll see if I can enable the self signed cert to see if that is the issue.

5

Ok @TannerYoung, thanks for the feedback, I wanted to be sure to get the intent before providing more information, to avoid sending you on a bad direction :slight_smile:

From the Traefik gRpc user guide,
my understanding is that Traefik has to dial with the grpc backend using the protocol h2c.

As the warning message, at the end of https://docs.traefik.io/v2.0/user-guides/grpc/#traefik-configuration, says: you are missing a label on your backend to tell Traefik to use h2c:
traefik.http.services.<my-service-name>.loadbalancer.server.scheme=h2c.

Also, just in doubt, you should explicit the port to use for communicating with the backend (Traefik auto-detects it when there is one, and only on exposed port, but better making this setting explicit to avoid any further issue).

Try the following:

# docker-compose.yml
# ...
grpc_server:
    image: xxx
    container_name: grpc_server
    labels:
      - traefik.http.routers.grpc_server.rule=Host(`grpc.${DOMAIN}`)
      - traefik.http.routers.grpc_server.entrypoints=grpc
      - traefik.http.routers.grpc_server.tls=true
      - traefik.http.routers.grpc_server.tls.certresolver=leresolver
      - traefik.http.services.grpc-svc.loadbalancer.server.scheme=h2c
      - traefik.http.services.grpc-svc.loadbalancer.server.port=8090
    expose:
      - 8090

About the self signed certificate of the grpc backend application, don't forget to add it to the rootCas colelction of Traefik, as explained in the user guide:
https://docs.traefik.io/v2.0/user-guides/grpc/#traefik-configuration_1

6

I've changed tactics for simplicity and enabled https on the grpc service with a self signed cert and passed it to traefik with the rootCa

I've added

grpc_server:
    image: xxx
    container_name: grpc_server
    labels:
      - traefik.http.routers.grpc_server.rule=Host(`grpc.${DOMAIN}`)
      - traefik.http.routers.grpc_server.entrypoints=grpc
      - traefik.http.routers.grpc_server.tls=true
      - traefik.http.routers.grpc_server.tls.certresolver=leresolver
      # https
      - "traefik.http.services.grpc-svc.loadbalancer.server.url=https://grpc.${DOMAIN}:8090"
    expose:
      - 8090

It doesn't seem like I should need to set up the frontend cert because the TLS challenge should handle that, correct? Yet this still doesn't work and doesn't reach my service via traefik, even if it works over ssl without traefik.

I really need something to get me unblocked here, and I'm not sure how to do it. I can't run it without traefik because i need more than a self signed cert, yet I can't figure out how to configure traefik to route to my service unless I turn TLS off entirely.

7

Perhaps it's how i'm signing the self signed cert within docker?

End to End TLS in gRPC on Kubernetes seems to indicate

"Your gRPC service needs to provide a certificate that contains the IP SAN, as Traefik connects to dynamic backends via IP, not hostname."

Perhaps within docker it needs to use the IP address or container name instead of the host name?

8

It is because the label traefik.http.services.grpc-svc.loadbalancer.server.url does not exist.

Please check the documentation at https://docs.traefik.io/v2.0/routing/providers/docker/ where you'll need 2 labels: loadbalancer.server.port and loadbalancer.server.scheme :slight_smile:

9

@TannerYoung did you find any solution ?

I can't call my grpc service via traefik. I am testing locally using docker-compose.
I have add more details here: DNS resolution failed while trying to access GRPC container via BloomRPC

Any help ?