Issues with gRPC and multiple instances

Hi all,
I need to solve a question. What exactly am I doing wrong?
I have this config file:

http:
  middlewares:
    grpcfix:
      stripPrefix:
        prefixes:
          - "/grpc/"

  routers:

    routerGrpc:
      rule: "Host(`www.local.dgraph.com`) && Path(`/grpc/`)" # Tested also without Path
      service: dgraph-grpc
      middlewares:
      - grpcfix

  services:

    dgraph-grpc:
      loadBalancer:
        servers:
        - url: h2c://localhost:9081
        - url: h2c://localhost:9082
        - url: h2c://localhost:9083
        - url: h2c://localhost:9084
        - url: h2c://localhost:9085
        - url: h2c://localhost:9086

My problem is:
1 - I can not connect the instance via gRPC to the path host_url/grpc/.
2 - I can only connect via host_url with the port of any instance. Which sounds like a bug to me.
3 - Even being able to access an instance via host_url + PORT yet there is no balancing.

That’s weird, it’s like Traefik was just redirecting my instance to the URL it administers. It does not balance or anything.

Does anybody have any idea?

Cheers.

Anybody? It would be interesting for someone to help me with this. I would recommend Traefik. However, without this gRPC thing, I can't.

PS. I was able to do it with NGINX. And it was quite easy. It should be with Traefik.

PS. I think the flag is wrong. Don't remmember if I choosed that one.

Hello @MichelDiz,

You have tagged this issue as Maesh, not Traefik, so it is in a different category.

You may want to update the tags to move this to the correct forum.

Thanks!

Yeah, that's odd cuz i don't even know what Maesh is. I was trying to edit this, but seems like we can edit flags after create the topic.

Btw, do you know some one who worked in the grpc support?

Cheers.

Hello @MichelDiz,

The Rule:

rule: "Host(`www.local.dgraph.com`) && Path(`/grpc/`)" 

would match www.local.dgraph.com/grpc/ as an exact match only.

Also, can you provide your Traefik configuration, and an example of the gRPC error you get when connecting?

But it doesn't tho. Not in my env.

There's no error. Actually it works, but not as it should. As I've mentioned previously.

To reproduce it, just clone the repo I've shared.
Run sh ./dgraph.sh --multiple
Edit the dynamic_conf.yml according to what I mentioned above

Then run sh ./traefik.sh

then you need to run a Dgraph client.
This fits the context https://github.com/MichelDiz/ItisTimetoReproduce/tree/master/reproduce003/grpc_client

You gonna need to edit the ClientStub from "localhost:9001" to www.local.dgraph.com/grpc/

So, it won't work. But, if you put www.local.dgraph.com:9081 it works. 9081 is the Dgraph's GRPC port. This means that Traefik is passing the Port directly and there is no balancing.

Cheers.

up+
need to know if it is possible

I have give another try. This time using k8s.

All HTTP works fine. Not the GRPC one.

Outside the cluster I got

curl -v  --insecure http://grpc.dgraph.me     
* Rebuilt URL to: http://grpc.dgraph.me/
*   Trying 192.168.10.86...
* TCP_NODELAY set
* Connected to grpc.dgraph.me (192.168.10.86) port 80 (#0)
> GET / HTTP/1.1
> Host: grpc.dgraph.me
> User-Agent: curl/7.54.0
> Accept: */*
> 
< HTTP/1.1 500 Internal Server Error
< Date: Wed, 26 Feb 2020 10:41:31 GMT
< Content-Length: 21
< Content-Type: text/plain; charset=utf-8
< 
* Connection #0 to host grpc.dgraph.me left intact
Internal Server Error%

And from a JS client I got

node index.js
(node:23320) UnhandledPromiseRejectionWarning: Error: 1 CANCELLED: Received http2 header with status: 404

Inside the cluster I got

curl -v  --insecure 10.42.0.21:9080
*   Trying 10.42.0.21:9080...
* TCP_NODELAY set
* Connected to 10.42.0.21 (10.42.0.21) port 9080 (#0)
> GET / HTTP/1.1
> Host: 10.42.0.21:9080
> User-Agent: curl/7.67.0
> Accept: */*
> 
* Received HTTP/0.9 when not allowed

* Closing connection 0
curl: (1) Received HTTP/0.9 when not allowed

Apparently, internally it works. But not externally.

What am I missing?

Cheers.

PS. I have used the HELM installation. Traefik 2.1.3

Logs

time="2020-02-26T10:15:49Z" level=info msg="Configuration loaded from flags."
time="2020-02-26T10:16:58Z" level=error msg="kubernetes service not found: default/dgraph-ratel" ingress=dgraph-ratel-ingress namespace=default providerName=kubernetescrd
time="2020-02-26T10:16:58Z" level=error msg="kubernetes service not found: default/dgraph-ratel" ingress=dgraph-ratel-ingress namespace=default providerName=kubernetescrd
time="2020-02-26T10:16:58Z" level=error msg="kubernetes service not found: default/dgraph-alpha" ingress=dgraph-http-ingress providerName=kubernetescrd namespace=default
time="2020-02-26T10:16:58Z" level=error msg="kubernetes service not found: default/dgraph-ratel" ingress=dgraph-ratel-ingress namespace=default providerName=kubernetescrd
time="2020-02-26T10:16:58Z" level=error msg="kubernetes service not found: default/dgraph-alpha" providerName=kubernetescrd ingress=dgraph-http-ingress namespace=default
time="2020-02-26T10:16:58Z" level=error msg="kubernetes service not found: default/dgraph-alpha-grpc" ingress=dgraph-grpc-ingress namespace=default providerName=kubernetescrd
time="2020-02-26T10:16:58Z" level=error msg="kubernetes service not found: default/dgraph-alpha-grpc" providerName=kubernetescrd ingress=dgraph-grpc-ingress namespace=default
time="2020-02-26T10:16:58Z" level=error msg="kubernetes service not found: default/dgraph-ratel" providerName=kubernetescrd ingress=dgraph-ratel-ingress namespace=default
time="2020-02-26T10:16:58Z" level=error msg="kubernetes service not found: default/dgraph-alpha" namespace=default providerName=kubernetescrd ingress=dgraph-http-ingress
time="2020-02-26T10:16:58Z" level=error msg="kubernetes service not found: default/dgraph-alpha" namespace=default providerName=kubernetescrd ingress=dgraph-http-ingress
time="2020-02-26T10:16:58Z" level=error msg="kubernetes service not found: default/dgraph-alpha-grpc" providerName=kubernetescrd ingress=dgraph-grpc-ingress namespace=default
time="2020-02-26T10:16:58Z" level=error msg="kubernetes service not found: default/dgraph-ratel" ingress=dgraph-ratel-ingress namespace=default providerName=kubernetescrd
time="2020-02-26T10:16:58Z" level=error msg="kubernetes service not found: default/dgraph-ratel" ingress=dgraph-ratel-ingress namespace=default providerName=kubernetescrd
time="2020-02-26T10:16:58Z" level=error msg="endpoints not found for default/dgraph-alpha" namespace=default providerName=kubernetescrd ingress=dgraph-http-ingress
time="2020-02-26T10:16:58Z" level=error msg="kubernetes service not found: default/dgraph-alpha-grpc" providerName=kubernetescrd ingress=dgraph-grpc-ingress namespace=default
time="2020-02-26T10:16:58Z" level=error msg="kubernetes service not found: default/dgraph-ratel" providerName=kubernetescrd ingress=dgraph-ratel-ingress namespace=default
time="2020-02-26T10:16:58Z" level=error msg="subset not found for default/dgraph-alpha" namespace=default providerName=kubernetescrd ingress=dgraph-http-ingress
time="2020-02-26T10:16:58Z" level=error msg="kubernetes service not found: default/dgraph-alpha-grpc" ingress=dgraph-grpc-ingress namespace=default providerName=kubernetescrd
time="2020-02-26T10:16:58Z" level=error msg="kubernetes service not found: default/dgraph-ratel" ingress=dgraph-ratel-ingress namespace=default providerName=kubernetescrd
time="2020-02-26T10:16:58Z" level=error msg="subset not found for default/dgraph-alpha" providerName=kubernetescrd ingress=dgraph-http-ingress namespace=default
time="2020-02-26T10:16:58Z" level=error msg="endpoints not found for default/dgraph-alpha-grpc" providerName=kubernetescrd ingress=dgraph-grpc-ingress namespace=default
time="2020-02-26T10:16:58Z" level=error msg="endpoints not found for default/dgraph-ratel" providerName=kubernetescrd ingress=dgraph-ratel-ingress namespace=default
time="2020-02-26T10:16:58Z" level=error msg="subset not found for default/dgraph-alpha" providerName=kubernetescrd ingress=dgraph-http-ingress namespace=default
time="2020-02-26T10:16:58Z" level=error msg="subset not found for default/dgraph-alpha-grpc" namespace=default providerName=kubernetescrd ingress=dgraph-grpc-ingress
time="2020-02-26T10:16:58Z" level=error msg="subset not found for default/dgraph-alpha-grpc" providerName=kubernetescrd ingress=dgraph-grpc-ingress namespace=default
time="2020-02-26T10:16:58Z" level=error msg="endpoints not found for default/dgraph-ratel" providerName=kubernetescrd namespace=default ingress=dgraph-ratel-ingress
time="2020-02-26T10:16:58Z" level=error msg="subset not found for default/dgraph-alpha" providerName=kubernetescrd ingress=dgraph-http-ingress namespace=default
time="2020-02-26T10:16:58Z" level=error msg="subset not found for default/dgraph-alpha" providerName=kubernetescrd ingress=dgraph-http-ingress namespace=default
time="2020-02-26T10:16:58Z" level=error msg="subset not found for default/dgraph-alpha-grpc" providerName=kubernetescrd ingress=dgraph-grpc-ingress namespace=default
time="2020-02-26T10:16:58Z" level=error msg="subset not found for default/dgraph-ratel" providerName=kubernetescrd ingress=dgraph-ratel-ingress namespace=default
time="2020-02-26T10:17:00Z" level=error msg="subset not found for default/dgraph-alpha-grpc" providerName=kubernetescrd ingress=dgraph-grpc-ingress namespace=default
time="2020-02-26T10:17:00Z" level=error msg="subset not found for default/dgraph-ratel" ingress=dgraph-ratel-ingress namespace=default providerName=kubernetescrd
time="2020-02-26T10:17:00Z" level=error msg="subset not found for default/dgraph-alpha" providerName=kubernetescrd ingress=dgraph-http-ingress namespace=default
time="2020-02-26T10:17:02Z" level=error msg="subset not found for default/dgraph-alpha-grpc" providerName=kubernetescrd ingress=dgraph-grpc-ingress namespace=default
time="2020-02-26T10:17:02Z" level=error msg="subset not found for default/dgraph-ratel" providerName=kubernetescrd ingress=dgraph-ratel-ingress namespace=default
time="2020-02-26T10:17:02Z" level=error msg="subset not found for default/dgraph-alpha" providerName=kubernetescrd ingress=dgraph-http-ingress namespace=default
time="2020-02-26T10:17:04Z" level=error msg="subset not found for default/dgraph-ratel" providerName=kubernetescrd ingress=dgraph-ratel-ingress namespace=default
time="2020-02-26T10:17:04Z" level=error msg="subset not found for default/dgraph-alpha" providerName=kubernetescrd ingress=dgraph-http-ingress namespace=default
time="2020-02-26T10:17:04Z" level=error msg="subset not found for default/dgraph-alpha-grpc" namespace=default providerName=kubernetescrd ingress=dgraph-grpc-ingress
time="2020-02-26T10:17:06Z" level=error msg="subset not found for default/dgraph-ratel" providerName=kubernetescrd ingress=dgraph-ratel-ingress namespace=default
time="2020-02-26T10:17:06Z" level=error msg="subset not found for default/dgraph-alpha" ingress=dgraph-http-ingress namespace=default providerName=kubernetescrd
time="2020-02-26T10:17:06Z" level=error msg="subset not found for default/dgraph-alpha-grpc" ingress=dgraph-grpc-ingress providerName=kubernetescrd namespace=default
time="2020-02-26T10:17:08Z" level=error msg="subset not found for default/dgraph-alpha-grpc" namespace=default providerName=kubernetescrd ingress=dgraph-grpc-ingress
time="2020-02-26T10:17:08Z" level=error msg="subset not found for default/dgraph-ratel" providerName=kubernetescrd ingress=dgraph-ratel-ingress namespace=default
time="2020-02-26T10:17:08Z" level=error msg="subset not found for default/dgraph-alpha" ingress=dgraph-http-ingress namespace=default providerName=kubernetescrd
time="2020-02-26T10:17:10Z" level=error msg="subset not found for default/dgraph-ratel" ingress=dgraph-ratel-ingress providerName=kubernetescrd namespace=default
time="2020-02-26T10:17:10Z" level=error msg="subset not found for default/dgraph-alpha" ingress=dgraph-http-ingress namespace=default providerName=kubernetescrd
time="2020-02-26T10:17:10Z" level=error msg="subset not found for default/dgraph-alpha-grpc" providerName=kubernetescrd ingress=dgraph-grpc-ingress namespace=default
time="2020-02-26T10:17:12Z" level=error msg="subset not found for default/dgraph-ratel" providerName=kubernetescrd ingress=dgraph-ratel-ingress namespace=default
time="2020-02-26T10:17:12Z" level=error msg="subset not found for default/dgraph-alpha" providerName=kubernetescrd namespace=default ingress=dgraph-http-ingress
time="2020-02-26T10:17:12Z" level=error msg="subset not found for default/dgraph-alpha-grpc" providerName=kubernetescrd ingress=dgraph-grpc-ingress namespace=default
time="2020-02-26T10:17:14Z" level=error msg="subset not found for default/dgraph-alpha-grpc" providerName=kubernetescrd ingress=dgraph-grpc-ingress namespace=default
time="2020-02-26T10:17:14Z" level=error msg="subset not found for default/dgraph-ratel" providerName=kubernetescrd ingress=dgraph-ratel-ingress namespace=default
time="2020-02-26T10:17:14Z" level=error msg="subset not found for default/dgraph-alpha" providerName=kubernetescrd ingress=dgraph-http-ingress namespace=default
time="2020-02-26T10:17:16Z" level=error msg="subset not found for default/dgraph-ratel" ingress=dgraph-ratel-ingress namespace=default providerName=kubernetescrd
time="2020-02-26T10:17:16Z" level=error msg="subset not found for default/dgraph-alpha" providerName=kubernetescrd ingress=dgraph-http-ingress namespace=default
time="2020-02-26T10:17:16Z" level=error msg="subset not found for default/dgraph-alpha-grpc" providerName=kubernetescrd ingress=dgraph-grpc-ingress namespace=default
time="2020-02-26T10:17:18Z" level=error msg="subset not found for default/dgraph-ratel" providerName=kubernetescrd ingress=dgraph-ratel-ingress namespace=default
time="2020-02-26T10:17:18Z" level=error msg="subset not found for default/dgraph-alpha" providerName=kubernetescrd namespace=default ingress=dgraph-http-ingress
time="2020-02-26T10:17:18Z" level=error msg="subset not found for default/dgraph-alpha-grpc" namespace=default providerName=kubernetescrd ingress=dgraph-grpc-ingress
time="2020-02-26T10:17:19Z" level=error msg="subset not found for default/dgraph-alpha" providerName=kubernetescrd ingress=dgraph-http-ingress namespace=default
time="2020-02-26T10:17:19Z" level=error msg="subset not found for default/dgraph-alpha-grpc" providerName=kubernetescrd ingress=dgraph-grpc-ingress namespace=default
time="2020-02-26T10:17:20Z" level=error msg="subset not found for default/dgraph-alpha" providerName=kubernetescrd ingress=dgraph-http-ingress namespace=default
time="2020-02-26T10:17:20Z" level=error msg="subset not found for default/dgraph-alpha-grpc" providerName=kubernetescrd ingress=dgraph-grpc-ingress namespace=default

State

I have done some progress, but nothing too substantial.

I did a test by adding a nodejs pod, and executing a GRPC client to 10.42.0.12:9080 and it works. That means, inside the cluster, it works just fine. But outside, no sign of light.

outside:
node index.js
(node:9584) UnhandledPromiseRejectionWarning: Error: 1 CANCELLED: Received http2 header with status: 404

I think I got at the same point that I was before. This feels like the same issue. Traefik can't load balance all GRPC programs out there. And maybe, this would work if I expose the 9080 port (to the world outside the cluster - Update: I have tested it, don't work...). So there's no reverse proxying here for GRPCs.

I really like Traefik, but it's kind of frustrating all this. Don't have a clue of what is happening, just have assumptions and no one really cares.

Cheers.

kubectl get all -A
NAMESPACE     NAME                                          READY   STATUS      RESTARTS   AGE
kube-system   pod/metrics-server-6d684c7b5-6nt6t            1/1     Running     0          8m47s
kube-system   pod/local-path-provisioner-58fb86bdfd-qq78v   1/1     Running     0          8m47s
kube-system   pod/coredns-d798c9dd-fnm4r                    1/1     Running     0          8m47s
kube-system   pod/helm-install-traefik-tlbxn                0/1     Completed   5          8m47s
kube-system   pod/svclb-traefik-7qw54                       2/2     Running     0          5m5s
kube-system   pod/traefik-68886476c5-5ptn6                  1/1     Running     0          5m5s
default       pod/dgraph-ratel-7459974489-qtxcg             1/1     Running     0          4m19s
default       pod/dgraph-alpha-0                            1/1     Running     0          4m19s
default       pod/dgraph-zero-0                             1/1     Running     0          4m19s
default       pod/dgraph-alpha-1                            1/1     Running     0          3m44s
default       pod/dgraph-zero-1                             1/1     Running     0          3m38s
default       pod/dgraph-alpha-2                            1/1     Running     0          3m22s
default       pod/dgraph-zero-2                             1/1     Running     0          3m16s

NAMESPACE     NAME                        TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                      AGE
default       service/kubernetes          ClusterIP      10.43.0.1       <none>          443/TCP                      9m3s
kube-system   service/kube-dns            ClusterIP      10.43.0.10      <none>          53/UDP,53/TCP,9153/TCP       9m2s
kube-system   service/metrics-server      ClusterIP      10.43.10.89     <none>          443/TCP                      9m2s
kube-system   service/traefik             LoadBalancer   10.43.29.9      192.168.10.86   80:30373/TCP,443:30312/TCP   5m5s
default       service/dgraph-zero         ClusterIP      10.43.77.166    <none>          5080/TCP,6080/TCP            4m19s
default       service/dgraph-alpha        ClusterIP      10.43.102.125   <none>          7080/TCP,8080/TCP            4m19s
default       service/dgraph-alpha-grpc   ClusterIP      10.43.167.64    <none>          9080/TCP                     4m19s
default       service/dgraph-ratel        ClusterIP      10.43.172.95    <none>          8000/TCP                     4m19s

NAMESPACE     NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
kube-system   daemonset.apps/svclb-traefik   1         1         1       1            1           <none>          5m5s

NAMESPACE     NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/metrics-server           1/1     1            1           9m2s
kube-system   deployment.apps/local-path-provisioner   1/1     1            1           9m2s
kube-system   deployment.apps/coredns                  1/1     1            1           9m2s
kube-system   deployment.apps/traefik                  1/1     1            1           5m5s
default       deployment.apps/dgraph-ratel             1/1     1            1           4m19s

NAMESPACE     NAME                                                DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/metrics-server-6d684c7b5            1         1         1       8m47s
kube-system   replicaset.apps/local-path-provisioner-58fb86bdfd   1         1         1       8m47s
kube-system   replicaset.apps/coredns-d798c9dd                    1         1         1       8m47s
kube-system   replicaset.apps/traefik-68886476c5                  1         1         1       5m5s
default       replicaset.apps/dgraph-ratel-7459974489             1         1         1       4m19s

NAMESPACE   NAME                            READY   AGE
default     statefulset.apps/dgraph-alpha   3/3     4m19s
default     statefulset.apps/dgraph-zero    3/3     4m19s

NAMESPACE     NAME                             COMPLETIONS   DURATION   AGE
kube-system   job.batch/helm-install-traefik   1/1           3m43s      8m55s

New error

  • Mark bundle as not supporting multiuse
curl -v --http2 --insecure grpc.dgraph.me
*   Trying 192.168.10.86:80...
* TCP_NODELAY set
* Connected to grpc.dgraph.me (192.168.10.86) port 80 (#0)
> GET / HTTP/1.1
> Host: grpc.dgraph.me
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 500 Internal Server Error
< Date: Fri, 28 Feb 2020 01:56:56 GMT
< Content-Length: 21
< Content-Type: text/plain; charset=utf-8
< 
* Connection #0 to host grpc.dgraph.me left intact
Internal Server Error%