Issues with gRPC and multiple instances

MichelDiz · November 15, 2019, 5:26am

Hi all,
I need to solve a question. What exactly am I doing wrong?
I have this config file:

http:
  middlewares:
    grpcfix:
      stripPrefix:
        prefixes:
          - "/grpc/"

  routers:

    routerGrpc:
      rule: "Host(`www.local.dgraph.com`) && Path(`/grpc/`)" # Tested also without Path
      service: dgraph-grpc
      middlewares:
      - grpcfix

  services:

    dgraph-grpc:
      loadBalancer:
        servers:
        - url: h2c://localhost:9081
        - url: h2c://localhost:9082
        - url: h2c://localhost:9083
        - url: h2c://localhost:9084
        - url: h2c://localhost:9085
        - url: h2c://localhost:9086

My problem is:
1 - I can not connect the instance via gRPC to the path host_url/grpc/.
2 - I can only connect via host_url with the port of any instance. Which sounds like a bug to me.
3 - Even being able to access an instance via host_url + PORT yet there is no balancing.

That’s weird, it’s like Traefik was just redirecting my instance to the URL it administers. It does not balance or anything.

Does anybody have any idea?

Cheers.

MichelDiz · November 17, 2019, 11:15pm

Anybody? It would be interesting for someone to help me with this. I would recommend Traefik. However, without this gRPC thing, I can't.

PS. I was able to do it with NGINX. And it was quite easy. It should be with Traefik.

PS. I think the flag is wrong. Don't remmember if I choosed that one.

daniel.tomcej · November 21, 2019, 5:19pm

Hello @MichelDiz,

You have tagged this issue as Maesh, not Traefik, so it is in a different category.

You may want to update the tags to move this to the correct forum.

Thanks!

MichelDiz · November 21, 2019, 5:41pm

Yeah, that's odd cuz i don't even know what Maesh is. I was trying to edit this, but seems like we can edit flags after create the topic.

Btw, do you know some one who worked in the grpc support?

Cheers.

daniel.tomcej · November 21, 2019, 7:39pm

Hello @MichelDiz,

The Rule:

rule: "Host(`www.local.dgraph.com`) && Path(`/grpc/`)"

would match www.local.dgraph.com/grpc/ as an exact match only.

Also, can you provide your Traefik configuration, and an example of the gRPC error you get when connecting?

MichelDiz · November 21, 2019, 8:55pm

But it doesn't tho. Not in my env.

There's no error. Actually it works, but not as it should. As I've mentioned previously.

To reproduce it, just clone the repo I've shared.
Run sh ./dgraph.sh --multiple
Edit the dynamic_conf.yml according to what I mentioned above

Then run sh ./traefik.sh

then you need to run a Dgraph client.
This fits the context ItisTimetoReproduce/reproduce003/grpc_client at master · OpenDgraph/ItisTimetoReproduce · GitHub

You gonna need to edit the ClientStub from "localhost:9001" to www.local.dgraph.com/grpc/

So, it won't work. But, if you put www.local.dgraph.com:9081 it works. 9081 is the Dgraph's GRPC port. This means that Traefik is passing the Port directly and there is no balancing.

Cheers.

MichelDiz · December 4, 2019, 5:28pm

up+
need to know if it is possible

MichelDiz · February 26, 2020, 7:34pm

I have give another try. This time using k8s.

All HTTP works fine. Not the GRPC one.

Outside the cluster I got

curl -v  --insecure http://grpc.dgraph.me     
* Rebuilt URL to: http://grpc.dgraph.me/
*   Trying 192.168.10.86...
* TCP_NODELAY set
* Connected to grpc.dgraph.me (192.168.10.86) port 80 (#0)
> GET / HTTP/1.1
> Host: grpc.dgraph.me
> User-Agent: curl/7.54.0
> Accept: */*
> 
< HTTP/1.1 500 Internal Server Error
< Date: Wed, 26 Feb 2020 10:41:31 GMT
< Content-Length: 21
< Content-Type: text/plain; charset=utf-8
< 
* Connection #0 to host grpc.dgraph.me left intact
Internal Server Error%

And from a JS client I got

node index.js
(node:23320) UnhandledPromiseRejectionWarning: Error: 1 CANCELLED: Received http2 header with status: 404

Inside the cluster I got

curl -v  --insecure 10.42.0.21:9080
*   Trying 10.42.0.21:9080...
* TCP_NODELAY set
* Connected to 10.42.0.21 (10.42.0.21) port 9080 (#0)
> GET / HTTP/1.1
> Host: 10.42.0.21:9080
> User-Agent: curl/7.67.0
> Accept: */*
> 
* Received HTTP/0.9 when not allowed

* Closing connection 0
curl: (1) Received HTTP/0.9 when not allowed

Apparently, internally it works. But not externally.

What am I missing?

Cheers.

PS. I have used the HELM installation. Traefik 2.1.3

gist.github.com

https://gist.github.com/MichelDiz/3f00ad08fa960130b42022e835fe7460

dgraph-ha_Traefik.yaml

# This is a headless service which is necessary for discovery for a dgraph-zero StatefulSet.
# https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#creating-a-statefulset
kind: IngressRoute
apiVersion: traefik.containo.us/v1alpha1
metadata:
  name: dgraph-ratel-ingress
  namespace: default 
  annotations:
    kubernetes.io/ingress.class: "traefik"
spec:

This file has been truncated. show original

Logs

time="2020-02-26T10:15:49Z" level=info msg="Configuration loaded from flags."
time="2020-02-26T10:16:58Z" level=error msg="kubernetes service not found: default/dgraph-ratel" ingress=dgraph-ratel-ingress namespace=default providerName=kubernetescrd
time="2020-02-26T10:16:58Z" level=error msg="kubernetes service not found: default/dgraph-ratel" ingress=dgraph-ratel-ingress namespace=default providerName=kubernetescrd
time="2020-02-26T10:16:58Z" level=error msg="kubernetes service not found: default/dgraph-alpha" ingress=dgraph-http-ingress providerName=kubernetescrd namespace=default
time="2020-02-26T10:16:58Z" level=error msg="kubernetes service not found: default/dgraph-ratel" ingress=dgraph-ratel-ingress namespace=default providerName=kubernetescrd
time="2020-02-26T10:16:58Z" level=error msg="kubernetes service not found: default/dgraph-alpha" providerName=kubernetescrd ingress=dgraph-http-ingress namespace=default
time="2020-02-26T10:16:58Z" level=error msg="kubernetes service not found: default/dgraph-alpha-grpc" ingress=dgraph-grpc-ingress namespace=default providerName=kubernetescrd
time="2020-02-26T10:16:58Z" level=error msg="kubernetes service not found: default/dgraph-alpha-grpc" providerName=kubernetescrd ingress=dgraph-grpc-ingress namespace=default
time="2020-02-26T10:16:58Z" level=error msg="kubernetes service not found: default/dgraph-ratel" providerName=kubernetescrd ingress=dgraph-ratel-ingress namespace=default
time="2020-02-26T10:16:58Z" level=error msg="kubernetes service not found: default/dgraph-alpha" namespace=default providerName=kubernetescrd ingress=dgraph-http-ingress
time="2020-02-26T10:16:58Z" level=error msg="kubernetes service not found: default/dgraph-alpha" namespace=default providerName=kubernetescrd ingress=dgraph-http-ingress
time="2020-02-26T10:16:58Z" level=error msg="kubernetes service not found: default/dgraph-alpha-grpc" providerName=kubernetescrd ingress=dgraph-grpc-ingress namespace=default
time="2020-02-26T10:16:58Z" level=error msg="kubernetes service not found: default/dgraph-ratel" ingress=dgraph-ratel-ingress namespace=default providerName=kubernetescrd
time="2020-02-26T10:16:58Z" level=error msg="kubernetes service not found: default/dgraph-ratel" ingress=dgraph-ratel-ingress namespace=default providerName=kubernetescrd
time="2020-02-26T10:16:58Z" level=error msg="endpoints not found for default/dgraph-alpha" namespace=default providerName=kubernetescrd ingress=dgraph-http-ingress
time="2020-02-26T10:16:58Z" level=error msg="kubernetes service not found: default/dgraph-alpha-grpc" providerName=kubernetescrd ingress=dgraph-grpc-ingress namespace=default
time="2020-02-26T10:16:58Z" level=error msg="kubernetes service not found: default/dgraph-ratel" providerName=kubernetescrd ingress=dgraph-ratel-ingress namespace=default
time="2020-02-26T10:16:58Z" level=error msg="subset not found for default/dgraph-alpha" namespace=default providerName=kubernetescrd ingress=dgraph-http-ingress
time="2020-02-26T10:16:58Z" level=error msg="kubernetes service not found: default/dgraph-alpha-grpc" ingress=dgraph-grpc-ingress namespace=default providerName=kubernetescrd
time="2020-02-26T10:16:58Z" level=error msg="kubernetes service not found: default/dgraph-ratel" ingress=dgraph-ratel-ingress namespace=default providerName=kubernetescrd
time="2020-02-26T10:16:58Z" level=error msg="subset not found for default/dgraph-alpha" providerName=kubernetescrd ingress=dgraph-http-ingress namespace=default
time="2020-02-26T10:16:58Z" level=error msg="endpoints not found for default/dgraph-alpha-grpc" providerName=kubernetescrd ingress=dgraph-grpc-ingress namespace=default
time="2020-02-26T10:16:58Z" level=error msg="endpoints not found for default/dgraph-ratel" providerName=kubernetescrd ingress=dgraph-ratel-ingress namespace=default
time="2020-02-26T10:16:58Z" level=error msg="subset not found for default/dgraph-alpha" providerName=kubernetescrd ingress=dgraph-http-ingress namespace=default
time="2020-02-26T10:16:58Z" level=error msg="subset not found for default/dgraph-alpha-grpc" namespace=default providerName=kubernetescrd ingress=dgraph-grpc-ingress
time="2020-02-26T10:16:58Z" level=error msg="subset not found for default/dgraph-alpha-grpc" providerName=kubernetescrd ingress=dgraph-grpc-ingress namespace=default
time="2020-02-26T10:16:58Z" level=error msg="endpoints not found for default/dgraph-ratel" providerName=kubernetescrd namespace=default ingress=dgraph-ratel-ingress
time="2020-02-26T10:16:58Z" level=error msg="subset not found for default/dgraph-alpha" providerName=kubernetescrd ingress=dgraph-http-ingress namespace=default
time="2020-02-26T10:16:58Z" level=error msg="subset not found for default/dgraph-alpha" providerName=kubernetescrd ingress=dgraph-http-ingress namespace=default
time="2020-02-26T10:16:58Z" level=error msg="subset not found for default/dgraph-alpha-grpc" providerName=kubernetescrd ingress=dgraph-grpc-ingress namespace=default
time="2020-02-26T10:16:58Z" level=error msg="subset not found for default/dgraph-ratel" providerName=kubernetescrd ingress=dgraph-ratel-ingress namespace=default
time="2020-02-26T10:17:00Z" level=error msg="subset not found for default/dgraph-alpha-grpc" providerName=kubernetescrd ingress=dgraph-grpc-ingress namespace=default
time="2020-02-26T10:17:00Z" level=error msg="subset not found for default/dgraph-ratel" ingress=dgraph-ratel-ingress namespace=default providerName=kubernetescrd
time="2020-02-26T10:17:00Z" level=error msg="subset not found for default/dgraph-alpha" providerName=kubernetescrd ingress=dgraph-http-ingress namespace=default
time="2020-02-26T10:17:02Z" level=error msg="subset not found for default/dgraph-alpha-grpc" providerName=kubernetescrd ingress=dgraph-grpc-ingress namespace=default
time="2020-02-26T10:17:02Z" level=error msg="subset not found for default/dgraph-ratel" providerName=kubernetescrd ingress=dgraph-ratel-ingress namespace=default
time="2020-02-26T10:17:02Z" level=error msg="subset not found for default/dgraph-alpha" providerName=kubernetescrd ingress=dgraph-http-ingress namespace=default
time="2020-02-26T10:17:04Z" level=error msg="subset not found for default/dgraph-ratel" providerName=kubernetescrd ingress=dgraph-ratel-ingress namespace=default
time="2020-02-26T10:17:04Z" level=error msg="subset not found for default/dgraph-alpha" providerName=kubernetescrd ingress=dgraph-http-ingress namespace=default
time="2020-02-26T10:17:04Z" level=error msg="subset not found for default/dgraph-alpha-grpc" namespace=default providerName=kubernetescrd ingress=dgraph-grpc-ingress
time="2020-02-26T10:17:06Z" level=error msg="subset not found for default/dgraph-ratel" providerName=kubernetescrd ingress=dgraph-ratel-ingress namespace=default
time="2020-02-26T10:17:06Z" level=error msg="subset not found for default/dgraph-alpha" ingress=dgraph-http-ingress namespace=default providerName=kubernetescrd
time="2020-02-26T10:17:06Z" level=error msg="subset not found for default/dgraph-alpha-grpc" ingress=dgraph-grpc-ingress providerName=kubernetescrd namespace=default
time="2020-02-26T10:17:08Z" level=error msg="subset not found for default/dgraph-alpha-grpc" namespace=default providerName=kubernetescrd ingress=dgraph-grpc-ingress
time="2020-02-26T10:17:08Z" level=error msg="subset not found for default/dgraph-ratel" providerName=kubernetescrd ingress=dgraph-ratel-ingress namespace=default
time="2020-02-26T10:17:08Z" level=error msg="subset not found for default/dgraph-alpha" ingress=dgraph-http-ingress namespace=default providerName=kubernetescrd
time="2020-02-26T10:17:10Z" level=error msg="subset not found for default/dgraph-ratel" ingress=dgraph-ratel-ingress providerName=kubernetescrd namespace=default
time="2020-02-26T10:17:10Z" level=error msg="subset not found for default/dgraph-alpha" ingress=dgraph-http-ingress namespace=default providerName=kubernetescrd
time="2020-02-26T10:17:10Z" level=error msg="subset not found for default/dgraph-alpha-grpc" providerName=kubernetescrd ingress=dgraph-grpc-ingress namespace=default
time="2020-02-26T10:17:12Z" level=error msg="subset not found for default/dgraph-ratel" providerName=kubernetescrd ingress=dgraph-ratel-ingress namespace=default
time="2020-02-26T10:17:12Z" level=error msg="subset not found for default/dgraph-alpha" providerName=kubernetescrd namespace=default ingress=dgraph-http-ingress
time="2020-02-26T10:17:12Z" level=error msg="subset not found for default/dgraph-alpha-grpc" providerName=kubernetescrd ingress=dgraph-grpc-ingress namespace=default
time="2020-02-26T10:17:14Z" level=error msg="subset not found for default/dgraph-alpha-grpc" providerName=kubernetescrd ingress=dgraph-grpc-ingress namespace=default
time="2020-02-26T10:17:14Z" level=error msg="subset not found for default/dgraph-ratel" providerName=kubernetescrd ingress=dgraph-ratel-ingress namespace=default
time="2020-02-26T10:17:14Z" level=error msg="subset not found for default/dgraph-alpha" providerName=kubernetescrd ingress=dgraph-http-ingress namespace=default
time="2020-02-26T10:17:16Z" level=error msg="subset not found for default/dgraph-ratel" ingress=dgraph-ratel-ingress namespace=default providerName=kubernetescrd
time="2020-02-26T10:17:16Z" level=error msg="subset not found for default/dgraph-alpha" providerName=kubernetescrd ingress=dgraph-http-ingress namespace=default
time="2020-02-26T10:17:16Z" level=error msg="subset not found for default/dgraph-alpha-grpc" providerName=kubernetescrd ingress=dgraph-grpc-ingress namespace=default
time="2020-02-26T10:17:18Z" level=error msg="subset not found for default/dgraph-ratel" providerName=kubernetescrd ingress=dgraph-ratel-ingress namespace=default
time="2020-02-26T10:17:18Z" level=error msg="subset not found for default/dgraph-alpha" providerName=kubernetescrd namespace=default ingress=dgraph-http-ingress
time="2020-02-26T10:17:18Z" level=error msg="subset not found for default/dgraph-alpha-grpc" namespace=default providerName=kubernetescrd ingress=dgraph-grpc-ingress
time="2020-02-26T10:17:19Z" level=error msg="subset not found for default/dgraph-alpha" providerName=kubernetescrd ingress=dgraph-http-ingress namespace=default
time="2020-02-26T10:17:19Z" level=error msg="subset not found for default/dgraph-alpha-grpc" providerName=kubernetescrd ingress=dgraph-grpc-ingress namespace=default
time="2020-02-26T10:17:20Z" level=error msg="subset not found for default/dgraph-alpha" providerName=kubernetescrd ingress=dgraph-http-ingress namespace=default
time="2020-02-26T10:17:20Z" level=error msg="subset not found for default/dgraph-alpha-grpc" providerName=kubernetescrd ingress=dgraph-grpc-ingress namespace=default

State

MichelDiz · February 27, 2020, 11:58pm

I have done some progress, but nothing too substantial.

I did a test by adding a nodejs pod, and executing a GRPC client to 10.42.0.12:9080 and it works. That means, inside the cluster, it works just fine. But outside, no sign of light.

outside:
node index.js
(node:9584) UnhandledPromiseRejectionWarning: Error: 1 CANCELLED: Received http2 header with status: 404

I think I got at the same point that I was before. This feels like the same issue. Traefik can't load balance all GRPC programs out there. And maybe, this would work if I expose the 9080 port (to the world outside the cluster - Update: I have tested it, don't work...). So there's no reverse proxying here for GRPCs.

I really like Traefik, but it's kind of frustrating all this. Don't have a clue of what is happening, just have assumptions and no one really cares.

Cheers.

kubectl get all -A
NAMESPACE     NAME                                          READY   STATUS      RESTARTS   AGE
kube-system   pod/metrics-server-6d684c7b5-6nt6t            1/1     Running     0          8m47s
kube-system   pod/local-path-provisioner-58fb86bdfd-qq78v   1/1     Running     0          8m47s
kube-system   pod/coredns-d798c9dd-fnm4r                    1/1     Running     0          8m47s
kube-system   pod/helm-install-traefik-tlbxn                0/1     Completed   5          8m47s
kube-system   pod/svclb-traefik-7qw54                       2/2     Running     0          5m5s
kube-system   pod/traefik-68886476c5-5ptn6                  1/1     Running     0          5m5s
default       pod/dgraph-ratel-7459974489-qtxcg             1/1     Running     0          4m19s
default       pod/dgraph-alpha-0                            1/1     Running     0          4m19s
default       pod/dgraph-zero-0                             1/1     Running     0          4m19s
default       pod/dgraph-alpha-1                            1/1     Running     0          3m44s
default       pod/dgraph-zero-1                             1/1     Running     0          3m38s
default       pod/dgraph-alpha-2                            1/1     Running     0          3m22s
default       pod/dgraph-zero-2                             1/1     Running     0          3m16s

NAMESPACE     NAME                        TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                      AGE
default       service/kubernetes          ClusterIP      10.43.0.1       <none>          443/TCP                      9m3s
kube-system   service/kube-dns            ClusterIP      10.43.0.10      <none>          53/UDP,53/TCP,9153/TCP       9m2s
kube-system   service/metrics-server      ClusterIP      10.43.10.89     <none>          443/TCP                      9m2s
kube-system   service/traefik             LoadBalancer   10.43.29.9      192.168.10.86   80:30373/TCP,443:30312/TCP   5m5s
default       service/dgraph-zero         ClusterIP      10.43.77.166    <none>          5080/TCP,6080/TCP            4m19s
default       service/dgraph-alpha        ClusterIP      10.43.102.125   <none>          7080/TCP,8080/TCP            4m19s
default       service/dgraph-alpha-grpc   ClusterIP      10.43.167.64    <none>          9080/TCP                     4m19s
default       service/dgraph-ratel        ClusterIP      10.43.172.95    <none>          8000/TCP                     4m19s

NAMESPACE     NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
kube-system   daemonset.apps/svclb-traefik   1         1         1       1            1           <none>          5m5s

NAMESPACE     NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/metrics-server           1/1     1            1           9m2s
kube-system   deployment.apps/local-path-provisioner   1/1     1            1           9m2s
kube-system   deployment.apps/coredns                  1/1     1            1           9m2s
kube-system   deployment.apps/traefik                  1/1     1            1           5m5s
default       deployment.apps/dgraph-ratel             1/1     1            1           4m19s

NAMESPACE     NAME                                                DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/metrics-server-6d684c7b5            1         1         1       8m47s
kube-system   replicaset.apps/local-path-provisioner-58fb86bdfd   1         1         1       8m47s
kube-system   replicaset.apps/coredns-d798c9dd                    1         1         1       8m47s
kube-system   replicaset.apps/traefik-68886476c5                  1         1         1       5m5s
default       replicaset.apps/dgraph-ratel-7459974489             1         1         1       4m19s

NAMESPACE   NAME                            READY   AGE
default     statefulset.apps/dgraph-alpha   3/3     4m19s
default     statefulset.apps/dgraph-zero    3/3     4m19s

NAMESPACE     NAME                             COMPLETIONS   DURATION   AGE
kube-system   job.batch/helm-install-traefik   1/1           3m43s      8m55s

New error

Mark bundle as not supporting multiuse

curl -v --http2 --insecure grpc.dgraph.me
*   Trying 192.168.10.86:80...
* TCP_NODELAY set
* Connected to grpc.dgraph.me (192.168.10.86) port 80 (#0)
> GET / HTTP/1.1
> Host: grpc.dgraph.me
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 500 Internal Server Error
< Date: Fri, 28 Feb 2020 01:56:56 GMT
< Content-Length: 21
< Content-Type: text/plain; charset=utf-8
< 
* Connection #0 to host grpc.dgraph.me left intact
Internal Server Error%

Topic		Replies	Views
gRPC service not accessible from Traefik Traefik v2 docker	7	1105	November 10, 2022
gRPC proxying issues only when there are multiple service instances Traefik v2 docker , consul-catalog	0	396	February 20, 2023
Access gRPC server through Traefikee Traefik v2 file , tcp	2	1329	February 2, 2023
Help needed with multiple traefik instances and routing to services on different servers Traefik v2 docker , tcp	4	938	March 23, 2024
Traefik v2 with gRPC (h2c) using Consul catalog Traefik v2 consul-catalog	2	2377	August 11, 2020

Issues with gRPC and multiple instances

Logs

State

New error

Related topics