I have been searching around for a few days now, but not come to a definitive answer. I am looking to see if it is possible to select between several gRPC services on the backend. I have successfully done it with NGINX, where you can put a path in the request, and route on that. I have yet to make that work in Traefik. The idea, obviously, is to route requests to service1 to the endpoint of service1 etc.
This is how I configured the IngressRoute:
kind: IngressRoute
metadata:
...
annotations:
traefik.ingress.kubernetes.io/router.middlewares: "traefik-v2-autodetect@kubernetescrd"
traefik.ingress.kubernetes.io/service.serversscheme: h2c
spec:
entryPoints:
- websecure
routes:
- kind: Rule
match: Host(`{{ .Values.environment.namespaceName }}.mydomain.com`) && Path(`/service1`)
services:
- name: whoami-grpc-svc
namespace: {{ .Values.environment.namespaceName }}
port: {{ .Values.whoAmI.grpcInternalPort }}
scheme: h2c
passHostHeader: true
tls:
secretName: {{ .Values.environment.namespaceName }}.mydomain-tls
I am sure I can't be the first one in the world to want to expose multiple gRPC services to the world via Traefik, so I am guessing there has got to be a solution to this. Any ideas?
The Traefik "gRPC examples" (link) show that it’s a regular router with rule, so you should be able to use Host() && PathPrefix() for different routers and services.
Hi,
Thanks for the reply. I have tried that - before posting the question. My IngressRoute looks like this:
spec:
entryPoints:
- websecure
routes:
- kind: Rule
match: Host(`{{ .Values.environment.namespaceName }}.mydomain.com`) && PathPrefix(`/service1`)
services:
- name: whoami-grpc-svc
namespace: {{ .Values.environment.namespaceName }}
port: {{ .Values.whoAmI.grpcInternalPort }}
scheme: h2c
passHostHeader: true
tls:
secretName: {{ .Values.environment.namespaceName }}.mydomain-tls
When invoked with grpcurl, like below, I get an error:
$ grpcurl -insecure -vv -d '{"name": "Henry"}' baz.mydomain.com:443/service1 Greeter/SayHello
Failed to dial target host "baz.mydomain.com:443/service1": dial tcp: lookup tcp/443/service1: nodename nor servname provided, or not known
I am wondering if the missing part is a rewrite of the path?
Your tool expects domain:port, so you can’t add a path there. I don’t know gRPC, maybe the path is reserved for method name. You would need to look into the specs, but that’s rather not Traefik related.
Thanks for the reply again.
I hear you on the tooling, so I wrote a client in .net as I have been using with an nginx ingress controller. I think it is very much a Traefik issue: Running the client targeting https://baz.mydomain.com/service1 and https://baz.mydomain.com/ when there is no PathPrefix on the match, things work. When I put the PathPrefix in place, however, none of them work yielding a 404.
192.168.65.3 - - [30/Oct/2023:20:13:40 +0000] "POST /Greeter/SayHello HTTP/2.0" 404 19 "-" "-" 4177 "-" "-" 0ms
192.168.65.3 - - [30/Oct/2023:20:14:23 +0000] "POST /Greeter/SayHello HTTP/2.0" 404 19 "-" "-" 4187 "-" "-" 0ms
I even tried a strip prefix middleware, but to no avail.
This is not a Traefik problem:
$ grpcurl -insecure -vv -d '{"name": "Henry"}' baz.mydomain.com:443/service1 Greeter/SayHello
Failed to dial target host "baz.mydomain.com:443/service1": dial tcp: lookup tcp/443/service1: nodename nor servname provided, or not known
What do you want to achieve? gRPC uses path in the way /service/method.
Thanks for the reply again 
I would like to route to multiple gRPC services on the backend based on the path. So, I could have baz.mydomain.com:443/service1 routing to Service1, baz.mydomain.com:443/service2 routing to Service2. It is possible in nginx - which can do this, but has other shortcomings I need working.
I guess I could make it work by doing service1.baz.mydomain.com and service2.baz.mydomain.com, but that would require extra work doing the DNS entries.
If you want /greeter1/sayHello and /greeter2/sayHello on different services, then just use a router per service:
Host(`example.com`) && PathPrefix(`/greeter1/`)
That is not really what I want. What I want is the client to call baz.mydomain.com/service1 and end up with service 1, and baz.mydomain.com/service2 ending up with service 2. Service 1 and 2 can potentially be different services.
I have tried your example from above, but it yields a 404. I guess gRPC forms the internal path ("/Greeter/SayHello") as part of the request based on the service name (implementation) and method on the service. I am wondering how nginx manages to do the path rewrite and dispatching.
192.168.65.3 - - [04/Nov/2023:11:51:41 +0000] "POST /Greeter/SayHello HTTP/2.0" 404 19 "-" "-" 911 "-" "-" 0ms
gRPC expects that the paths match the API methods.
You could deploy a stripPrefixRegex middleware and adds it to your two routes.
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
name: strip-prefix-regex
spec:
stripPrefixRegex:
regex:
- "^/[^/]+"
Hi! Thanks for the reply. For some reason it still returns a 404 even with the middleware in place.
192.168.65.3 - - [08/Nov/2023:19:23:36 +0000] "POST /Greeter/SayHello HTTP/2.0" 404 19 "-" "-" 509 "-" "-" 0ms
I can't really figure out if this means that the backend is not found, or if the method on the backend is not found.
Change access logging to JSON, AFAIK then you can see the Traefik status code and a target service status code (if Traefik got a response).
Was a good idea, but it yielded nothing. Same error message.
I was comparing the log entries from a successful, regular http call to the gRPC. It seems something is amiss in the configuration of the router. In the normal http call, the name of the router is given in the logs. In the case of the gRPC, no router is named, though one exists.
It would be nice to have more diagnostics options.
For diagnostics you can enable and check Traefik debug log and Traefik dashboard.
All seems fine in this one (note, I know the path is "/update", rather than "/service1".)
Yet, all see in the logs are this:
192.168.65.3 - - [09/Nov/2023:15:14:49 +0000] "POST /Greeter/SayHello HTTP/2.0" 404 19 "-" "-" 328 "-" "-" 0ms
Well, we can only find our if the 404 is from Traefik or just passed back from the target service when you enable and check access log in JSON or debug log.
I am missing something here. I think I have all of the settings configured. My k8s is configured like this:
additionalArguments:
- "--entrypoints.proxy.address=:6809/tcp"
- "--log.level=DEBUG"
- "--log.format=json"
- "--providers.kubernetescrd"
- "--providers.kubernetesingress=true"
- "--serversTransport.insecureSkipVerify=true"
- "--api=true"
- "--api.insecure=true"
- "--api.dashboard=true"
- "--accesslog"
Yet, all I get when I invoke the service is this:
192.168.65.3 - - [09/Nov/2023:17:48:54 +0000] "POST /Greeter/SayHello HTTP/2.0" 404 19 "-" "-" 111 "-" "-" 0ms
Are you referring to the json emitted as the service is created?
Try adding
- "--accesslog.format=json"
Did that. Got this:
{"ClientAddr":"192.168.65.3:55064","ClientHost":"192.168.65.3","ClientPort":"55064","ClientUsername":"-","DownstreamContentSize":19,"DownstreamStatus":404,"Duration":108537,"GzipRatio":0,"OriginContentSize":0,"OriginDuration":0,"OriginStatus":0,"Overhead":108537,"RequestAddr":"baz.mydomain.com","RequestContentSize":0,"RequestCount":12,"RequestHost":"baz.mydomain.com","RequestMethod":"POST","RequestPath":"/Greeter/SayHello","RequestPort":"-","RequestProtocol":"HTTP/2.0","RequestScheme":"https","RetryAttempts":0,"StartLocal":"2023-11-09T20:36:58.788020511Z","StartUTC":"2023-11-09T20:36:58.788020511Z","TLSCipher":"TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLSVersion":"1.2","entryPointName":"websecure","level":"info","msg":"","time":"2023-11-09T20:36:58Z"}
I am not sure what I should be looking for here... Did notice some entries in the logs due to a failure in configuring TLS ("Error configuring TLS: secret baz/baz.mydomain-tls does not exist"), though they do exist.
No upstream, so error is from Traefik. I see websecure in there, which I did not expect.
Share your full Traefik static and dynamic config.
