gRPC service not accessible from Traefik

tavsec · November 9, 2022, 8:11am

I'm trying to route traffic for my gRPC service with Traefik. My Traefik configuration:

accessLog: 
  filePath: "/var/logs/access.log"

api:
  dashboard: true
  insecure: true

metrics:
  prometheus: {}

providers:
  nomad: 
    endpoint:
      address: http://host.docker.internal:4646

entryPoints:
  web:
    http2: 
      maxConcurrentStreams: 250
    address: ":80"
  grpc:
    http2: 
      maxConcurrentStreams: 250
    address: ":81"

When I spin up the container of my gRPC service, it is registered to Traefik, with h2c protocol:

I'm trying to route traffic for my gRPC service with Traefik. My Traefik configuration:

accessLog: 
  filePath: "/var/logs/access.log"

api:
  dashboard: true
  insecure: true

metrics:
  prometheus: {}

providers:
  nomad: 
    endpoint:
      address: http://host.docker.internal:4646

entryPoints:
  web:
    http2: 
      maxConcurrentStreams: 250
    address: ":80"
  grpc:
    http2: 
      maxConcurrentStreams: 250
    address: ":81"

When I spin up the container of my gRPC service, it is registered to Traefik, with h2c protocol:

When I send the gRPC request from Postman directly to the allocated node (127.0.0.1:29165), the request works:

But if I want to send the request to the host, specified in rules (grpc.localhost), I get an error:

I tried changing entryPoint ports, and also disabling host header forwarding, but the problem persists.

Any help would be appreciated! Thank you.

kevinpollet · November 10, 2022, 8:34am

Hello @tavsec and thanks for your interest in Traefik,

After looking at your static configuration I see nothing wrong.

My first guess would be that grpc.localhost hostname is not resolvable from your host. What is the output of the following command ping grpc.localhost?

Please, could you also provide your dynamic configuration, and the debug logs when you are making a request through Traefik?

tavsec · November 10, 2022, 10:06am

Hey @kevinpollet, thanks for answering!
grpc.localhost is pointing to the 127.0.0.1 in my /etc/hosts file, so when I ping it I got the following response:

 >> ping grpc.localhost
PING grpc.localhost (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.043 ms
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.108 ms
64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.078 ms
64 bytes from 127.0.0.1: icmp_seq=3 ttl=64 time=0.108 ms
64 bytes from 127.0.0.1: icmp_seq=4 ttl=64 time=0.125 ms
64 bytes from 127.0.0.1: icmp_seq=5 ttl=64 time=0.086 ms
64 bytes from 127.0.0.1: icmp_seq=6 ttl=64 time=0.083 ms
^C
--- grpc.localhost ping statistics ---
7 packets transmitted, 7 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.043/0.090/0.125/0.025 ms

The dynamic config/labels for my gRPC containers are:

"traefik.http.routers.grpc.rule": "Host(`grpc.localhost`)",
"traefik.http.routers.grpc.service": "grpc",
"traefik.enable": "true",
"traefik.http.services.grpc.loadbalancer.server.scheme": "h2c"

I am not exactly sure whay you mean by debug logs? If you mean stdout of Traefik container, this is the only line that gets printed:

time="2022-11-10T09:59:48Z" level=info msg="Configuration loaded from file: /etc/traefik/traefik.yml"

One thing that I found though, is that if I look at the access.log, there is no entry about gRPC service, only the Prometheus ones:

192.168.112.2 - - [10/Nov/2022:10:03:00 +0000] "GET /metrics HTTP/1.1" 200 1734 "-" "-" 58 "prometheus@internal" "-" 3ms

192.168.112.2 - - [10/Nov/2022:10:03:10 +0000] "GET /metrics HTTP/1.1" 200 1733 "-" "-" 59 "prometheus@internal" "-" 1ms

192.168.112.2 - - [10/Nov/2022:10:03:20 +0000] "GET /metrics HTTP/1.1" 200 1733 "-" "-" 60 "prometheus@internal" "-" 4ms

192.168.112.2 - - [10/Nov/2022:10:03:30 +0000] "GET /metrics HTTP/1.1" 200 1735 "-" "-" 61 "prometheus@internal" "-" 2ms

192.168.112.2 - - [10/Nov/2022:10:03:40 +0000] "GET /metrics HTTP/1.1" 200 1727 "-" "-" 62 "prometheus@internal" "-" 4ms

192.168.112.1 - - [10/Nov/2022:10:03:42 +0000] "GET /api/overview HTTP/1.1" 200 506 "-" "-" 63 "api@internal" "-" 0ms

192.168.112.2 - - [10/Nov/2022:10:03:50 +0000] "GET /metrics HTTP/1.1" 200 1729 "-" "-" 64 "prometheus@internal" "-" 1ms

192.168.112.2 - - [10/Nov/2022:10:04:00 +0000] "GET /metrics HTTP/1.1" 200 1735 "-" "-" 65 "prometheus@internal" "-" 3ms

192.168.112.2 - - [10/Nov/2022:10:04:10 +0000] "GET /metrics HTTP/1.1" 200 1734 "-" "-" 66 "prometheus@internal" "-" 1ms

192.168.112.2 - - [10/Nov/2022:10:04:20 +0000] "GET /metrics HTTP/1.1" 200 1734 "-" "-" 67 "prometheus@internal" "-" 4ms

kevinpollet · November 10, 2022, 10:20am

I am not exactly sure whay you mean by debug logs?

I mean the debug log level as explained in Traefik Logs Documentation - Traefik

tavsec · November 10, 2022, 10:35am

Ah okay, there you go:

time="2022-11-10T10:34:15Z" level=info msg="Configuration loaded from file: /etc/traefik/traefik.yml"
time="2022-11-10T10:34:15Z" level=info msg="Traefik version 2.9.4 built on 2022-10-27T18:44:34Z"
time="2022-11-10T10:34:15Z" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"grpc\":{\"address\":\":81\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}},\"traefik\":{\"address\":\":8080\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}},\"web\":{\"address\":\":80\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}}},\"providers\":{\"providersThrottleDuration\":\"2s\",\"nomad\":{\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"endpoint\":{\"address\":\"http://host.docker.internal:4646\"},\"prefix\":\"traefik\",\"exposedByDefault\":true,\"refreshInterval\":\"15s\"}},\"api\":{\"insecure\":true,\"dashboard\":true},\"metrics\":{\"prometheus\":{\"buckets\":[0.1,0.3,1.2,5],\"addEntryPointsLabels\":true,\"addServicesLabels\":true,\"entryPoint\":\"traefik\"}},\"log\":{\"level\":\"DEBUG\",\"format\":\"common\"},\"accessLog\":{\"filePath\":\"/var/logs/access.log\",\"format\":\"common\",\"filters\":{},\"fields\":{\"defaultMode\":\"keep\",\"headers\":{\"defaultMode\":\"drop\"}}}}"
time="2022-11-10T10:34:15Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\n"
time="2022-11-10T10:34:15Z" level=debug msg="Configured Prometheus metrics" metricsProviderName=prometheus
time="2022-11-10T10:34:15Z" level=info msg="Starting provider aggregator aggregator.ProviderAggregator"
time="2022-11-10T10:34:15Z" level=debug msg="Starting TCP Server" entryPointName=web
time="2022-11-10T10:34:15Z" level=debug msg="Starting TCP Server" entryPointName=grpc
time="2022-11-10T10:34:15Z" level=info msg="Starting provider *traefik.Provider"
time="2022-11-10T10:34:15Z" level=debug msg="*traefik.Provider provider configuration: {}"
time="2022-11-10T10:34:15Z" level=debug msg="Starting TCP Server" entryPointName=traefik
time="2022-11-10T10:34:15Z" level=info msg="Starting provider *acme.ChallengeTLSALPN"
time="2022-11-10T10:34:15Z" level=debug msg="*acme.ChallengeTLSALPN provider configuration: {}"
time="2022-11-10T10:34:15Z" level=info msg="Starting provider *nomad.Provider"
time="2022-11-10T10:34:15Z" level=debug msg="*nomad.Provider provider configuration: {\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"endpoint\":{\"address\":\"http://host.docker.internal:4646\"},\"prefix\":\"traefik\",\"exposedByDefault\":true,\"refreshInterval\":\"15s\"}"
time="2022-11-10T10:34:15Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"api\":{\"entryPoints\":[\"traefik\"],\"service\":\"api@internal\",\"rule\":\"PathPrefix(`/api`)\",\"priority\":2147483646},\"dashboard\":{\"entryPoints\":[\"traefik\"],\"middlewares\":[\"dashboard_redirect@internal\",\"dashboard_stripprefix@internal\"],\"service\":\"dashboard@internal\",\"rule\":\"PathPrefix(`/`)\",\"priority\":2147483645},\"prometheus\":{\"entryPoints\":[\"traefik\"],\"service\":\"prometheus@internal\",\"rule\":\"PathPrefix(`/metrics`)\",\"priority\":2147483647}},\"services\":{\"api\":{},\"dashboard\":{},\"noop\":{},\"prometheus\":{}},\"middlewares\":{\"dashboard_redirect\":{\"redirectRegex\":{\"regex\":\"^(http:\\\\/\\\\/(\\\\[[\\\\w:.]+\\\\]|[\\\\w\\\\._-]+)(:\\\\d+)?)\\\\/$\",\"replacement\":\"${1}/dashboard/\",\"permanent\":true}},\"dashboard_stripprefix\":{\"stripPrefix\":{\"prefixes\":[\"/dashboard/\",\"/dashboard\"]}}},\"serversTransports\":{\"default\":{\"maxIdleConnsPerHost\":200}}},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=internal
time="2022-11-10T10:34:15Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"grpc\":{\"service\":\"grpc\",\"rule\":\"Host(`grpc.localhost`)\"}},\"services\":{\"grpc\":{\"loadBalancer\":{\"servers\":[{\"url\":\"h2c://127.0.0.1:20480\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"udp\":{}}" providerName=nomad
time="2022-11-10T10:34:15Z" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default
time="2022-11-10T10:34:15Z" level=debug msg="Added outgoing tracing middleware prometheus@internal" routerName=prometheus@internal middlewareName=tracing middlewareType=TracingForwarder entryPointName=traefik
time="2022-11-10T10:34:15Z" level=debug msg="Added outgoing tracing middleware api@internal" middlewareName=tracing middlewareType=TracingForwarder entryPointName=traefik routerName=api@internal
time="2022-11-10T10:34:15Z" level=debug msg="Added outgoing tracing middleware dashboard@internal" entryPointName=traefik routerName=dashboard@internal middlewareName=tracing middlewareType=TracingForwarder
time="2022-11-10T10:34:15Z" level=debug msg="Creating middleware" routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal middlewareType=StripPrefix entryPointName=traefik
time="2022-11-10T10:34:15Z" level=debug msg="Adding tracing to middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal
time="2022-11-10T10:34:15Z" level=debug msg="Creating middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex
time="2022-11-10T10:34:15Z" level=debug msg="Setting up redirection from ^(http:\\/\\/(\\[[\\w:.]+\\]|[\\w\\._-]+)(:\\d+)?)\\/$ to ${1}/dashboard/" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex
time="2022-11-10T10:34:15Z" level=debug msg="Adding tracing to middleware" middlewareName=dashboard_redirect@internal entryPointName=traefik routerName=dashboard@internal
time="2022-11-10T10:34:15Z" level=debug msg="Creating middleware" entryPointName=traefik middlewareType=Recovery middlewareName=traefik-internal-recovery
time="2022-11-10T10:34:15Z" level=debug msg="Creating middleware" middlewareType=Metrics entryPointName=grpc middlewareName=metrics-entrypoint
time="2022-11-10T10:34:15Z" level=debug msg="Creating middleware" middlewareType=Metrics entryPointName=web middlewareName=metrics-entrypoint
time="2022-11-10T10:34:15Z" level=debug msg="Creating middleware" entryPointName=traefik middlewareName=metrics-entrypoint middlewareType=Metrics
time="2022-11-10T10:34:15Z" level=debug msg="Creating middleware" entryPointName=grpc middlewareName=metrics-entrypoint middlewareType=Metrics
time="2022-11-10T10:34:15Z" level=debug msg="Creating middleware" entryPointName=web middlewareName=metrics-entrypoint middlewareType=Metrics
time="2022-11-10T10:34:15Z" level=debug msg="Creating middleware" middlewareName=metrics-entrypoint middlewareType=Metrics entryPointName=traefik
time="2022-11-10T10:34:15Z" level=debug msg="No entryPoint defined for this router, using the default one(s) instead: [grpc web]" routerName=grpc
time="2022-11-10T10:34:16Z" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default
time="2022-11-10T10:34:16Z" level=debug msg="Added outgoing tracing middleware dashboard@internal" middlewareType=TracingForwarder entryPointName=traefik routerName=dashboard@internal middlewareName=tracing
time="2022-11-10T10:34:16Z" level=debug msg="Creating middleware" middlewareName=dashboard_stripprefix@internal middlewareType=StripPrefix routerName=dashboard@internal entryPointName=traefik
time="2022-11-10T10:34:16Z" level=debug msg="Adding tracing to middleware" routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal entryPointName=traefik
time="2022-11-10T10:34:16Z" level=debug msg="Creating middleware" middlewareType=RedirectRegex entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal
time="2022-11-10T10:34:16Z" level=debug msg="Setting up redirection from ^(http:\\/\\/(\\[[\\w:.]+\\]|[\\w\\._-]+)(:\\d+)?)\\/$ to ${1}/dashboard/" routerName=dashboard@internal middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex entryPointName=traefik
time="2022-11-10T10:34:16Z" level=debug msg="Adding tracing to middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal
time="2022-11-10T10:34:16Z" level=debug msg="Added outgoing tracing middleware prometheus@internal" routerName=prometheus@internal entryPointName=traefik middlewareName=tracing middlewareType=TracingForwarder
time="2022-11-10T10:34:16Z" level=debug msg="Added outgoing tracing middleware api@internal" routerName=api@internal middlewareName=tracing middlewareType=TracingForwarder entryPointName=traefik
time="2022-11-10T10:34:16Z" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=traefik
time="2022-11-10T10:34:16Z" level=debug msg="Creating middleware" middlewareType=Pipelining entryPointName=grpc routerName=grpc@nomad serviceName=grpc middlewareName=pipelining
time="2022-11-10T10:34:16Z" level=debug msg="Creating middleware" middlewareName=metrics-service middlewareType=Metrics entryPointName=grpc routerName=grpc@nomad serviceName=grpc
time="2022-11-10T10:34:16Z" level=debug msg="Creating load-balancer" routerName=grpc@nomad serviceName=grpc entryPointName=grpc
time="2022-11-10T10:34:16Z" level=debug msg="Creating server 0 h2c://127.0.0.1:20480" serviceName=grpc serverName=0 entryPointName=grpc routerName=grpc@nomad
time="2022-11-10T10:34:16Z" level=debug msg="child h2c://127.0.0.1:20480 now UP"
time="2022-11-10T10:34:16Z" level=debug msg="Propagating new UP status"
time="2022-11-10T10:34:16Z" level=debug msg="Added outgoing tracing middleware grpc" entryPointName=grpc routerName=grpc@nomad middlewareName=tracing middlewareType=TracingForwarder
time="2022-11-10T10:34:16Z" level=debug msg="Creating middleware" entryPointName=grpc middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2022-11-10T10:34:16Z" level=debug msg="Creating middleware" entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2022-11-10T10:34:16Z" level=debug msg="Creating middleware" entryPointName=grpc middlewareName=metrics-entrypoint middlewareType=Metrics
time="2022-11-10T10:34:16Z" level=debug msg="Creating middleware" entryPointName=web middlewareType=Metrics middlewareName=metrics-entrypoint
time="2022-11-10T10:34:16Z" level=debug msg="Creating middleware" middlewareType=Metrics entryPointName=traefik middlewareName=metrics-entrypoint
time="2022-11-10T10:34:16Z" level=debug msg="Creating middleware" middlewareName=metrics-entrypoint middlewareType=Metrics entryPointName=grpc
time="2022-11-10T10:34:16Z" level=debug msg="Creating middleware" entryPointName=web middlewareType=Metrics middlewareName=metrics-entrypoint
time="2022-11-10T10:34:16Z" level=debug msg="Creating middleware" entryPointName=traefik middlewareName=metrics-entrypoint middlewareType=Metrics
time="2022-11-10T10:34:30Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"grpc\":{\"service\":\"grpc\",\"rule\":\"Host(`grpc.localhost`)\"}},\"services\":{\"grpc\":{\"loadBalancer\":{\"servers\":[{\"url\":\"h2c://127.0.0.1:20480\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"udp\":{}}" providerName=nomad
time="2022-11-10T10:34:30Z" level=debug msg="Skipping unchanged configuration." providerName=nomad

kevinpollet · November 10, 2022, 12:55pm

One thing that I found though, is that if I look at the access.log, there is no entry about gRPC service, only the Prometheus ones:

Are you sure that Postman is able to resolve the grpc.localhost hostname?

tavsec · November 10, 2022, 1:54pm

Hey, I tried sending gRPC request to port :80 (grpc.localhost:80), I get the following line in access log:

192.168.160.1 - - [10/Nov/2022:13:52:19 +0000] "POST /helloworld.Helloworld/SayHello HTTP/2.0" 502 11 "-" "-" 222 "grpc@nomad" "h2c://127.0.0.1:20480" 8ms

but I get different error:

I don't know why Traefik is routing through port 80, when I set the entrypoint to gRPC (port 81).

kevinpollet · November 10, 2022, 4:26pm

I don't know why Traefik is routing through port 80, when I set the entrypoint to gRPC (port 81).

That is because the router is attached to all entry points (grpc and web as shown in the UI). Thus, the grpc service is reachable from the web entrypoint. To attach to only to the grpc entrypoint you can use the entrypoints label (traefik.http.routers.grpc.entrypoints=grpc) as described in Traefik Nomad Service Discovery Routing - Traefik

Topic		Replies	Views
DNS resolution failed while trying to access GRPC container via BloomRPC Traefik v2 docker , docker-swarm	0	1521	April 5, 2021
How to set up Traefik v2 with gRPC and docker Traefik v2 docker	8	4438	April 6, 2021
Access gRPC server through Traefikee Traefik v2 file , tcp	2	1329	February 2, 2023
Set up Traefik V2 with grpc Traefik v2	3	1630	June 30, 2019
How to make gRPC request through Traefik from external network (outside docker container)? Traefik v2 docker	0	1030	April 3, 2020

gRPC service not accessible from Traefik

Related topics