# Routing GRPC through http router/service rules not working

**URL:** <https://community.traefik.io/t/routing-grpc-through-http-router-service-rules-not-working/5048>\
**Category:** Traefik v2\
**Tags:** file\
**Created:** [March 22, 2020, 3:04am UTC](https://community.traefik.io/t/routing-grpc-through-http-router-service-rules-not-working/5048 "2020-03-22T03:04:23Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![snowzach](https://sea2.discourse-cdn.com/flex020/user_avatar/community.traefik.io/snowzach/32/1784_2.png) [@snowzach](https://community.traefik.io/u/snowzach)\
**Post date:** [March 22, 2020, 3:04am UTC](https://community.traefik.io/t/routing-grpc-through-http-router-service-rules-not-working/5048/1 "2020-03-22T03:04:23Z")

</div>

I am trying to route grpc through Traefik with a bunch of other https rules. The back end service I am working with supports GRPC and HTTP on the same port unencrypted.

I have my tls entrypoint:

```auto
    [http.routers.myservice]
      entryPoints = ["https"]
      rule = "Host(`myservice.domain.org`) || PathPrefix(`/namespace.ClientRPC`)"
      service = "myservice"
      priority = 1000
      [http.routers.myservice.tls]
        certResolver = "letsencrypt"
      [[http.routers.myservice.tls.domains]]
        main = "*.domain.org"
        sans = ["domain.org"]

    [http.services.myservice.loadBalancer]
      [[http.services.myservice.loadBalancer.servers]]
        url = "h2c://insideserver.domain.org:8080"

```

When doing plain https, it works, the service answers, no problem there. When I attempt to use a GRPC client, it doesn't match the host nor PathPrefix name. It's like it doesn't hit any of the http rules.

This is the log when I make a GRPC request:

```auto
traefik | 192.168.3.12 - - [21/Mar/2020:15:31:20 +0000] "POST /namespace.ClientRPC/Search HTTP/2.0" - - "-" "grpc-go/1.28.0" 13 "-" "-" 0ms

```

I have played around with curl just to see if I could get it to trigger the rule... This works and the rule and it's routed to my grpc service.

```auto
curl -X POST --http2 'https://myservice.domain.org:443/namespace.ClientRPC/Search' -d "" -H 'Content-Type: application/grpc' -H 'Accept: application/grpc'

```

If I remove the `https` from the beginning of the URL, it's like traefik doesn't recognize it. I get the same behavior as the grpc client where it's just ignored and matches nothing. I am not totally sure if curl is calling it any differently.

```auto
curl -X POST --http2 'myservice.domain.org:443/namespace.ClientRPC/Search' -d "" -H 'Content-Type: application/grpc' -H 'Accept: application/grpc'

```

Any idea's? Why are grpc requests not handled/matched like http requests? (Aren't they just http2 requests?)

---

<div class="post-metadata">

**Author:** ![snowzach](https://sea2.discourse-cdn.com/flex020/user_avatar/community.traefik.io/snowzach/32/1784_2.png) [@snowzach](https://community.traefik.io/u/snowzach)\
**Post date:** [March 25, 2020, 1:50pm UTC](https://community.traefik.io/t/routing-grpc-through-http-router-service-rules-not-working/5048/2 "2020-03-25T13:50:06Z")

</div>

I even tried just getting it to match with `HostRegexp(`{any:.\*}`)` and nothing else on the entrypoint. It does not match anything coming from the GRPC client.

---

<div class="post-metadata">

**Author:** ![daniel.tomcej](https://sea2.discourse-cdn.com/flex020/user_avatar/community.traefik.io/daniel.tomcej/32/262_2.png) [@daniel.tomcej](https://community.traefik.io/u/daniel.tomcej)\
**Post date:** [March 25, 2020, 1:55pm UTC](https://community.traefik.io/t/routing-grpc-through-http-router-service-rules-not-working/5048/3 "2020-03-25T13:55:23Z")

</div>

```auto
curl -X POST --http2 'myservice.domain.org:443/namespace.ClientRPC/Search' -d "" -H 'Content-Type: application/grpc' -H 'Accept: application/grpc'

```

This attempts to connect via http to port 443. Not quite what you are intending to do.

Can you enable debug logging, and then have your client send a request?

The debug logs should show more request details, even ones that fail.

---

<div class="post-metadata">

**Author:** ![snowzach](https://sea2.discourse-cdn.com/flex020/user_avatar/community.traefik.io/snowzach/32/1784_2.png) [@snowzach](https://community.traefik.io/u/snowzach)\
**Post date:** [March 25, 2020, 2:04pm UTC](https://community.traefik.io/t/routing-grpc-through-http-router-service-rules-not-working/5048/4 "2020-03-25T14:04:11Z")

</div>

I want to make an https request using grpc at the end of the day. I was just trying to debug using CURL since when you use a GRPC client I haven't figured out how to get it to dump everything in the request.

If I enable debugging with:

```auto
[log]
  level = "DEBUG"

[accessLog]
  [accessLog.fields]
    [accessLog.fields.headers]
      defaultMode = "keep"

```

and I make the request with both curl and my GRPC client, the output is still just

```auto
traefik | 192.168.1.10 - - [25/Mar/2020:14:02:05 +0000] "POST /namespace.ClientRPC/Search HTTP/2.0" - - "-" "curl/7.58.0" 1 "-" "-" 0ms
traefik | 192.168.3.12 - - [25/Mar/2020:14:03:07 +0000] "POST /namespace.ClientRPC/Search HTTP/2.0" - - "-" "grpc-go/1.28.0" 2 "-" "-" 0ms

```

---

<div class="post-metadata">

**Author:** ![daniel.tomcej](https://sea2.discourse-cdn.com/flex020/user_avatar/community.traefik.io/daniel.tomcej/32/262_2.png) [@daniel.tomcej](https://community.traefik.io/u/daniel.tomcej)\
**Post date:** [March 25, 2020, 2:13pm UTC](https://community.traefik.io/t/routing-grpc-through-http-router-service-rules-not-working/5048/5 "2020-03-25T14:13:28Z")

</div>

@snowzach,

Can you please provide your entire Traefik configuration?

The debug logging should show the full incoming request, and backend request.

---

<div class="post-metadata">

**Author:** ![snowzach](https://sea2.discourse-cdn.com/flex020/user_avatar/community.traefik.io/snowzach/32/1784_2.png) [@snowzach](https://community.traefik.io/u/snowzach)\
**Post date:** [March 25, 2020, 2:27pm UTC](https://community.traefik.io/t/routing-grpc-through-http-router-service-rules-not-working/5048/6 "2020-03-25T14:27:00Z")

</div>

That's what's so strange... when I make the request with CURL and I prefix the domain name with `https` it work and it does dump the whole request:

```auto
curl -X POST --http2 'https://myservice.domain.org:443/namespace.ClientRPC/Search' -d "" -H 'Content-Type: application/grpc' -H 'Accept: application/grpc'

```

Produces:

```auto
traefik | time="2020-03-25T14:22:00Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/namespace.ClientRPC/Search\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"application/grpc\"],\"Content-Length\":[\"0\"],\"Content-Type\":[\"application/grpc\"],\"User-Agent\":[\"curl/7.58.0\"],\"X-Forwarded-Host\":[\"spot.prozach.org\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"63ef84bac1cc\"],\"X-Real-Ip\":[\"192.168.3.12\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"spot.prozach.org\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"192.168.3.12:46804\",\"RequestURI\":\"/namespace.ClientRPC/Search\",\"TLS\":null}"
traefik | time="2020-03-25T14:22:00Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" ForwardURL="h2c://myservice.domain.or:8990" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/namespace.ClientRPC/Search\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"application/grpc\"],\"Content-Length\":[\"0\"],\"Content-Type\":[\"application/grpc\"],\"User-Agent\":[\"curl/7.58.0\"],\"X-Forwarded-Host\":[\"spot.prozach.org\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"63ef84bac1cc\"],\"X-Real-Ip\":[\"192.168.3.12\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"spot.prozach.org\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"192.168.3.12:46804\",\"RequestURI\":\"/namespace.ClientRPC/Search\",\"TLS\":null}"
traefik | time="2020-03-25T14:22:02Z" level=debug msg="'499 Client Closed Request' caused by: context canceled"
traefik | time="2020-03-25T14:22:02Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/namespace.ClientRPC/Search\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"application/grpc\"],\"Content-Length\":[\"0\"],\"Content-Type\":[\"application/grpc\"],\"User-Agent\":[\"curl/7.58.0\"],\"X-Forwarded-Host\":[\"spot.prozach.org\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"63ef84bac1cc\"],\"X-Real-Ip\":[\"192.168.3.12\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"spot.prozach.org\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"192.168.3.12:46804\",\"RequestURI\":\"/namespace.ClientRPC/Search\",\"TLS\":null}"
traefik | 192.168.3.12 - - [25/Mar/2020:14:22:00 +0000] "POST /namespace.ClientRPC/Search HTTP/2.0" 499 21 "-" "curl/7.58.0" 2 "spot@file" "h2c://myservice.domain.or:8990" 1805ms

```

(The GRPC server is connected and waiting at the end there so I have to stop curl with Ctrl-C so you see the context canceled message there)

That's why I'm confused, I don't even think it's thinking it's an HTTP request at this point so maybe it doesn't process through the routers.

Some of my config is from the docker provider so it's not practical/possible for me to provide the whole config. I can try to come up with a minimal example though.

---

<div class="post-metadata">

**Author:** ![snowzach](https://sea2.discourse-cdn.com/flex020/user_avatar/community.traefik.io/snowzach/32/1784_2.png) [@snowzach](https://community.traefik.io/u/snowzach)\
**Post date:** [March 25, 2020, 2:31pm UTC](https://community.traefik.io/t/routing-grpc-through-http-router-service-rules-not-working/5048/7 "2020-03-25T14:31:00Z")

</div>

As well, here's the curl debug output with https missing from the hostname

```auto
curl -v -X POST --http2 'myservice.domain.org:443/namespace.ClientRPC/Search' -d "" -H 'Content-Type: application/grpc' -H 'Accept: application/grpc'
Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying 192.168.1.10... * TCP_NODELAY set
* Connected to myservice.domain.org (192.168.1.10) port 443 (#0)
> POST /namespace.ClientRPC/Search HTTP/1.1
> Host: myservice.domain.org:443
> User-Agent: curl/7.58.0 > Connection: Upgrade, HTTP2-Settings
> Upgrade: h2c
> HTTP2-Settings: AAMAAABkAARAAAAAAAIAAAAA
> Content-Type: application/grpc
> Accept: application/grpc
> Content-Length: 0
> < HTTP/1.1 101 Switching Protocols
< Connection: Upgrade
< Upgrade: h2c
* Received 101
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 404
< content-type: text/plain; charset=utf-8
< x-content-type-options: nosniff
< content-length: 19
< date: Wed, 25 Mar 2020 14:28:15 GMT
* HTTP error before end of send, stop sending
<
404 page not found
* Connection #0 to host myservice.domain.org left intact

```

And then with the `https://`

```auto
 curl -v -X POST --http2 'https://myservice.domain.org:443/namespace.ClientRPC/Search' -d "" -H 'Content-Type: application/grpc' -H 'Accept: application/grpc'
Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying 192.168.1.10... * TCP_NODELAY set
* Connected to myservice.domain.org (192.168.1.10) port 443 (#0)
* ALPN, offering h2 * ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22): * TLSv1.3 (IN), TLS handshake, Unknown (8):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22): * TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Client hello (1): * TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate: * subject: CN=*.domain.org
* start date: Feb 15 01:43:15 2020 GMT
* expire date: May 15 01:43:15 2020 GMT
* subjectAltName: host "myservice.domain.org" matched cert's "*.domain.org"
* issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3 * SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (OUT), TLS Unknown, Unknown (23): * Using Stream ID: 1 (easy handle 0x5620f28194b0)
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
> POST /namespace.ClientRPC/Search HTTP/2
> Host: myservice.domain.org
> User-Agent: curl/7.58.0
> Content-Type: application/grpc
> Accept: application/grpc
> Content-Length: 0
>
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
<Ctrl-C>

```

---

<div class="post-metadata">

**Author:** ![daniel.tomcej](https://sea2.discourse-cdn.com/flex020/user_avatar/community.traefik.io/daniel.tomcej/32/262_2.png) [@daniel.tomcej](https://community.traefik.io/u/daniel.tomcej)\
**Post date:** [March 25, 2020, 2:33pm UTC](https://community.traefik.io/t/routing-grpc-through-http-router-service-rules-not-working/5048/8 "2020-03-25T14:33:04Z")

</div>

@snowzach

I need the static configuration (entrypoints/provider config), and the configuration where you define that router (which looks like the file provider).

---

<div class="post-metadata">

**Author:** ![snowzach](https://sea2.discourse-cdn.com/flex020/user_avatar/community.traefik.io/snowzach/32/1784_2.png) [@snowzach](https://community.traefik.io/u/snowzach)\
**Post date:** [March 25, 2020, 2:40pm UTC](https://community.traefik.io/t/routing-grpc-through-http-router-service-rules-not-working/5048/9 "2020-03-25T14:40:37Z")

</div>

I figured it out... The reason this is happening is because with this config, it requires TLS. When I make my curl request without `https` it tries in the clear first. I assume because there's no tls, it doesn't match that router. I just created another router and omitted the tls portion and the grpc client worked. I need to tinker with my grpc client and get it to force it to try TLS first.

Thanks for being my rubber duck. Hopefully someone else finds this useful. 🙂

```auto
    [http.routers.myservice]
      entryPoints = ["https"]
      rule = "Host(`myservice.domain.org`) || PathPrefix(`/namespace.ClientRPC`)"
      service = "myservice"
      priority = 1000
      [http.routers.myservice.tls]
        certResolver = "letsencrypt"
      [[http.routers.myservice.tls.domains]]
        main = "*.domain.org"
        sans = ["domain.org"]

    [http.routers.myservicenotls]
      entryPoints = ["https"]
      rule = "Host(`myservice.domain.org`) || PathPrefix(`/namespace.ClientRPC`)"
      service = "myservice"
      priority = 1000

```

---

<div class="post-metadata">

**Author:** ![snowzach](https://sea2.discourse-cdn.com/flex020/user_avatar/community.traefik.io/snowzach/32/1784_2.png) [@snowzach](https://community.traefik.io/u/snowzach)\
**Post date:** [March 25, 2020, 2:58pm UTC](https://community.traefik.io/t/routing-grpc-through-http-router-service-rules-not-working/5048/10 "2020-03-25T14:58:53Z")

</div>

And if anyone else is having this with a GRPC client, if you are using something like LetEncrypt in front of a grpc service, you need to setup your go grpc client using the system cert pool to force it to use TLS on the connection.

```auto
	// Get the system cert pool
	certPool, err := x509.SystemCertPool()
	if err != nil {
		log.Fatalf("Could not get system cert pool: %v", err)
	}

	dialOptions := []grpc.DialOption{
		grpc.WithBlock(),
		grpc.WithTransportCredentials(credentials.NewClientTLSFromCert(certPool, "")),
	}

```