Routing GRPC through http router/service rules not working

Traefik Traefik v2
1

I am trying to route grpc through Traefik with a bunch of other https rules. The back end service I am working with supports GRPC and HTTP on the same port unencrypted.

I have my tls entrypoint:

    [http.routers.myservice]
      entryPoints = ["https"]
      rule = "Host(`myservice.domain.org`) || PathPrefix(`/namespace.ClientRPC`)"
      service = "myservice"
      priority = 1000
      [http.routers.myservice.tls]
        certResolver = "letsencrypt"
      [[http.routers.myservice.tls.domains]]
        main = "*.domain.org"
        sans = ["domain.org"]

    [http.services.myservice.loadBalancer]
      [[http.services.myservice.loadBalancer.servers]]
        url = "h2c://insideserver.domain.org:8080"

When doing plain https, it works, the service answers, no problem there. When I attempt to use a GRPC client, it doesn't match the host nor PathPrefix name. It's like it doesn't hit any of the http rules.

This is the log when I make a GRPC request:

traefik     | 192.168.3.12 - - [21/Mar/2020:15:31:20 +0000] "POST /namespace.ClientRPC/Search HTTP/2.0" - - "-" "grpc-go/1.28.0" 13 "-" "-" 0ms

I have played around with curl just to see if I could get it to trigger the rule... This works and the rule and it's routed to my grpc service.

curl -X POST --http2 'https://myservice.domain.org:443/namespace.ClientRPC/Search' -d "" -H 'Content-Type: application/grpc' -H 'Accept: application/grpc'

If I remove the https from the beginning of the URL, it's like traefik doesn't recognize it. I get the same behavior as the grpc client where it's just ignored and matches nothing. I am not totally sure if curl is calling it any differently.

curl -X POST --http2 'myservice.domain.org:443/namespace.ClientRPC/Search' -d "" -H 'Content-Type: application/grpc' -H 'Accept: application/grpc'

Any idea's? Why are grpc requests not handled/matched like http requests? (Aren't they just http2 requests?)

2

I even tried just getting it to match with HostRegexp({any:.*}) and nothing else on the entrypoint. It does not match anything coming from the GRPC client.

3 
curl -X POST --http2 'myservice.domain.org:443/namespace.ClientRPC/Search' -d "" -H 'Content-Type: application/grpc' -H 'Accept: application/grpc'

This attempts to connect via http to port 443. Not quite what you are intending to do.

Can you enable debug logging, and then have your client send a request?

The debug logs should show more request details, even ones that fail.

4

I want to make an https request using grpc at the end of the day. I was just trying to debug using CURL since when you use a GRPC client I haven't figured out how to get it to dump everything in the request.

If I enable debugging with:

[log]
  level = "DEBUG"

[accessLog]
  [accessLog.fields]
    [accessLog.fields.headers]
      defaultMode = "keep"

and I make the request with both curl and my GRPC client, the output is still just

traefik     | 192.168.1.10 - - [25/Mar/2020:14:02:05 +0000] "POST /namespace.ClientRPC/Search HTTP/2.0" - - "-" "curl/7.58.0" 1 "-" "-" 0ms
traefik     | 192.168.3.12 - - [25/Mar/2020:14:03:07 +0000] "POST /namespace.ClientRPC/Search HTTP/2.0" - - "-" "grpc-go/1.28.0" 2 "-" "-" 0ms
5

@snowzach,

Can you please provide your entire Traefik configuration?

The debug logging should show the full incoming request, and backend request.

6

That's what's so strange... when I make the request with CURL and I prefix the domain name with https it work and it does dump the whole request:

curl -X POST --http2 'https://myservice.domain.org:443/namespace.ClientRPC/Search' -d "" -H 'Content-Type: application/grpc' -H 'Accept: application/grpc'

Produces:

traefik     | time="2020-03-25T14:22:00Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/namespace.ClientRPC/Search\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"application/grpc\"],\"Content-Length\":[\"0\"],\"Content-Type\":[\"application/grpc\"],\"User-Agent\":[\"curl/7.58.0\"],\"X-Forwarded-Host\":[\"spot.prozach.org\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"63ef84bac1cc\"],\"X-Real-Ip\":[\"192.168.3.12\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"spot.prozach.org\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"192.168.3.12:46804\",\"RequestURI\":\"/namespace.ClientRPC/Search\",\"TLS\":null}"
traefik     | time="2020-03-25T14:22:00Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" ForwardURL="h2c://myservice.domain.or:8990" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/namespace.ClientRPC/Search\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"application/grpc\"],\"Content-Length\":[\"0\"],\"Content-Type\":[\"application/grpc\"],\"User-Agent\":[\"curl/7.58.0\"],\"X-Forwarded-Host\":[\"spot.prozach.org\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"63ef84bac1cc\"],\"X-Real-Ip\":[\"192.168.3.12\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"spot.prozach.org\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"192.168.3.12:46804\",\"RequestURI\":\"/namespace.ClientRPC/Search\",\"TLS\":null}"
traefik     | time="2020-03-25T14:22:02Z" level=debug msg="'499 Client Closed Request' caused by: context canceled"
traefik     | time="2020-03-25T14:22:02Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/namespace.ClientRPC/Search\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"application/grpc\"],\"Content-Length\":[\"0\"],\"Content-Type\":[\"application/grpc\"],\"User-Agent\":[\"curl/7.58.0\"],\"X-Forwarded-Host\":[\"spot.prozach.org\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"63ef84bac1cc\"],\"X-Real-Ip\":[\"192.168.3.12\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"spot.prozach.org\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"192.168.3.12:46804\",\"RequestURI\":\"/namespace.ClientRPC/Search\",\"TLS\":null}"
traefik     | 192.168.3.12 - - [25/Mar/2020:14:22:00 +0000] "POST /namespace.ClientRPC/Search HTTP/2.0" 499 21 "-" "curl/7.58.0" 2 "spot@file" "h2c://myservice.domain.or:8990" 1805ms

(The GRPC server is connected and waiting at the end there so I have to stop curl with Ctrl-C so you see the context canceled message there)

That's why I'm confused, I don't even think it's thinking it's an HTTP request at this point so maybe it doesn't process through the routers.

Some of my config is from the docker provider so it's not practical/possible for me to provide the whole config. I can try to come up with a minimal example though.

7

As well, here's the curl debug output with https missing from the hostname

curl -v -X POST --http2 'myservice.domain.org:443/namespace.ClientRPC/Search' -d "" -H 'Content-Type: application/grpc' -H 'Accept: application/grpc'
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 192.168.1.10...                                                                                                                                                                                                                   * TCP_NODELAY set
* Connected to myservice.domain.org (192.168.1.10) port 443 (#0)
> POST /namespace.ClientRPC/Search HTTP/1.1
> Host: myservice.domain.org:443
> User-Agent: curl/7.58.0                                                                                                                                                                                                                    > Connection: Upgrade, HTTP2-Settings
> Upgrade: h2c
> HTTP2-Settings: AAMAAABkAARAAAAAAAIAAAAA
> Content-Type: application/grpc
> Accept: application/grpc
> Content-Length: 0
>                                                                                                                                                                                                                                            < HTTP/1.1 101 Switching Protocols
< Connection: Upgrade
< Upgrade: h2c
* Received 101
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 404
< content-type: text/plain; charset=utf-8
< x-content-type-options: nosniff
< content-length: 19
< date: Wed, 25 Mar 2020 14:28:15 GMT
* HTTP error before end of send, stop sending
<
404 page not found
* Connection #0 to host myservice.domain.org left intact

And then with the https://

 curl -v -X POST --http2 'https://myservice.domain.org:443/namespace.ClientRPC/Search' -d "" -H 'Content-Type: application/grpc' -H 'Accept: application/grpc'
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 192.168.1.10...                                                                                                                                                                                                                   * TCP_NODELAY set
* Connected to myservice.domain.org (192.168.1.10) port 443 (#0)
* ALPN, offering h2                                                                                                                                                                                                                          * ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):                                                                                                                                                                                        * TLSv1.3 (IN), TLS handshake, Unknown (8):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):                                                                                                                                                                                        * TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Client hello (1):                                                                                                                                                                                        * TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:                                                                                                                                                                                                                        *  subject: CN=*.domain.org
*  start date: Feb 15 01:43:15 2020 GMT
*  expire date: May 15 01:43:15 2020 GMT
*  subjectAltName: host "myservice.domain.org" matched cert's "*.domain.org"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3                                                                                                                                                                              *  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):                                                                                                                                                                                                  * Using Stream ID: 1 (easy handle 0x5620f28194b0)
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
> POST /namespace.ClientRPC/Search HTTP/2
> Host: myservice.domain.org
> User-Agent: curl/7.58.0
> Content-Type: application/grpc
> Accept: application/grpc
> Content-Length: 0
>
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
<Ctrl-C>
8

@snowzach

I need the static configuration (entrypoints/provider config), and the configuration where you define that router (which looks like the file provider).

9

I figured it out... The reason this is happening is because with this config, it requires TLS. When I make my curl request without https it tries in the clear first. I assume because there's no tls, it doesn't match that router. I just created another router and omitted the tls portion and the grpc client worked. I need to tinker with my grpc client and get it to force it to try TLS first.

Thanks for being my rubber duck. Hopefully someone else finds this useful. :slight_smile:

    [http.routers.myservice]
      entryPoints = ["https"]
      rule = "Host(`myservice.domain.org`) || PathPrefix(`/namespace.ClientRPC`)"
      service = "myservice"
      priority = 1000
      [http.routers.myservice.tls]
        certResolver = "letsencrypt"
      [[http.routers.myservice.tls.domains]]
        main = "*.domain.org"
        sans = ["domain.org"]

    [http.routers.myservicenotls]
      entryPoints = ["https"]
      rule = "Host(`myservice.domain.org`) || PathPrefix(`/namespace.ClientRPC`)"
      service = "myservice"
      priority = 1000
1 Like
10

And if anyone else is having this with a GRPC client, if you are using something like LetEncrypt in front of a grpc service, you need to setup your go grpc client using the system cert pool to force it to use TLS on the connection.

	// Get the system cert pool
	certPool, err := x509.SystemCertPool()
	if err != nil {
		log.Fatalf("Could not get system cert pool: %v", err)
	}

	dialOptions := []grpc.DialOption{
		grpc.WithBlock(),
		grpc.WithTransportCredentials(credentials.NewClientTLSFromCert(certPool, "")),
	}