Traefik advertises HTTP/2 when backend doesn't support it

nemo · September 21, 2019, 2:34am

TOML Configuration: https://github.com/captn3m0/nebula/pull/1/files#diff-6008c32764f1a0b6e4608da70e407896

Docker Labels: https://github.com/captn3m0/nebula/pull/1/files#diff-a965718d1af7fde39a65ffa650b46ac8R9-R18

Intention here is to run the following configuration on port 443:

SNI(Host1) -> Terminate TLS -> Forward to a docker service which only understands HTTP/1
SNI(*) -> Forward to a IP:Port combination, which can handle TLS

Fairly simple configuration, but I'm hitting the same issue as Traefikv2 HTTP/2.0. Traefik seems to advertise HTTP/2 on the first router, the TLS negotiation succeeds, and then Traefik forwards HTTP/2 to the backend where it fails since the backend does not support HTTP/2.

Here is a log (server is live, can be tested):

curl https://kaarana.captnemo.in:8443  -vvv
*   Trying 139.59.22.234:8443...
* TCP_NODELAY set
* Connected to kaarana.captnemo.in (139.59.22.234) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=kaarana.captnemo.in
*  start date: Sep 21 00:45:50 2019 GMT
*  expire date: Dec 20 00:45:50 2019 GMT
*  subjectAltName: host "kaarana.captnemo.in" matched cert's "kaarana.captnemo.in"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55afd7b817b0)
> GET / HTTP/2
> Host: kaarana.captnemo.in:8443
> User-Agent: curl/7.66.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* http2 error: Remote peer returned unexpected data while we expected SETTINGS frame.  Perhaps, peer does not support HTTP/2 properly.
* Connection #0 to host kaarana.captnemo.in left intact
curl: (16) Error in the HTTP2 framing layer

Forcing http1.1 using curl works:

curl https://kaarana.captnemo.in:8443  --http1.1 -Ivv
*   Trying 139.59.22.234:8443...
* TCP_NODELAY set
* Connected to kaarana.captnemo.in (139.59.22.234) port 8443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=kaarana.captnemo.in
*  start date: Sep 21 00:45:50 2019 GMT
*  expire date: Dec 20 00:45:50 2019 GMT
*  subjectAltName: host "kaarana.captnemo.in" matched cert's "kaarana.captnemo.in"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
> HEAD / HTTP/1.1
> Host: kaarana.captnemo.in:8443
> User-Agent: curl/7.66.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Sat, 21 Sep 2019 02:34:15 GMT
Date: Sat, 21 Sep 2019 02:34:15 GMT
< Server: Apache/2.4.38 (Debian)
Server: Apache/2.4.38 (Debian)
< X-Powered-By: PHP/7.3.9
X-Powered-By: PHP/7.3.9
< Link: <https://kaarana.captnemo.in:8443/index.php?rest_route=/>; rel="https://api.w.org/"
Link: <https://kaarana.captnemo.in:8443/index.php?rest_route=/>; rel="https://api.w.org/"
< Content-Type: text/html; charset=UTF-8
Content-Type: text/html; charset=UTF-8

< 
* Connection #0 to host kaarana.captnemo.in left intact

nemo · September 21, 2019, 2:43am

As a workaround, switching from the TCP router to the HTTP router worked. I assumed that one entrypoint couldn't have a mix of http and tcp, but it seems to be supported:

    "traefik.http.routers.kaarana"                  = "true"
    "traefik.http.routers.kaarana.entrypoints"      = "web-secure"
    "traefik.http.routers.kaarana.rule"             = "Host(`kaarana.captnemo.in`)"
    "traefik.http.routers.kaarana.tls.certResolver" = "default"

instead of

"traefik.tcp.routers.kaarana.rule" = "HostSNI(`kaarana.captnemo.in`)"
"traefik.tcp.routers.kaarana.tls"  = "true"
"traefik.tcp.routers.kaarana.entrypoints"                 = "web-secure"
"traefik.tcp.routers.kaarana.tls.certResolver"    = "default"
"traefik.tcp.routers.kaarana.tls.domains[0].main" = "kaarana.captnemo.in"

EDIT: This didn't fix the issue, TCP routing broke.

ldez · September 21, 2019, 2:44am

Hello,

In the v2 the static configuration and the dynamic configuration cannot be mixed by default: you have to create 2 files.

The dynamic configuration:

The static configuration:

so:

# traefik.toml
# Static configuration

[providers.docker]
  exposedByDefault = false
  network = "traefik"
  defaultRule = ""

[providers.file]
  filename = "dynamic.toml"

[entryPoints]
  [entryPoints.web]
    address = ":80"

  [entryPoints.web-secure]
    address = ":443"

[certificatesResolvers.default.acme]
  email = "certs@captnemo.in"
  storage = "/acme/acme.json"
  [certificatesResolvers.default.acme.httpChallenge]
    # used during the challenge
    entryPoint = "web"

# dynamic.toml
# Dynamic configuration


[http.middlewares]
  [http.middlewares.everything.redirectScheme]
    scheme = "https"

[tcp.routers]
  [tcp.routers.forwardtohome]
    entryPoints = ["web-secure"]
    rule = "HostSNI(`emby.bb8.fun`, `git.captnemo.in`)"
    service = "homeserver"
    [tcp.routers.forwardtohome.tls]
    passthrough = true

[tcp.services]
  [tcp.services.homeserver.loadBalancer]
    [[tcp.services.homeserver.loadBalancer.servers]]
      address = "10.8.0.14:443"

[tls.options]
  [tls.options.foo]
    minVersion = "VersionTLS12"
    cipherSuites = [
      "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
      "TLS_RSA_WITH_AES_256_GCM_SHA384"
    ]

Traefik on TCP only connect a port to another without any interaction with the content of the connection.

Why are you using TCP to handle HTTP?

ldez · September 21, 2019, 2:48am

Traefik Routers Documentation - Traefik
“HTTP & TCP on the same port? Yes, there is Traefik for that!” — Traefik 2.0. The Wait Is Over! | by Gérald Croës | containous | Medium

nemo · September 21, 2019, 2:55am

Thanks a lot!

Splitting the configuration into two worked.

Topic		Replies	Views
Forward http(s) traffic not matching SNI to outside Traefik v2 docker	2	479	June 2, 2023
Unclear how to use 2.2 http redirection Traefik v2 docker	8	866	April 11, 2020
Traefikv2 HTTP/2.0 Traefik v2 docker-swarm	6	4052	October 31, 2019
Traefik v2 TCP and HTTP Routers on Same EntryPoint Traefik v2 docker-swarm	11	6043	February 4, 2020
Info and installation improvement Traefik v2 docker , tcp	7	755	November 5, 2023

Traefik advertises HTTP/2 when backend doesn't support it

Related topics