Connection errors for tcp endpoint via IngressRouteTCP

Traefik Traefik v2
,
1

In a nutshell, this is my goal:

vault-infra
vault-infra1280×960 213 KB

I am trying to connect to a Hashicorp Vault server listening on tcp port 8200. The Vault server itself is configured to run with tls disabled. Why? Because I want to handle tls termination at the traefik level using a letsencrypt certificate.

As a side note, I want to mention that I had this exact thing up and running with Appscode Voyager. I recently started playing with Traefik and as I start to loving it I wanted to move my entire infrastructure over. I was successful so far with all the IngressRoute resources. But IngressRouteTCP is another story. I am struggling here as I face a number of errors which I don't know to resolve.

IngressRouteTCP

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  creationTimestamp: "2019-12-18T09:13:49Z"
  generation: 2
  labels:
    app: operator
    chart: operator-0.2.0
    heritage: Helm
    release: vault
  name: vault-operator-vault
  namespace: kube-system
  resourceVersion: "19813200"
  selfLink: /apis/traefik.containo.us/v1alpha1/namespaces/kube-system/ingressroutetcps/vault-operator-vault
  uid: 2c6c15ec-9aeb-448d-9423-8285dc769af3
spec:
  entryPoints:
  - vault
  routes:
  - kind: Rule
    match: HostSNI(`vault.soosap.co`)
    services:
    - name: vault-operator
      port: 8200
      terminationDelay: 100
  tls:
    certResolver: letsencrypt
    passthrough: false

Traefik Admin UI

image
image1852×840 67.9 KB

The Traefik admin ui is giving the first indication that something is not right. Take a look at the server status error.

wget vs. curl vs vault cli
I also face a weird result when interacting with the endpoint using wget and curl. The former returns a 200 success while the latter reports an error.

$ wget -q --server-response https://vault.soosap.co:8200/v1/sys/health

  HTTP/1.1 200 OK
  Cache-Control: no-store
  Content-Type: application/json
  Date: Wed, 18 Dec 2019 10:09:40 GMT
  Content-Length: 294

$ curl -v -I https://vault.soosap.co:8200/v1/sys/health

*   Trying XX.XX.XXX.XXX:8200...
* TCP_NODELAY set
* Connected to vault.soosap.co (XX.XX.XXX.XXX) port 8200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /usr/local/etc/openssl@1.1/cert.pem
  CApath: /usr/local/etc/openssl@1.1/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=vault.soosap.co
*  start date: Dec 17 10:57:44 2019 GMT
*  expire date: Mar 16 10:57:44 2020 GMT
*  subjectAltName: host "vault.soosap.co" matched cert's "vault.soosap.co"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fe0c880f200)
> HEAD /v1/sys/health HTTP/2
> Host: vault.soosap.co:8200
> user-agent: curl/7.67.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* http2 error: Remote peer returned unexpected data while we expected SETTINGS frame.  Perhaps, peer does not support HTTP/2 properly.
* Connection #0 to host vault.soosap.co left intact
curl: (16) Error in the HTTP2 framing layer

$ vault status

Error checking seal status: Get https://vault.soosap.co:8200/v1/sys/seal-status: unexpected EOF

Traefik container logs

time="2019-12-18T10:15:58Z" level=debug msg="No secret name provided" providerName=kubernetescrd
time="2019-12-18T10:15:58Z" level=debug msg="No secret name provided" providerName=kubernetescrd
time="2019-12-18T10:15:58Z" level=debug msg="No secret name provided" providerName=kubernetescrd
time="2019-12-18T10:15:58Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-12-18T10:15:58Z" level=debug msg="Handling connection from XXX.XX.XX.XX:XXXX"
time="2019-12-18T10:15:58Z" level=error msg="Error during connection: read tcp 100.96.11.192:42832->100.96.11.157:8200: read: connection reset by peer"
time="2019-12-18T10:15:58Z" level=debug msg="Error while terminating connection: close tcp 100.96.11.192:42832->100.96.11.157:8200: shutdown: transport endpoint is not connected"
time="2019-12-18T10:15:58Z" level=debug msg="No secret name provided" providerName=kubernetescrd
time="2019-12-18T10:15:58Z" level=debug msg="No secret name provided" providerName=kubernetescrd
time="2019-12-18T10:15:58Z" level=debug msg="No secret name provided" providerName=kubernetescrd
time="2019-12-18T10:15:58Z" level=debug msg="No secret name provided" providerName=kubernetescrd

Any help to debug or understand what is going on is much appreciated :slight_smile: