Help with TLS Handshake Failure

The-Inamati · June 21, 2024, 3:25pm

Hello all,

I am running Traefik binary and I am having an issue with one of my services.

I have several services working correctly but this one is being stubborn and I ran out of ideas for troubleshooting the issue.

I get a Bad Gateway error when accessing the service via FQDN. Access via IP is OK. Port is reachable from Traefik VM as well.

On Traefik logs I get this

Running wireshark on the Traefik VM doesn't give me much information as well.

The Traefik configuration pertaining to this service is below:

routers:
(...)
    ap:
      entryPoints:
        - websecure
      rule: "Host(`ap.example.com`)"
(...)

services:
(...)
    ap-service:
      loadBalancer:
        serversTransport: skipverify
        servers:
          - url: "https://10.0.100.4:7443"
(...)

When connecting directly via IP I noticed that the negotiated cipher was TLS_RSA_WITH_AES_128_GCM_SHA256. I double checked and my configuration supports it.

Any help is appreciated. Thanks is advance.

habitats-tech · June 22, 2024, 3:51am

What service are you trying to access. I mean what app/server is sitting behind the service.

Consider the following:
a) try - url: "http://10.0.100.4:7443"
b) if you have to declare a serversTransports:

I know the above might be counter intuitive but when you have NGiNX fronting another app these become important if you have not specifically configured NGiNX.

bluepuma77 · June 22, 2024, 6:54am

Share your full Traefik static and dynamic config, and docker-compose.yml if used.

It doesn’t help if you only share selective parts:

The-Inamati · June 23, 2024, 2:02pm

Hello,

I am trying to reach an Access Point. I have tried with and without that option and with https and http and the result is pretty much the same.

The-Inamati · June 23, 2024, 2:19pm

Hello,

My static configuration is:

# Traefik Static Configuration

# (Global Config)
global:
  checkNewVersion: true
  sendAnonymousUsage: false

## (API)
api:
  dashboard: true

# (PILOT)
pilot:
  dashboard: false

# (Logs)
log:
  filePath: "/var/log/traefik.log"
  format: json
  level: "DEBUG"

accessLog:
  filePath: "/var/log/traefik-access.log"
  format: json
  filters:
    statusCodes:
      - "400-499"
      - "204-299"
      - "500-599"
  bufferingSize: 100

# (Providers)
providers:
  file:
    directory: "/etc/traefik/rules"
    watch: true

# (EntryPoints)
entryPoints:
  web:
    address: ":80"
    # (Optional)
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https

  websecure:
    address: ":443"
    http:
      middlewares:
        - rate-limit
        - secure-headers
      tls:
        certResolver: letsencrypt
        domains:
          - main: "domain1"
            sans:
              - "*.domain1"
          - main: "domain2"
            sans:
              - "*.domain2"
          - main: "domain3"
            sans:
              - "*.domain3"
          - main: "domain4"
            sans:
              - "*.domain4"
          - main: "domain5"
            sans:
              - "*.domain5"

# (Certificate Resolvers)
certificatesResolvers:
  letsencrypt:
    acme:
      email: myemail@domain4.com
      storage: /etc/traefik/acme/acme.json
      dnsChallenge:
        provider: cloudflare
        resolvers:
          - "1.1.1.1:53"
          - "1.0.0.1:53"

My routers and services configuration is:

http:
  serversTransports:
    skipverify:
      insecureSkipVerify: true

  routers:
    dashboard:
      entryPoints:
        - websecure
      rule: Host(`traefik.domain1`)
      middlewares:
        - Whitelist
        - auth
      service: api@internal
    sub1:
      entryPoints:
        - websecure
      rule: "Host(`sub1.domain1`)"
      middlewares:
        - Whitelist
      service: sub1-service
    sub2:
      entryPoints:
        - websecure
      rule: "Host(`sub2.domain1`)"
      middlewares:
        - Whitelist
      service: sub2-service
    sub3:
      entryPoints:
        - websecure
      rule: "Host(`sub3.domain1`)"
      middlewares:
        - Whitelist
      service: sub3-service
    ap:
      entryPoints:
        - websecure
      rule: "Host(`ap.domain1`)"
      middlewares:
        - Whitelist
      service: ap-service
    sub4:
      entryPoints:
        - websecure
      rule: "Host(`sub4.domain1`)"
      middlewares:
        - Whitelist
      service: sub4-service
    sub5:
      entryPoints:
        - websecure
      rule: "Host(`sub5.domain1`)"
      middlewares:
        - Whitelist
      service: sub5-service
    sub6:
      entryPoints:
        - websecure
      rule: "Host(`sub6.domain1`)"
      middlewares:
        - Whitelist
      service: sub6-service
    sub7:
      entryPoints:
        - websecure
      rule: "Host(`sub7.domain1`)"
      middlewares:
        - Whitelist
      service: sub7-service
    catchall1:
      entryPoints:
        - websecure
      rule: 'HostRegexp(`^.+\.domain1\.org$`) || Host(`domain1`)'
      middlewares:
        - Whitelist
      priority: 2
      service: catchall1-service
    catchall2:
      entryPoints:
        - websecure
      rule: 'HostRegexp(`^.+\.domain2\.com$`) || Host(`domain2.com`)'
      middlewares:
        - Whitelist
      service: domain2-service
    catchall3:
      entryPoints:
        - websecure
      rule: 'HostRegexp(`^.+\.domain3\.dev$`) || Host(`domain3.dev`)'
      middlewares:
        - Whitelist
      service: catchall3-service
    sub8:
      entryPoints:
        - websecure
      rule: "Host(`sub8.domain4`)"
      service: catchall4-service
    chatchall4:
      entryPoints:
        - websecure
      rule: 'HostRegexp(`^.+\.domain4\.com$`) || HostRegexp(`^.+\.domain5\.com$`) || Host(`domain4`) || Host(`domain5`)'
      middlewares:
        - Whitelist
      priority: 2
      service: catchall4-service


  services:
    sub1-service:
      loadBalancer:
        servers:
          - url: "http://10.20.30.2"
    sub2-service:
      loadBalancer:
        servers:
          - url: "http://10.20.30.3"
    sub3-service:
      loadBalancer:
        serversTransport: skipverify
        servers:
          - url: "https://10.20.30.5"
    ap-service:
      loadBalancer:
        serversTransport: skipverify
        servers:
          - url: "https://10.0.100.4:7443"
    sub4-service:
      loadBalancer:
        serversTransport: skipverify
        servers:
          - url: "https://10.20.30.1:7443"
    sub5-service:
      loadBalancer:
        serversTransport: skipverify
        servers:
          - url: "https://10.20.30.42:8006"
    sub6-service:
      loadBalancer:
        servers:
          - url: "http://10.20.40.3:8123"
    sub7-service:
      loadBalancer:
        serversTransport: skipverify
        servers:
          - url: "https://10.20.30.17:8006"
    catchall1-service:
      loadBalancer:
        serversTransport: skipverify
        servers:
          - url: "https://10.20.50.29"
    catchall2-service:
      loadBalancer:
        serversTransport: skipverify
        servers:
          - url: "https://10.20.60.2"
    catchall3-service:
      loadBalancer:
        serversTransport: skipverify
        servers:
          - url: "https://10.20.70.3"
    catchall4-service:
      loadBalancer:
        serversTransport: skipverify
        servers:
          - url: "https://10.20.80.2"

My middleware configuration is:

http:
  middlewares:
    rate-limit:
      rateLimit:
        average: 1000
        burst: 500
    secure-headers:
      headers:
        accessControlAllowMethods:
          - "GET"
          - "OPTIONS"
          - "PUT"
        accessControlMaxAge: 100
        hostsProxyHeaders: 
          - "X-Forwarded-Host"
        sslRedirect: true
        stsSeconds: 63072000
        stsIncludeSubdomains: true
        stsPreload: true
        forceSTSHeader: true
        contentTypeNosniff: true  
        browserXssFilter: true  
        referrerPolicy: "same-origin"  
        featurePolicy: "camera 'none'; geolocation 'none'; microphone 'none'; payment 'none'; usb 'none'; vr 'none';"  
        customResponseHeaders:
          X-Robots-Tag: "none,noarchive,nosnippet,notranslate,noimageindex,"
          server: ""
    Whitelist:
      ipAllowList:
        sourceRange: 
          - "20.0.0.0/8" 
    auth:
      basicAuth:
        users:
          - "user1:$apr1$7PlLQqR3$D.LKY0nz8sqgUaAZZTdMf."

I also have a tls.yaml file with:

tls:
  options:
    default:
      minVersion: VersionTLS12
      cipherSuites:
        - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
        - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
        - TLS_RSA_WITH_AES_128_GCM_SHA256

The last cipher was added as a test since that is the cipher negotiated when I access via IP directly.

bluepuma77 · June 23, 2024, 3:49pm

Can you ping and wget from host and from within Traefik container?

Share your docker-compose.yml with all the network settings.

Bad gateway indicates a network issue.

The-Inamati · June 23, 2024, 5:27pm

Hello,

There is no docker compose file. As I stated I am running Traefik as a binary directly on the VM.

Bad gateway doesn't necessarily mean a network issue. It means that the intended service is not reachable. In my case I believe it might have something to do with the TLS negotiation.

I did a traffic capture on the Traefik VM and after the client hello the server hello responds with handshake failure as you can see below.

The reason why I think it might be some configuration in Traefik is because I tested with Caddy and with HAProxy as well and it works so I assume I might be missing something on the configuration.

From the Traefik VM I can ping and also telnet to the port 7443 on the AP so no issue there.

bluepuma77 · June 23, 2024, 5:45pm

Most people here use containers, and "Bad gateway" with containers was always a network issue.

If you want to find the issue, I would get rid of all special config (like TLS), test it, then add it back, piece by piece.

The-Inamati · June 23, 2024, 9:11pm

Hello,

Already did and it's always the same issue so I assume that there is some configuration that I need to add and not remove.

bluepuma77 · June 24, 2024, 7:41am

So you fully removed tls.options and it still does not work?

This looks okay. What's the target service? Is it old? I read somewhere that Traefik disabled some old TLS versions, maybe that's the reason.

The-Inamati · June 24, 2024, 4:24pm

Hello,

The TLS version it negotiates is v1.2 so it should be fine but to be honest I have no idea. I have never seen this error before.

bluepuma77 · June 24, 2024, 4:39pm

Did you try to fully remove tls.options?

The-Inamati · June 24, 2024, 4:53pm

Hello,

Yes. The issue is the same.

bluepuma77 · June 25, 2024, 6:42am

What does that mean? Traefik via IP works, but now via domain? Does the domain resolve to the same IP? Did you use the same protocol and port?

The-Inamati · June 28, 2024, 10:49am

Hello,
Thank you very much for your help but I ended up reverting back to Caddy. I'm guessing it has something to do with the way Traefik negotiates TLS.

With Caddy everything is working.

system · July 1, 2024, 10:50am

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Reconfigure cipher suites in services/loadBalancer Traefik v2 file	2	42	November 19, 2024
TLS Handshake error Traefik v2 docker	5	3956	August 10, 2023
[solved] Remote error: tls: bad certificate" with traefik 2.3.5 Traefik v2 docker , letsencrypt-acme	0	4601	December 15, 2020
Http: TLS handshake error from : remote error: tls: unknown certificate" Traefik v2 docker	9	10420	August 2, 2023
Traefik as a loadbalancer for kube-apiserver Traefik v2 file , tcp	1	816	May 21, 2020

Help with TLS Handshake Failure

Related topics