TCP errors when using Traefik as cluster loadbalancer

Traefik Traefik v3 (latest)
,
1

I'm new to using Traefik and I want to use it as a cluster load balancer for a HA K3s install. I've got everything set up and things work, but I have the following TCP error from Traefik logs when I run kubectl get nodes.

Aug 28 10:22:16 ctrl-prime systemd[1]: Stopped traefik proxy.
Aug 28 10:22:16 ctrl-prime systemd[1]: Started traefik proxy.
Aug 28 10:22:29 ctrl-prime traefik[4681]: 2024-08-28T10:22:29Z ERR Error while handling TCP connection error="readfrom tcp 192.168.1.106:56866->192.168.1.201:6443: read tcp 192.168.1.106:6443->192.168.1.177:49733: read: connection reset by peer"
Aug 28 10:22:30 ctrl-prime traefik[4681]: 2024-08-28T10:22:30Z ERR Error while handling TCP connection error="readfrom tcp 192.168.1.106:38978->192.168.1.142:6443: read tcp 192.168.1.106:6443->192.168.1.177:49734: read: connection reset by peer"

I'm going through troubleshooting right now and would like to rule out Traefik as an issue as I think it's something to do with the K3s nodes.

This is my static config

---
global:
  checkNewVersion: true
  sendAnonymousUsage: false
tcpServersTransport:
  dialKeepAlive: -1s
  tls:
    insecureSkipVerify: true
entryPoints:
  web:
    address: ":80"
  websecure:
    address: ":443"
  kubectrl:
    address: ":6443"
    reusePort: true
providers:
  providersThrottleDuration: 30s
  file:
    watch: true
    filename: /etc/traefik/config.yml
    debugLogGeneratedTemplate: true
api:
  insecure: true
  dashboard: true
  debug: true
  disableDashboardAd: true
...

And my dynamic config

---
tcp:
  routers:
    to-k3s-control-plane:
      entryPoints:
        - "kubectrl"
      rule: "HostSNI(`*`)"
      service: "k3s-control-plane"
  services:
    k3s-control-plane:
      loadBalancer:
        servers:
        - address: "192.168.1.188:6443" # ctrl-1
        - address: "192.168.1.201:6443" # ctrl-2
        - address: "192.168.1.142:6443" # ctrl-3
...

And the Systemd unit

[Unit]
Description=Traefik edge proxy
After=network-online.target
Wants=network-online.target systemd-networkd-wait-online.service

[Service]
Restart=on-abnormal

User=traefik
Group=traefik

ExecStart=/usr/local/bin/traefik
LimitNOFILE=1048576
PrivateTmp=true
PrivateDevices=false
ProtectHome=true
ProtectSystem=full
ReadWritePaths=/etc/traefik/acme
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

The K3s nodes are all amd running on version v1.30.4+k3s1

Any help would be appreciated, thanks in advance!

2

For any one else that encounters errors like this, turns out the culprit was my host network. Make sure the public IP is accessible and not blocked by any firewall rules

https://www.reddit.com/r/k3s/comments/1dipr0p/access_the_cluster_using_kubectl/?rdt=63472

3

Another addition. It appears the real culprit was the Windows machine I was using to connect to the cluster. I tried on my Debian machines and saw no errors in the logs.

Something that helped me debug this was access logs, which I didn't have on in my config. Add this to the static config so you can see what accesses are happening Traefik Access Logs Documentation - Traefik