Traefik cluster with etcd

Traefik Traefik v3 (latest)
1

I'm trying to back my traefik cluster with etcd. I've added the config to etcd, I setup a watcher before I pushed to the KV and see the values get created, however traefik complains that its watcher isn't actually seeing any changes.

PUT
/traefik/tcp/routers/k3s-api-router/entryPoints/0
k3s-api
PUT
/traefik/tcp/routers/k3s-api-router/service
k3s-api-svc
PUT
/traefik/tcp/routers/k3s-api-router/rule
HostSNI(`*`)
PUT
/traefik/tcp/routers/k3s-api-router/tls/passthrough
true
PUT
/traefik/tcp/services/k3s-api-svc/loadBalancer/servers/0/address
10.0.1.110:6443
PUT
/traefik/tcp/services/k3s-api-svc/loadBalancer/servers/1/address
10.0.1.120:6443
PUT
/traefik/tcp/services/k3s-api-svc/loadBalancer/servers/2/address
10.0.1.130:6443

Jan 07 09:59:16 traefik traefik[2727]: 2025-01-07T09:59:16-06:00 DBG github.com/traefik/traefik/v3/pkg/provider/kv/storewrapper.go:60 > WatchTree: traefik
Jan 07 09:59:16 traefik traefik[2727]: 2025-01-07T09:59:16-06:00 ERR github.com/traefik/traefik/v3/pkg/provider/kv/kv.go:127 > Provider error, retrying in 2.398054204s error="failed to watch KV: key not found in store" providerName=etcd

my traefik.yaml:

# /etc/traefik/traefik.yaml

global:
  checkNewVersion: false
  sendAnonymousUsage: false

log:
  level: DEBUG

# Entry points
entryPoints:
  web:
    address: ":80"
  websecure:
    address: ":443"
  k3s-api:
    address: ":6443"

providers:
  etcd:
    endpoints:
      - "10.0.1.140:2379"
      - "10.0.1.141:2379"
      - "10.0.1.142:2379"
    rootKey: "traefik"
    tls:
      ca: "/etc/traefik/certs/ca.pem"
      cert: "/etc/traefik/certs/traefik-client.pem"
      key: "/etc/traefik/certs/traefik-client-key.pem"

api:
  dashboard: true
  insecure: true

metrics:
  prometheus: {}

I'm going to compile a version locally and begin to debug this, will keep this thread updated if I make any progress.