TCP balancing not working when container on different node

Hello, everyone.

I recently ran into a strange problem when testing Traefik's performance in Docker Swarm. The task is quite simple, to make a TCP balancer for Zabbix Proxy, which would move freely across the cluster, but there would be a single entry point for Zabbix server.
The problem is that when Traefik and Zabbix containers are on the same node - everything works as it should. But as soon as the proxy is moved to another node - the connection to the Zabbix server is dropped.
In Traefik dashboard there are no changes, router and service are located correctly. Everything is tested in a lab environment, no firewalls or closed ports.
I created the overlay network beforehand
docker network create --driver overlay --attachable true --name traefik-proxy
What I noticed is that the docker network inspect command shows different results on different nodes. It only sees connected addresses on its own node.
The point of using Traefik is that it knows which node the service is on and dynamically adjusts the balancing so that it doesn't load other nodes, like the built-in swarm ingress balancer.

Thank you all in advance for any help you can give me

Here is the config of Traefik (v3.3.4)

services:
  traefik:
    image: traefik:latest
    hostname: '{{.Node.Hostname}}'
    deploy:
      mode: global
      placement:
        constraints:
          - node.role==manager
      labels:
        - traefik.enable=true
        - traefik.http.routers.mydashboard.rule=Host(`traefik.example.com`) && PathPrefix(`/dashboard`) || PathPrefix(`/api`)
        - traefik.http.routers.mydashboard.service=api@internal
        - traefik.http.routers.mydashboard.middlewares=myauth
        - traefik.http.services.dummy-svc.loadbalancer.server.port=777
        - traefik.http.routers.mydashboard.tls=true
        - traefik.http.middlewares.myauth.basicauth.users=test:$$apr1$$H6uskkkW$$IgXLP6ewTrSuBkTrqE8wj/
    command:
      - --api.dashboard=true
      - --log.level=INFO
      - --accesslog=true
      - --providers.swarm.exposedByDefault=false
      - --providers.swarm.network=traefik-proxy
      - --entrypoints.web.http.redirections.entrypoint.to=websecure
      - --entryPoints.web.http.redirections.entrypoint.scheme=https
      - --entrypoints.websecure.asDefault=true
      - --providers.file.directory=/etc/traefik/dynamic
      - --entrypoints.web.address=:80
      - --entrypoints.websecure.address=:443
      - --entrypoints.zabbix_passive.address=:10050
      - --entrypoints.zabbix_active.address=:10051
    networks:
      - traefik-proxy
    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
      - target: 10050
        published: 10050
        protocol: tcp
        mode: host
      - target: 10051
        published: 10051
        protocol: tcp
        mode: host
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    configs:
      - source: traefik_config
        target: /etc/traefik/dynamic/traefik-certs.yaml
      - source: traefik_cert
        target: /certs/cert.crt
      - source: traefik_key
        target: /certs/key.key

configs:
  traefik_config:
    file: traefik-certs.yaml
  traefik_cert:
    file: cert.crt
  traefik_key:
    file: key.key

networks:
  traefik-proxy:
    external: true

Here is the config of Zabbix proxy

services:
  zabbix-proxy:
    image: zabbix/zabbix-proxy-sqlite3:7.0-alpine-latest
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
      labels:
        - "traefik.enable=true"
        - "traefik.tcp.routers.zabbix-proxy-active.rule=HostSNI(`*`)"
        - "traefik.tcp.routers.zabbix-proxy-active.entrypoints=zabbix_active"
        - "traefik.tcp.services.zabbix-proxy-active.loadbalancer.server.port=10051"
        - "traefik.tcp.routers.zabbix-proxy-active.tls=false"
        - "traefik.tcp.routers.zabbix-proxy-passive.rule=HostSNI(`*`)"
        - "traefik.tcp.routers.zabbix-proxy-passive.entrypoints=zabbix_passive"
        - "traefik.tcp.services.zabbix-proxy-passive.loadbalancer.server.port=10050"
        - "traefik.tcp.routers.zabbix-proxy-passive.tls=false"
        - "traefik.swarm.network=traefik-proxy"
        - "traefik.tcp.routers.zabbix-proxy-active.service=zabbix-proxy-active"
        - "traefik.tcp.routers.zabbix-proxy-passive.service=zabbix-proxy-passive"
    environment:
      - ZBX_HOSTNAME=ZabbixProxy
      - ZBX_SERVER_HOST=zabbix.example.com
    networks:
      - traefik-proxy

networks:
  traefik-proxy:
    external: true

What does docker node ls tell you?

Is your Docker overlay network okay? No issues with MTU because of VLANs? Have you tried a simple ping between nodes in Docker network with payload > 1500 bytes?

That everything's all good. However, one time I had to rebuild a cluster because a worker node would not come out of the ‘Down’ status after a reboot, even though it was listed as ‘Active’
As for the network, I'm having trouble telling what's wrong with it. It exists. Services are created without errors. IP addresses are assigned, nslookup resolves container names between nodes. Pings go back and forth of any size ping -s 100 and ping -s 2000 give the same result.
Both nodes are indeed in a dedicated VLAN, but so far this has not affected Docker in any way

BTW, both Docker and Traefik are latest version available

You think this worked and the cluster is healthy? Or can you try a full rebuild?

Rebuild didn't help. But I think I'm starting to see what's wrong. There are two network adapters inside the container. One has the address of my overlay network, and the other appears to be the address of the gwbridge. When I traceroute to the zabbix server, the hops go through the bridge address.
That's probably why I'm having problems on the second node. On it, the zabbix proxy communicates with the server through its bridge interface instead of redirecting traffic through the balancer. And the zabbix server looks just at Traefik's IP address and does not respond to requests from the wrong address