Share your full Traefik static and dynamic config, and docker-compose.yml if used.
Enable and check Traefik debug log (doc) and Traefik access log in more detailed JSON format (doc).
You removed the IPs, so we can’t see if these are external or internal connections. If it’s external, a lot of people are scanning and testing servers for vulnerabilities all day long.
Seemingly related post (link).
I don’t think it’s a Docker Swarn network issue, as we usually see "Bad Gateway" errors there or connections fail because of wrong MTU, but that does not concern the first byte, but rather > 1400 bytes.