This is caused due to the additional "overlay" network; you can bypass this by using "host" networking. The only downside is that you can only run one instance of Traefik per node. That being said, you should really be spreading your Traefik instances across nodes anyway, so it isn't really a problem.
services:
traefik:
...
ports:
- target: 80
published: 80
protocol: tcp
mode: host
- target: 443
published: 443
protocol: tcp
mode: host