Traefik will use all service IPs to forward traffic round robin. If Traefik only shares one network (of two) with the service, every other forward will fail.
If you set docker.network, Traefik will only use that network to forward traffic to. It can be set globally or per service.