How can I migrate to docker host networking?

I would like to move to docker host networking.
Currently I am using the "natting" from docker itself. Every service in docker, which I would like to proxy is in a specific docker network.
Due to this config, I am unable to whitelist based on IP, since Traefik only sees the docker IP and not the real IP.
How can I migrate my traefik + services to show the "real ips"?
When I change the docker-compose file from traefik to "network_mode: host" my service become unavailable.

Current config + an example docker service can be found here: https://paste.cerny.li/m-l9Zzs3/

When we use Traefik with Docker and configuration discovery, we see the client IP in X-Forwarded-For and X-Real-Ip HTTP headers. You can not use the IP from the incoming connection, as that is always the IP from the reverse proxy.

There is no need to use full "host networking", we usually just expose the ports 80 and 443 on the host (in our case using Docker Swarm).

services:
  traefik:
    image: traefik:v2.8.5
    command: --configFile=/traefik.yml 
    ports:
      # listen on host ports without ingress network
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
    ...