If I'm not mistaking, option 2 won't work, as the default (when giving no argument for the network) of docker is to create a new and own bridge network for every container. That on the other hand means that both your containers will be in their own network and don't know about each other or their ports.
This is unless you bind the ports to the host (as you did for traefik ports: ...). Now they can talk to each other but are binded to the host which means that everbody connecting to the host can talk directly to them. But as you want to use traefik as a reverse-proxy, you don't want anybody other than traefik to talk directly to the other containers.
So in conclusion, you need one external network in which all your containers that should be proxied and traefik communicate. Moreover, the only container to bind ports to the host will be traefik here. That means, your first example is correct.
If no network is provided, Docker compose will create a default network, with all containers inside the compose file attached to it.
I personally prefer explicit networks, so you can manage which containers are connected to which network. For example, you can have one proxy network to connect Traefik with web apps and one database network connecting web apps with a database.
For improved security, you could create an own network for every web app with Traefik, so web apps cannot connect to each other.
I do the same thing as you said, I connect different kinds of containers to different networks, such as monitoring network, database network. But some containers don't need to share network with others, so I let them use the network that Docker Compose creates by default. It would be great if Traefik could work without connect to these networks, this is helpful in reducing the coupling between the various modules.
Most of the examples I've seen online have Traefik on the same network as the exposed containers and specifically require this or it won't work.
My question is, is it possible to not be on the same network? (I've tested it and it works, unless I'm doing something wrong) And are there any negative side effects?
Traefik with providers.docker or providers.swarm will use Docker service/container IPs from Docker networks to connect to the target service. So Traefik and target service need to be connected to the same Docker network, implicit or explicit.
Traefik with providers.docker or providers.swarm will use Docker service/container IPs from Docker networks to connect to the target service. So Traefik and target service need to be connected to the same Docker network, implicit or explicit.
Yes, I also found this in the official documentation. But it could work if Traefik and exposed containers are not connected to the same network. I don't know why. Here's my test files (Changed from the official example):
I can't really test it atm, I'm sorry. But regardless of whether this works or not, I wouldn't ever expose the container to be directly accessible, as this is what you want to avoid using traefik.
On the one hand, it is just bad practice to not use own networks for traefik. On the other hand, it is also pretty bad in terms of security. Because then somebody could still connect to the server without being routed through traefik.
Really, the only exception I would make here - and have done in my home setup - is if some container inevitably needs network_mode: host to be set on the docker container (e.g. HomeAssistant for me).