Traefik in linux container LXC finding docker services in another LXC, Or 1 traefik multiple LXC

I'm still not closer why some docker service from another LXC are found and accessible in the browser.
Other services are found and it looks like everything is ok in Traefik but they aren't accessible in the browser. See (Dynamic config file)

Can somebody point me to investigation tips.

Kind regards
Guy

Great, let’s start again.

So you have two LXC containers and within each you installed standard Docker CE (not Swarm) and run multiple services/containers incl. Traefik.

How do you enable inter LXC communication? Did you set up some kind of routing? Or some kind of port forwarding?

By default containers are all about isolation, so by default one Docker in LXC should not be able to talk to another Docker container in another LXC.

Yes this is what baffles me to.

I'm reading about network namespaces but the LXC don't seem to use this.
https://iximiuz.com/en/posts/container-networking-is-simple/

And look what docker network inspect socket-proxy gives.

root@traefik:/home/guyf/docker# docker network inspect socket_proxy
[
    {
        "Name": "socket_proxy",
        "Id": "b7db0b819868c8541f194df16384c0f49e1a42c9e64af1362455e67857c27798",
        "Created": "2023-09-05T16:22:14.162319445Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "192.168.91.0/24",
                    "Gateway": "192.168.91.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "06f2af482625cf1267e15bf335bbd5461d90832501382af9d3e7960dbd62f31a": {
                "Name": "traefik",
                "EndpointID": "45f44c4bf441a82bb9e185d6b112f261ae1518dc4ca39642320708314d470bce",
                "MacAddress": "02:42:c0:a8:5b:02",
                "IPv4Address": "192.168.91.2/24",
                "IPv6Address": ""
            },
            "1966c557874b543d7a6159b75ba7fc8f98665989288851e749c22413d902c743": {
                "Name": "socket-proxy",
                "EndpointID": "e8a774c9eb4785f11b75b0486006009f04bd688088eae517d0aeec09afdb67ad",
                "MacAddress": "02:42:c0:a8:5b:fe",
                "IPv4Address": "192.168.91.254/24",
                "IPv6Address": ""
            },
            "653c0c41ce447b5a19af678cec8fb098042b3905c11ca59b3954cde826eaa954": {
                "Name": "portainer",
                "EndpointID": "f46d2ada45ac5f7a3edb85eb19b99418576cb6de32b04bafc59c4953afb01f5e",
                "MacAddress": "02:42:c0:a8:5b:05",
                "IPv4Address": "192.168.91.5/24",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "socket_proxy",
            "com.docker.compose.project": "docker",
            "com.docker.compose.version": "2.18.1"
        }
    }
]

Still Traefik find some apps and plays it role as a reverse-proxy.

this is traefik.log

time="2023-09-11T17:20:10+02:00" level=info msg="Traefik version 2.9.10 built on 2023-04-06T16:15:08Z"
time="2023-09-11T17:20:10+02:00" level=info msg="Stats collection is enabled."
time="2023-09-11T17:20:10+02:00" level=info msg="Many thanks for contributing to Traefik's improvement by allowing us to receive anonymous information from your configuration."
time="2023-09-11T17:20:10+02:00" level=info msg="Help us improve Traefik by leaving this feature on :)"
time="2023-09-11T17:20:10+02:00" level=info msg="More details on: https://doc.traefik.io/traefik/contributing/data-collection/"
time="2023-09-11T17:20:10+02:00" level=info msg="Starting provider aggregator aggregator.ProviderAggregator"
time="2023-09-11T17:20:10+02:00" level=info msg="Starting provider *file.Provider"
time="2023-09-11T17:20:11+02:00" level=info msg="Starting provider *traefik.Provider"
time="2023-09-11T17:20:11+02:00" level=info msg="Starting provider *docker.Provider"
time="2023-09-11T17:20:11+02:00" level=info msg="Starting provider *acme.ChallengeTLSALPN"
time="2023-09-11T17:20:11+02:00" level=info msg="Starting provider *acme.Provider"
time="2023-09-11T17:20:11+02:00" level=info msg="Testing certificate renew..." providerName=dns-cloudflare.acme ACME CA="https://acme-v02.api.letsencrypt.org/directory"
time="2023-09-11T17:30:11+02:00" level=info msg="Anonymous stats sent to https://collect.traefik.io/9vxmmkcdmalbdi635d4jgc5p5rx0h7h8: {\"global\":{\"checkNewVersion\":true,\"sendAnonymousUsage\":true},\"serversTransport\":{\"maxIdleConnsPerHost\">
time="2023-09-11T17:30:11+02:00" level=warning msg="A new release has been found: 2.10.4. Please consider updating."

Yes. Traefik Docker Configuration Discovery finds the other Docker services/containers, that run within the same LXC. That’s expected.

What you could do is to run a second Traefik in your second LXC, forward all unknown requests from first Traefik to second. You just* need to make the second Traefik reachable from within first LCX, but at least not all services. It’s kind of hacky, there was a recent discussion with 2 configurations.

* Well, that "just" might be a bit misleading, as ChatGPT has a long 5 step process to enable this. Note that ChatGPT is sometimes wrong.
Maybe Proxmox has a simple switch to forward an external LXC port to the inside.