NOTE: This configuration worked for about a year. It was surviving docker/docker-compose updates as well as traefik ones under v1. I noticed it stopped working today and only thing which happened was I upgraded the host of new docker version (via apt update/upgrade), but I doubt it will the root cause.
UPDATE: I downgraded docker-ce and docker-ce-cli to the previous version and issue is still present. I am isolating problem to the traefik itself? But still makes me wonder what has changed since it stopped working by itself..
I am having service (some client) as a docker container accessing the Internet via nordvpn container (network_mode: container:nordvpn). So far everything works as expected. However, routing to the service via its FQDN registered through labels in docker-compose service section stopped working (gateway timeout). Tried via file settings in toml file but with the same result. If I remove the NordVPN step all works as expected but stops right after I put it back on.. Tried some other different service/client with exactly the same behavior.
Error in the browser is Gateway timed out. Domain is successfully registered inside acme.json file.
Just a note, there is no other option accessing the service this way only via registering it on the particular subdomain. When network_mode: container:nordvpn is used no ports or exposed are allowed. Also you cannot access host network from the container as well - docker limitation on the linux.
I tried upgrading nordvpn and the client to the latest version and it did not help.
INFO:
traefik:v1.7.24 (was happening at 1.7.20 as well)
Distributor ID: Debian
Description: Debian GNU/Linux 10 (buster)
Release: 10
Codename: buster
uname -a
Linux caradhras 4.19.0-8-amd64 #1 SMP Debian 4.19.98-1 (2020-01-26) x86_64 GNU/Linux
docker -v
Docker version 19.03.8, build afacb8b7f0
docker-compose -v
docker-compose version 1.25.3, build d4d1b42b
Here is the debug output while accessing the service at client.example.com:
traefik | time="2020-04-19T00:32:04Z" level=debug msg="'502 Bad Gateway' caused by: read tcp 127.0.0.1:55982->127.0.0.1:52880: read: connection reset by peer"
traefik | time="2020-04-19T00:32:04Z" level=debug msg="vulcand/oxy/forward/http: Round trip: http://127.0.0.1:52880, code: 502, Length: 11, duration: 31.519073882s tls:version: 303, tls:resume:false, tls:csuite:c02f, tls:server:client.example.com"
traefik | time="2020-04-19T00:32:04Z" level=debug msg="vulcand/oxy/forward/http: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"127.0.0.1:52880\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"],\"Accept-Encoding\":[\"gzip\"],\"Accept-Language\":[\"en-GB,en;q=0.9,sk-SK;q=0.8,sk;q=0.7,en-US;q=0.6,cs;q=0.5\"],\"Cache-Control\":[\"max-age=0\"],\"Cdn-Loop\":[\"cloudflare\"],\"Cf-Connecting-Ip\":[\"95.82.163.133\"],\"Cf-Ipcountry\":[\"CZ\"],\"Cf-Ray\":[\"586285f3cedbfc9d-VIE\"],\"Cf-Request-Id\":[\"0231720c5f0000fc9d8489f200000001\"],\"Cf-Visitor\":[\"{\\\"scheme\\\":\\\"https\\\"}\"],\"Connection\":[\"Keep-Alive\"],\"Cookie\":[\"__cfduid=dc23e9ddb2199cdf12c21581894613bf91587253874\"],\"Referer\":[\"https://client.example.com/\"],\"Sec-Fetch-Dest\":[\"document\"],\"Sec-Fetch-Mode\":[\"navigate\"],\"Sec-Fetch-Site\":[\"same-origin\"],\"Sec-Fetch-User\":[\"?1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36\"],\"X-Forwarded-For\":[\"95.82.163.133\"],\"X-Forwarded-Proto\":[\"https\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"client.example.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"172.68.51.25:58136\",\"RequestURI\":\"/\",\"TLS\":null}"
traefik | time="2020-04-19T00:32:04Z" level=debug msg="vulcand/oxy/forward: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"127.0.0.1:52880\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"],\"Accept-Encoding\":[\"gzip\"],\"Accept-Language\":[\"en-GB,en;q=0.9,sk-SK;q=0.8,sk;q=0.7,en-US;q=0.6,cs;q=0.5\"],\"Cache-Control\":[\"max-age=0\"],\"Cdn-Loop\":[\"cloudflare\"],\"Cf-Connecting-Ip\":[\"95.82.163.133\"],\"Cf-Ipcountry\":[\"CZ\"],\"Cf-Ray\":[\"586285f3cedbfc9d-VIE\"],\"Cf-Request-Id\":[\"0231720c5f0000fc9d8489f200000001\"],\"Cf-Visitor\":[\"{\\\"scheme\\\":\\\"https\\\"}\"],\"Connection\":[\"Keep-Alive\"],\"Cookie\":[\"__cfduid=dc23e9ddb2199cdf12c21581894613bf91587253874\"],\"Referer\":[\"https://client.example.com/\"],\"Sec-Fetch-Dest\":[\"document\"],\"Sec-Fetch-Mode\":[\"navigate\"],\"Sec-Fetch-Site\":[\"same-origin\"],\"Sec-Fetch-User\":[\"?1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36\"],\"X-Forwarded-For\":[\"95.82.163.133\"],\"X-Forwarded-Proto\":[\"https\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"client.example.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"172.68.51.25:58136\",\"RequestURI\":\"/\",\"TLS\":null}"
traefik | time="2020-04-19T00:32:04Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"],\"Accept-Encoding\":[\"gzip\"],\"Accept-Language\":[\"en-GB,en;q=0.9,sk-SK;q=0.8,sk;q=0.7,en-US;q=0.6,cs;q=0.5\"],\"Cache-Control\":[\"max-age=0\"],\"Cdn-Loop\":[\"cloudflare\"],\"Cf-Connecting-Ip\":[\"95.82.163.133\"],\"Cf-Ipcountry\":[\"CZ\"],\"Cf-Ray\":[\"586285f3cedbfc9d-VIE\"],\"Cf-Request-Id\":[\"0231720c5f0000fc9d8489f200000001\"],\"Cf-Visitor\":[\"{\\\"scheme\\\":\\\"https\\\"}\"],\"Connection\":[\"Keep-Alive\"],\"Cookie\":[\"__cfduid=dc23e9ddb2199cdf12c21581894613bf91587253874\"],\"Referer\":[\"https://client.example.com/\"],\"Sec-Fetch-Dest\":[\"document\"],\"Sec-Fetch-Mode\":[\"navigate\"],\"Sec-Fetch-Site\":[\"same-origin\"],\"Sec-Fetch-User\":[\"?1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36\"],\"X-Forwarded-For\":[\"95.82.163.133\"],\"X-Forwarded-Proto\":[\"https\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"client.example.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"172.68.51.25:58136\",\"RequestURI\":\"/\",\"TLS\":null}"
And from access.log:
MY_PUBLIC_IP - - [18/Apr/2020:23:49:17 +0000] "GET / HTTP/1.1" 504 15 "https://client.example.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36" 333 "Host-client-example-com-1" "http://172.17.0.3:52880" 30000ms
Any directions/feedback will be appreciated since I am clueless now 