I have a small HTTP API that is used by terraform to store the state. I am running both Traefik and that API with docker compose. When I use that service through traefik, terraform will intermittently get stuck waiting for a response. While this happens most of the time, sometimes the requests go through as expected.
When the issue happens, the last log I will see from traefik is Service selected by WRR: http://172.19.0.2:8000. There is no corresponding log in my API, as if the request never reached it.
In order to isolate the problem, I have tried the following:
- Pointing terraform to my service directly, without traefik
- Running traefik outside of docker, but with the service itself still in docker
- Using curl to replicate terraform's requests
In all of those cases cases, everything works as intended. The combination of terraform + traefik + docker seems to be the key here. I don't know if terraform is doing something weird, but I can't find anything even with logs set to DEBUG. What is even more weird is that if traefik itself is running directly on my host, it works too.
I can work around the problem by configuring a responseHeaderTimeout, but I don't understand why I would even have to do that when everything should be working fine, as evidenced by everything working perfectly fine when traefik is running on the host.
I have made a repository with a reproducible case: