We are running gRPC services using ECS provider. We are having an issue right now where the server registers with traefik before it is ready and results in failing requests. Our service takes about a minute to start up before it would pass health checks. We are not sure how to health check a gRPC service as it appears all gRPC requests are POSTs and I can find anything to set the method on a health check.
How do other people solve this problem?
Some other ways I can think of but none really seems ideal
- Enable retry middleware to find a healthy server and don't open the socket until the service is ready
** Does the retry middleware go through all the servers or just pick them at random? - Does traefik integrate with ECS health checks at all?
- Run health checks on a different port using http, although it doesn't seem like we can expose multiple ports in bridge mode with traefik as ECS assigns them random ports