We've been using Traefik Proxy for quite a while now on our GKE clusters and all has been running smooth thus far, so we decided to go ahead and use Mesh as well, specially because the retry feature would come in really handy for us.
Placing the Mesh on our clusters has given us no issues so far, the deployment with the Helm chart is plain and simple and easy to configure with the values.yaml file that is provided as example in the repo. However, when it comes to make use of features such as the automatic retry , as the documentation is a bit scarce we've found ourselves lost trying to make it work.
After adding the annotation mesh.traefik.io/retry-attempts: 3 to the Service, we would be expecting that when attempting to request data from any unhealthy pod, the logs do not show any kind of evidence that the Mesh is attempting up to 3 times to deliver that request to that pod. However, we're just seeing one request, which is the initial one, returning a 500 as expected but afterwards there's no indication that Mesh has retried to deliver that request to the unhealthy pod. A simple example of what we're testing would be:
Request --> Service A --> Service B (unhealthy/scaled to 0)
Service A is making the request to service B by using the Mesh DNS routing <service-name>.<namespace>.traefik.mesh, and we've already verified that when the Service B is up and running we get the usual behaviour and the expected response is returned.
Is there something in the setup that we may be missing? Perhaps we're not properly testing the feature and so we're not able to see any indication of the retries happening.
Any kind of guidance will be highly appreciated, many thanks in advance!