Thanks although I think I figured out my issue.
The solution is to add passHostHeader = false
on [http.services. remote_api.loadBalancer]
For some reason I had tried to add it to [[http.services.remote_api.loadBalancer.servers]]
instead, but after re-reading the docs I found my issue.