So the problem here is the IP's as a few people are saying. The issue is that the IP that the container is presenting from is different than the IP which resolves to the DNS (because of the swarm loadbalancer) so the certificate is being labeled as invalid. There is no easy way to specify what the IP is prior to the containers deployment so we are stuck.
Now I tested this using both a File provider and a Docker label provider in v2.2 and here are the results:
| | TLS v1.2 | TLS v1.3 |
| File Provider | works | not work |
| Docker | not work. |. not work |
So in TLS v1.3 even if I specify the "url" as a dns within the file provider or using Docker it still sees the wrong IP and throws an error. However if I use the filep rovider and downgrade to v1.2 it works without using the insecureSkipVerify. So I think the feature/bug is that we should be able to specify that the Docker service use DNS for the TLS v1.2 case and then we will have to come up with a technical solution for the IP addresses later