insecureSkipVerify explanation

aleksvujic · October 7, 2019, 8:22am

I am using the certificate that we purchased. I configured my loadbalancer server to use https scheme like so:

traefik.http.services.test-service.loadbalancer.server.port=443
traefik.http.services.test-service.loadbalancer.server.scheme=https

When I want to access the server, I get the following error:

'500 Internal Server Error' caused by: x509: cannot validate certificate for 10.0.7.237 because it doesn't contain any IP SANs"

If I add the following code to static configuration the error goes away and everything works fine:

serversTransport:
  insecureSkipVerify: true

I read the documentation about insecureSkipVerify but it is very short and doesn't describe it in detail. I have 3 questions:

What exactly does insecureSkipVerify do?
Is it safe to set it to true in production environment with sensitive user data (passwords)?
Is there any way to not use insecureSkipVerify but get it working? I read about TCP routers which allow passthrough, but then I would have to use TCP service as well. I have to support sticky sessions. It seems to me that TCP services don't support them.

Ch1ch1 · October 7, 2019, 9:32am

Hi @aleksvujic

serversTransport:
  insecureSkipVerify: true

Disables SSL certificate verification between your traefik instance and your backend.
This means that your certificate does not need to be a valid one.

You can safely use this configuration in production, the fact that the certificate is not valid on your internal network is not a security concern, you flow will still be encrypted.

You can set insecureSkipVerify to false and bring the ca certificate to traefik, this way traefik can validate the certificate :

serversTransport:
  insecureSkipVerify: false
  rootCAs:
    - yourCAcert.crt

aleksvujic · October 7, 2019, 9:54am

Documentation states the following:

rootCAs is the list of certificates (as file paths, or data bytes) that will be set as Root Certificate Authorities when using a self-signed TLS certificate.

I am not using self-signed certificate. Does this still apply for certificates that are not self-signed? Do we need to provide path to certificate inside container?

Ch1ch1 · October 7, 2019, 10:07am

If you are using valid certificates, and want to use sticky sessions.

You have to terminate your ssl connexion at traefik and communicate with your backend with or without ssl. (with an other certificate).

aleksvujic · October 7, 2019, 10:13am

I provided path to rootCA file inside container but I am still getting the same error:

serversTransport:
  insecureSkipVerify: false
  rootCAs:
    - /cert/mycert.crt

Ch1ch1 · October 7, 2019, 12:24pm

Server transport is for the certificate between traefik and your backend.

You have to define the certificate to use for traefik to crypt the connection (traefik <-> client)

https://docs.traefik.io/https/tls/#certificates-definition

aleksvujic · October 7, 2019, 12:26pm

Traefik <=> client connection is already secured (HTTPS). I want to secure Traefik <=> server(s) so I think my approach is correct. I just don't understand why it still throws the error below even if I specified rootCA.

'500 Internal Server Error' caused by: x509: cannot validate certificate for 10.0.7.237 because it doesn't contain any IP SANs"

Ch1ch1 · October 7, 2019, 12:29pm

Could you please provide your configurations.

aleksvujic · October 7, 2019, 12:38pm

docker-compose-Traefik.yml

version: "3.7"

services:
  traefik:
    image: "traefik:v2.0"
    networks:
      - traefik-net
    ports:
      - "9000:9000"
      - "9191:9191"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "./TraefikConfig/:/etc/traefik/"
      - "/home/admin/cert/:/cert/"
    deploy:
      replicas: 1
      labels:
        # enable Traefik for this service
        - "traefik.enable=true"
        - "traefik.docker.network=traefik-net"

        # dashboard
        - "traefik.http.routers.traefik.rule=Host(`website.example.com`)"
        - "traefik.http.routers.traefik.entrypoints=traefik"
        - "traefik.http.routers.traefik.tls=true"
        - "traefik.http.routers.traefik.service=api@internal"

        # service
        - "traefik.http.services.traefik.loadbalancer.server.port=8080"
        - "traefik.http.services.traefik.loadbalancer.server.scheme=https"

networks:
  traefik-net:
    external: true
    name: traefik-net

docker-compose-Application.yml

version: "3.7"
services:
  web:
    image: myimage:latest
    networks:
      - traefik-net
    deploy:
      replicas: 3
      labels:
        # enable Traefik for this service
        - "traefik.enable=true"
        - "traefik.docker.network=traefik-net"

        # router
        - "traefik.http.routers.application.rule=Host(`website.example.com`)"
        - "traefik.http.routers.application.entrypoints=my-entrypoint
        - "traefik.http.routers.application.tls=true"
        - "traefik.http.routers.application.service=application"

        # service
        - "traefik.http.services.application.loadbalancer.server.port=443"
        - "traefik.http.services.application.loadbalancer.server.scheme=https"
        - "traefik.http.services.application.loadbalancer.sticky=true"
        - "traefik.http.services.application.loadbalancer.sticky.cookie.name=StickyCookieTest"
        - "traefik.http.services.application.loadbalancer.sticky.cookie.secure=true"

networks:
  traefik-net:
    external: true
    name: traefik-net

TraefikConfig/traefik.yml

log:
  level: DEBUG

api:
  dashboard: true
  insecure: false

serversTransport:
  insecureSkipVerify: false
  rootCAs:
    - /cert/rootCertificate.crt

providers:
  file:
    directory: "/etc/traefik"
    watch: true
  docker:
    swarmMode: true
    exposedByDefault: false

entrypoints:
  traefik:
    address: ":9000"
  my-entrypoint:
    address: ":9191"

TraefikConfig/dynamic_conf.yml

tls:
  stores:
    default:
      defaultCertificate:
        certFile: /cert/example.com.crt
        keyFile: /cert/example.com.key
  certificates:
    - certFile: /cert/example.com.crt
      keyFile: /cert/example.com.key
      stores:
        - default

I have the following files in /home/admin/cert/ directory:

example.com.crt
example.com.key
rootCertificate.crt

Ch1ch1 · October 7, 2019, 12:43pm

The error in the log remind me one thing:

When you created your self signed certificate have you fill this part : CN=something.com

Sorry for my missunderstanding of your problem..
I think it comes from your certificate, the insecureSkipVerify: true configuration should validate all self signed certificate

aleksvujic · October 7, 2019, 12:55pm

CN in certificate is set to *.example.com.

I don't understand what you mean by that. I want to use my certificate for Traefik <=> servers communication as well, I don't want to allow insecure communication. I noticed the following:

If I add insecureSkipVerify: true and remove rootCAs section -> WORKS
If I add insecureSkipVerify: true and keep rootCAs section -> ERROR
If I remove insecureSkipVerify: true and keep rootCAs section -> ERROR
If I remove insecureSkipVerify: true and remove rootCAs section -> ERROR

Ch1ch1 · October 7, 2019, 1:33pm

The case 1 is working as intended with your configuration.

(the case 3 should work too.. that is the part that i don't understand.. !)
I'll have to run some tests and will write down the results here.

But if you go with the case 1, i think there is no security concern, and all communications will be encrypted. There is no need for the internal certificate to be a valid one, if i understand your concern : you just want the internal communications to be encrypted.

aleksvujic · October 7, 2019, 1:39pm

I tried just one more time to be sure and I can confirm that case 3 doesn't work. When you run the tests, please post the results here.

Yes correct, you understand my concern correctly, I want internal (Traefik<=>servers) communication to be encrypted as well.

Ch1ch1 · October 7, 2019, 1:52pm

Ok i'll comme back to you with my results.

aleksvujic · October 14, 2019, 5:30am

@Ch1ch1 Any update on this?

dash042 · October 25, 2019, 5:05pm

I have the same issue... need to use insecureskipverify=true
I am using kubernetes-CRD and I can't find an example configuration - so if anyone has gotten this to work with self-signed (private CA) CRTs - I'd love an exchange

Jaans · October 27, 2019, 2:15am

@aleksvujic Just a thought, but do you have the full certificate chain configured on your load balancers?

My interpretation of the insecureSkipVerify setting is to ignore SSL/TLS validations between Traefik and backend servers (in your case the load balancers).

Additionally, my understanding of the rootCAs setting is that it only applies to self-signed certificates.

So it would seem that Traefik is finding the certificate presented on the load balancer service to be invalid, possibly because the certificate there is invalid or the full chain isn't present on that service.

You could try to install the intermediate / root certificates on the load balancer, or if you are using PEM format, to include the intermediate/root certs below the existing server cert.

Update: I almost forgot, the error about IP SANs makes me wonder if Traefik is connecting to the backend server by IP address instead of DNS name (which should match the certificate domains). Not sure if Traefik can even use DNS for backend services

aleksvujic · November 4, 2019, 7:35am

I have thought of this as well. Can somebody from the Traefik team clarify how Traefik addresses servers? Does it use IP address or host name? If it uses IP address, then the error is obvious. Can we force Traefik to address servers by hostname?

zespri · November 4, 2019, 8:34am

@aleksvujic I think there is some confusion in regrads to two separate communctions:

Browser <==> traefik
Traefik <==> your app

We are talking about number 2, right? Did you bake the certificates into your application image, along with private keys? I do not see you mounting them in a volume in your config.

aleksvujic · November 4, 2019, 8:56am

@zespri Yes, we are talking about number 2. Containers have a directory with necessary certificates mounted, I skipped it in my initial post to keep it as simple as possible.

We have a .NET Core app and we provided the certificate information in docker-compose file like so:

version: "3.7"
services:
  web:
    image: myimage:0.0.1
    environment:
      ASPNETCORE_ENVIRONMENT: Development
      ASPNETCORE_URLS: https://+443;http://+80
      ASPNETCORE_HTTPS_PORT: 443
      ASPNETCORE_Kestrel__Certificates__Default__Password: password
      ASPNETCORE_Kestrel__Certificates__Default__Path: /https/certificate.pfx
    volumes:
      - "/home/admin/Cert:/https/"

With this configuration, containers are able to accept HTTPS requests, but only if I use insecureSkipVerify. But I am not sure why I have to use this flag.

Topic		Replies	Views
insecureSkipVerify How to apply this at a service level in docker compose Traefik v2 docker , docker-swarm	6	5815	October 23, 2022
InsecureSkipVerify = true Traefik v2 docker	4	4349	August 19, 2021
insecureSkipVerify on internal network Traefik v2 docker , letsencrypt-acme	0	965	November 17, 2020
Traefik 2.9.x and Unifi-Controller as backend - internal server error 500 Traefik v2 docker	1	1340	April 24, 2023
ServerTransport and insecureSkipVerify specific hosts Traefik v2 docker , file	2	2940	September 21, 2022

insecureSkipVerify explanation

Related topics