Hi,
I'm trying to expose an ingress port to the kubernetes dashboard running in https via Traefik, but not having much luck. I'm using k3s.
Using cert-manager I've obtained certificates for the dashboard domain and reflected these into the kubernetes-dashboard namespace. I have then configured kubernetes-dashboard to use these following this guide: dashboard/installation.md at master · kubernetes/dashboard · GitHub
Then I defined an ingress route for the dashboard:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: kubernetes-dashboard
namespace: kubernetes-dashboard
annotations:
kubernetes.io/ingress.class: "traefik"
traefik.ingress.kubernetes.io/service.serversscheme: https
spec:
rules:
- host: k8s.domain.co.uk
http:
paths:
- path: /
pathType: Exact
backend:
service:
name: kubernetes-dashboard
port:
number: 443
tls:
- secretName: kubernetes-dashboard-certs
hosts:
- k8s.domain.co.uk
This matches, but I get a 500 internal server error navigating to it, with the kubernetes dashboard logs saying
2023/03/05 14:09:22 http: TLS handshake error from 10.42.0.8:58448: remote error: tls: bad certificate
There isn't anything in the traefik logs but it seems these might be restricted to errors only and I'm not sure how to change that, some debug logging would be useful. Edit: I have since managed to get debug logs and can see level=debug msg="'500 Internal Server Error' caused by: x509: cannot validate certificate for 10.42.0.3 because it doesn't contain any IP SANs"
From googling around it does appear to be an issue with the fact that the dashboard is using a https backend. However having read Forwarding to https backend fails · Issue #7462 · traefik/traefik · GitHub I think Traefik should be able to recognize the backend is https and not to attempt to connect over http, by the port number being 443 and also the annotation. I also found this traefik - SSL issue while reaching kubernetes dashboard - DevOps Stack Exchange but not sure it applies as I am not using the self-signed certs of the dashboard. Nevertheless I did try making an IngressRoute and ServiceTransport, but having deleted the ingress above any applying these I just get a 404.
Any help would be greatly appreciated!
Have you figured it out? I'm also struggling with this.
In addition, I'm trying to route traffic through a cloudflare/argo tunnel that ends at traefik.
@HWiese1980 Sorry for the late reply -- I did manage to figure it out after a lot of trial and error. Since it was a while ago, I'm not 100% sure this was the fix but I remember adding a HelmChartConfig resource to k3s (Helm | K3s). I created this file at /var/lib/rancher/k3s/server/manifests/traefik-config.yaml:
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
name: traefik
namespace: kube-system
spec:
valuesContent: |-
globalArguments:
- "--serversTransport.insecureSkipVerify=true"
I think this causes traefik to be started with that insecureSkipVerify option which means it doesn't validate SSL certs. It's debatable as to whether that's the best option security-wise but it was fine for my use case, as my cluster is single-node and traffic only passes through the internal network.
1 Like
Hey, thanks for the reply. Yeah, that works, however, it's a global argument, so maybe not desired in each and every case. Thanks nevertheless for digging it up!
In the meantime I figured out that you can create a ServersTransport object that can be added to an IngressRoute that does exactly that, but only for a specific IngressRoute. I'll write my solution down in case anyone else runs into the same problem.
Like so
apiVersion: traefik.io/v1alpha1
kind: ServersTransport
metadata:
name: no-verify-tls
spec:
insecureSkipVerify: true
and then use it in the IngressRoute
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: some-ingress
spec:
entryPoints:
- web
routes:
- match: Host(`somedomain.mydomain.com`)
kind: Rule
services:
- name: some-service
port: 8443
serversTransport: no-verify-tls
Keep in mind though that the ServersTransport object has to be in the same namespace as the service that you address in the route, not in the namespace that your Traefik controller runs in.
Some keywords: SSL, TLS, Traefik, insecure, self-signed, certificate, local, kubernetes, k8s, ServersTransport, IngressRoute
Thank you for the excellent suggestion.
I'm running Traefik in AKS and using a self signed wildcard certificate that looks like CN *.dev.example-domain.com and SAN *.dev.example-domain.com issued by dev.example-domain.com.
I wanted to understand this a bit further, why can't Traefik use my self signed wildcard certificate and trust it so I don't have to skip TLS verification if I'm using that same wildcard cert for kubernetes-dashboard as well?
If I talk to the dashboard directly with Openssl client and provide it my cert and key I get the following error
openssl s_client -connect 10.xx.0.xx:8443 -cert tls.crt -key tls.key
error:
SSL handshake has read 1504 bytes and written 357 bytes
Verification error: unable to verify the first certificate
but if I additionally provide the ca like this:
openssl s_client -connect 10.xx.0.xx:8443 -cert tls.crt -key tls.key -CAfile ca.crt
I get SSL handshake has read 1504 bytes and written 357 bytes
Verification: OK
the verification is working correctly.
Setting the log level to TRACE I'm able to see the error when I don't skip TLS verification in my Traefik logs:
time="2024-03-06T09:42:18Z" level=debug msg="'500 Internal Server Error' caused by: tls: failed to verify certificate: x509: cannot validate certificate for 10.xx.0.xx because it doesn't contain any IP SANs"
I've added the below config to try and get traefik to recognize the ca below. what I don't understand is why openssl client does not have the IP SANs issues using the same certificate and from an IP address? If I trust the CA in my browser I'm able to browse to any Traefik ingress for any app securely, seeing my self signed cert so I know Traefik is correctly dealing with my self signed cert. it almost seems like Traefik is not using the same self signed cert to talk to the Kubernetes Dashboard.
Any insight into this would be much appreciated.
tls:
stores:
default:
rootCAs:
- "/certs/ca.crt"
defaultCertificate:
certFile: "/certs/tls.crt"
keyFile: "/certs/tls.key"
options:
default:
minVersion: VersionTLS12
sniStrict: false
options:
rootCAs:
file:
path: "/certs/ca.crt"
3 days of screaming at bad tutorials and you hit the nail. Now it works. Thank you and now I know why it just did not tell me what was wrong.