Hi - I'm having a hard time getting some Kubernetes services to work via IngressRoute with TLS, specifically kube-prometheus-stack containing Prometheus, Alertmanager and Grafana.
When creating IngressRoutes for these services using the web endpoint without tls everything works fine. As soon as I change to websecure the web uis throw 404 errors.
Other services like the Traefik dashboard and Longhorn UI work fine with TLS.
Example Grafana:
Working IngressRoute without TLS:
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: grafana-http
namespace: prometheus-stack
spec:
entryPoints:
- web
routes:
- kind: Rule
match: Host(`grafana.hnfdbu.de`)
services:
- name: prometheus-stack-grafana
port: http-web
Not working IngressRoute with TLS:
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: grafana
namespace: prometheus-stack
spec:
entryPoints:
- websecure
routes:
- kind: Rule
match: Host(`grafana.hnfdbu.de`)
services:
- name: prometheus-stack-grafana
port: http-web
tls:
secretName: hnfdbu-de-live-tls
No errors in Traefik log, no errors in Grafana log.
curl -v https://grafana.hnfdbu.de
* Host grafana.hnfdbu.de:443 was resolved.
* IPv6: (none)
* IPv4: 192.168.111.100
* Trying 192.168.111.100:443...
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / x25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
* subject: CN=*.hnfdbu.de
* start date: Nov 18 20:38:55 2024 GMT
* expire date: Feb 16 20:38:54 2025 GMT
* subjectAltName: host "grafana.hnfdbu.de" matched cert's "*.hnfdbu.de"
* issuer: C=US; O=Let's Encrypt; CN=R10
* SSL certificate verify ok.
* Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* Connected to grafana.hnfdbu.de (192.168.111.100) port 443
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://grafana.hnfdbu.de/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: grafana.hnfdbu.de]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.11.0]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: grafana.hnfdbu.de
> User-Agent: curl/8.11.0
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Request completely sent off
< HTTP/2 302
< cache-control: no-store
< content-type: text/html; charset=utf-8
< date: Thu, 21 Nov 2024 14:56:57 GMT
< location: /login
< x-content-type-options: nosniff
< x-frame-options: deny
< x-xss-protection: 1; mode=block
< content-length: 29
<
<a href="/login">Found</a>.
* Connection #0 to host grafana.hnfdbu.de left intact
One of the failed endpoints mentioned in the browser console:
curl -v https://grafana.hnfdbu.de/api/dashboards/home
* Host grafana.hnfdbu.de:443 was resolved.
* IPv6: (none)
* IPv4: 192.168.111.100
* Trying 192.168.111.100:443...
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / x25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
* subject: CN=*.hnfdbu.de
* start date: Nov 18 20:38:55 2024 GMT
* expire date: Feb 16 20:38:54 2025 GMT
* subjectAltName: host "grafana.hnfdbu.de" matched cert's "*.hnfdbu.de"
* issuer: C=US; O=Let's Encrypt; CN=R10
* SSL certificate verify ok.
* Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* Connected to grafana.hnfdbu.de (192.168.111.100) port 443
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://grafana.hnfdbu.de/api/dashboards/home
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: grafana.hnfdbu.de]
* [HTTP/2] [1] [:path: /api/dashboards/home]
* [HTTP/2] [1] [user-agent: curl/8.11.0]
* [HTTP/2] [1] [accept: */*]
> GET /api/dashboards/home HTTP/2
> Host: grafana.hnfdbu.de
> User-Agent: curl/8.11.0
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Request completely sent off
< HTTP/2 404
< content-type: text/plain; charset=utf-8
< x-content-type-options: nosniff
< content-length: 19
< date: Thu, 21 Nov 2024 15:16:07 GMT
<
404 page not found
* Connection #0 to host grafana.hnfdbu.de left intact
I hope someone can provide any hints on how to solve this as I have no idea how to track this down. As written above the same IngressRoute works for other services (except a different Host match rule) like the Traefik dashboard or Longhorn UI.
Thanks in advance,
Dennis