Scenario:
- kubernetes 1.27.4 with three egress nodes
- velolabcrio16
- velolabcrio17
- velolabcrio18
- Traefik 2.10.3|3.0.0
- pods deployed on all three egress nodes
- namespace 'traefik'
- Nginx 1.24.0
- serving a simply http page
- namespace 'tlstest'
- service name 'nginx'
- Testing pod
- namespace 'default'
- Cert-Manager 1.12.1 for Cluster TLS certificates
Traefik service ports, to show Node port-mapping
kubectl -n traefik get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
dashboard LoadBalancer 10.107.59.151 <pending> 8080:30046/TCP 79d
services LoadBalancer 10.111.150.165 <pending> 8153:32389/TCP,8053:30812/UDP,9100:30901/TCP,8123:31099/UDP 79d
web LoadBalancer 10.103.43.250 <pending> 8000:32562/TCP,8443:32443/TCP 79d
The pertinent portions of the Traefik deployment
kubectl -n traefik get deployment traefik -ojsonpath='{.spec.template.spec.containers}{"\n"}'
[{"args":["--api.dashboard=true","--api.insecure=true","--entrypoints.dnstcp.address=:8153/tcp","--entrypoints.dnsudp.address=:8053/udp","--entrypoints.influx.address=:8086/tcp","--entrypoints.ntp.address=:8123/udp","--entrypoints.metrics.address=:9100/tcp","--entrypoints.pgsql.address=:15432/tcp","--entrypoints.web.address=:8000/tcp","--entrypoints.websecure.address=:8443/tcp","--entrypoints.websecure.http.tls","--global.checknewversion","--global.sendanonymoususage","--log=true","--log.level=INFO","--metrics.prometheus=true","--metrics.prometheus.addrouterslabels=true","--metrics.prometheus.buckets=0.1,0.3,1.2,5.0","--metrics.prometheus.entrypoint=metrics","--ping=true","--providers.kubernetescrd","--providers.kubernetescrd.allowCrossNamespace=true","--providers.kubernetescrd.allowexternalnameservices=true","--providers.kubernetesingress"],"image":"traefik:v2.10.3","imagePullPolicy":"IfNotPresent","name":"traefik","ports":[{"containerPort":8080,"name":"dashboard","protocol":"TCP"},{"containerPort":8153,"name":"dnstcp","protocol":"TCP"},{"containerPort":8053,"name":"dnsudp","protocol":"UDP"},{"containerPort":8086,"name":"influx","protocol":"TCP"},{"containerPort":9100,"name":"metrics","protocol":"TCP"},{"containerPort":8123,"name":"ntp","protocol":"UDP"},{"containerPort":15432,"name":"pgsql","protocol":"TCP"},{"containerPort":8000,"name":"web","protocol":"TCP"},{"containerPort":8443,"name":"web-secure","protocol":"TCP"}],"resources":{},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","volumeMounts":[{"mountPath":"/data/tls/external","name":"vol-tls-external"},{"mountPath":"/data/tls/internal","name":"vol-tls-internal"}]}]
The Nginx deployment details
---
apiVersion: v1
kind: Namespace
metadata:
name: tlstest
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: nginx-tls
namespace: tlstest
spec:
commonName: tls-test
secretName: cert-secret-tls-test
usages:
- server auth
- client auth
dnsNames:
- nginx
- nginx.tlstest
issuerRef:
name: cm-ca
kind: ClusterIssuer
group: cert-manager.io
---
apiVersion: v1
kind: Secret
type: Opaque
metadata:
name: cert-secret-tls-test
namespace: tlstest
labels:
app: tls-test
---
apiVersion: v1
data:
index-html: |
<html>
<body>
<h1>Traefik serverTransport</h1>
<p>TLS testing!</p>
</body>
</html>
nginx-conf: |
daemon off;
error_log /dev/stdout debug;
events {}
http {
access_log /dev/stdout;
include /etc/nginx/conf.d/*.conf;
log_format main '$remote_addr - $remote_user [$time_local "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"';
ssl_certificate /data/tls/tls.crt;
ssl_certificate_key /data/tls/tls.key;
#ssl_prefer_server_ciphers on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
ssl_trusted_certificate /data/tls/ca.crt;
#ssl_session_timeout 5m;
}
tls-test-conf: |
server {
index index.html;
listen 80;
root /www/data;
server_name nginx-notls;
}
server {
index index.html;
listen 443 ssl;
root /www/data;
server_name nginx;
#ssl_verify_client on;
}
kind: ConfigMap
metadata:
name: tls-test
namespace: tlstest
labels:
app: tls-test
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: tls-test
namespace: tlstest
labels:
app: tls-test
spec:
replicas: 1
selector:
matchLabels:
app: tls-test
template:
metadata:
labels:
app: tls-test
spec:
containers:
- name: tls-test
image: velocitisystemengineer/alpine-tools:latest
imagePullPolicy: IfNotPresent
ports:
- name: web
containerPort: 80
- name: web-secure
containerPort: 443
command: ["/bin/sh"]
args: ["-c", "nginx -c /etc/nginx/nginx.conf"]
resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 50m
memory: 64Mi
volumeMounts:
- mountPath: "/data/tls"
name: vol-tls
- mountPath: "/etc/nginx/nginx.conf"
name: nginx-conf
subPath: "nginx.conf"
- mountPath: "/etc/nginx/conf.d/server.conf"
name: tls-test-conf
subPath: "server.conf"
- mountPath: "/www/data/index.html"
name: index-html
subPath: "index.html"
dnsPolicy: ClusterFirst
hostname: nginx
volumes:
- configMap:
defaultMode: 420
items:
- key: nginx-conf
path: nginx.conf
name: tls-test
name: nginx-conf
- configMap:
defaultMode: 420
items:
- key: tls-test-conf
path: server.conf
name: tls-test
name: tls-test-conf
- configMap:
defaultMode: 420
items:
- key: index-html
path: index.html
name: tls-test
name: index-html
- name: vol-tls
secret:
secretName: cert-secret-tls-test
defaultMode: 436
---
apiVersion: v1
kind: Service
metadata:
annotations:
traefik.ingress.kubernetes.io/service.serverstransport: traefik-tlstest
name: nginx
namespace: tlstest
labels:
app: tls-test
spec:
ports:
- name: test
port: 8080
targetPort: 80
- name: tls-test
port: 8443
targetPort: 443
selector:
app: tls-test
type: ClusterIP
---
apiVersion: traefik.io/v1alpha1
kind: ServersTransport
metadata:
name: tlstest
namespace: traefik
spec:
certificatesSecrets:
- certificate-internal
insecureSkipVerify: false
rootCAsSecrets:
- certificate-internal
serverName: "nginx"
---
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: stripprefix
namespace: tlstest
spec:
stripPrefix:
prefixes:
- "/test/"
- "/tlstest/"
forceSlash: false
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: tls-test
namespace: tlstest
labels:
app: tls-test
spec:
entryPoints:
- websecure
routes:
- kind: Rule
match: PathPrefix(`/test/`)
middlewares:
- name: stripprefix
services:
- name: nginx
port: 8080
- kind: Rule
match: PathPrefix(`/tlstest/`)
middlewares:
- name: stripprefix
services:
- name: nginx
port: 8443
serversTransport: traefik-tlstest@kubernetescrd
---
When checking for TLS connection validity/availability on the Nginx pod in tlstest namespace, from another pod in the default namespace within the cluster...
tlsint="-n default exec -ti $(kubectl -n default get pods | grep tools | awk '{print $1;}') -- curl --cacert /data/tls/ca.crt --cert /data/tls/tls.crt --key /data/tls/tls.key"
kubectl $tlsint -Lv https://nginx.tlstest:8443/
the results are good
* processing: https://nginx.tlstest:8443/
* Trying 10.101.48.236:8443...
* Connected to nginx.tlstest (10.101.48.236) port 8443
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CAfile: /data/tls/ca.crt
* CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted http/1.1
* Server certificate:
* subject: CN=tls-test
* start date: Aug 6 22:40:28 2023 GMT
* expire date: Nov 4 22:40:28 2023 GMT
* subjectAltName: host "nginx.tlstest" matched cert's "nginx.tlstest"
* issuer: CN=cm-ca
* SSL certificate verify ok.
* using HTTP/1.1
> GET / HTTP/1.1
> Host: nginx.tlstest:8443
> User-Agent: curl/8.2.1
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/1.1 200 OK
< Server: nginx/1.24.0
< Date: Thu, 17 Aug 2023 03:28:31 GMT
< Content-Type: text/html
< Content-Length: 83
< Last-Modified: Sun, 06 Aug 2023 22:40:27 GMT
< Connection: keep-alive
< ETag: "64d0215b-53"
< Accept-Ranges: bytes
<
<html>
<body>
<h1>Traefik serverTransport</h1>
<p>TLS testing!</p>
</body>
</html>
* Connection #0 to host nginx.tlstest left intact
When checking from the same testing pod to the egress Nodeport
tlsext="-n default exec -ti $(kubectl -n default get pods | grep tools | awk '{print $1;}') -- curl --cacert /data/ssl/ca.crt --cert /data/ssl/tls.crt --key /data/ssl/tls.key"
kubectl $tlsext -kLv https://velolabcrio16:32443/tlstest/
Traefik terminates the TLS session get's terminated at the entryPoint, then routes/forwards the traffic as a non-https request to the TLS enabled backend Nginx service. And yes, we did use ignore certificate errors (-k) with cURL since the entryPoint TLS certificates are self-signed, not provided by Cert-Manager
* processing: https://velolabcrio16:32443/tlstest/
* Trying 10.50.1.69:32443...
* Connected to velolabcrio16 (10.50.1.69) port 32443
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN: server accepted h2
* Server certificate:
* subject: CN=TRAEFIK DEFAULT CERT
* start date: Aug 12 16:42:57 2023 GMT
* expire date: Aug 11 16:42:57 2024 GMT
* issuer: CN=TRAEFIK DEFAULT CERT
* SSL certificate verify result: self-signed certificate (18), continuing anyway.
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* using HTTP/2
* h2 [:method: GET]
* h2 [:scheme: https]
* h2 [:authority: velolabcrio16:32443]
* h2 [:path: /tlstest/]
* h2 [user-agent: curl/8.2.1]
* h2 [accept: */*]
* Using Stream ID: 1
> GET /tlstest/ HTTP/2
> Host: velolabcrio16:32443
> User-Agent: curl/8.2.1
> Accept: */*
>
< HTTP/2 400
< content-type: text/html
< date: Thu, 17 Aug 2023 02:51:46 GMT
< server: nginx/1.24.0
< content-length: 255
<
<html>
<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<center>The plain HTTP request was sent to HTTPS port</center>
<hr><center>nginx/1.24.0</center>
</body>
</html>
* Connection #0 to host velolabcrio16 left intact
In the serversTransport configuration, changing .spec.insecureSkipVerify from 'false' to 'true' does not make a difference in the behavior/symptom. Traefik router will still send the request for PathPrefix '/tlstest' not as https. Logs from the Nginx pod confirm
kubectl -n tlstest logs $(kubectl --kubeconfig .kube/k8s.conf -n tlstest get pods | grep tls | awk '{print $1;}') -f
2023/08/16 23:12:57 [info] 7#7: *17 client sent plain HTTP request to HTTPS port while reading client request headers, client: 10.51.0.240, server: nginx, request: "GET / HTTP/1.1", host: "velolabcrio16:32443"
10.51.0.240 - - [16/Aug/2023:23:12:57 -0500] "GET / HTTP/1.1" 400 255 "-" "curl/8.2.1"
The question is, what does the configuration need to be in order for Traefik to perform TLS from the entryPoint to the TLS enabled service pod?