How to have namespaced ingressroutes, middlewares, certificates and secrets in Kubernetes?

[rofz]

Hello,

I'm currently working on creating an architecture design where whoever owns a particular kubernetes namespace can manage their own ingressroutes, middleware, CERTIFICATES and SECRETS. (Is this possible?)

I've tried the scenario, but I keep getting the error below:

time="2022-07-19T15:01:33Z" level=debug msg="TLS: strict SNI enabled - No certificate found for domain: "whoami-prod.k8s.homenet.lan", closing connection"
time="2022-07-19T15:01:33Z" level=debug msg="http: TLS handshake error from 192.168.1.96:48550: tls: no certificates configured"

Note: sniStrict is set to true only due to the fact that I was always getting traefik's default certificate. It was a way to try to force it to push the correct certificate. (Maybe it's the culprit now, though I'm not entirely sure)

This is the scenario:

• traefik is in namespace 100-ingresscontroller
• app (whoami) is in namespace 200-proj1-prod
• all objects (ingressroute, middleware, certificate and its own secret) are inside namespace 200-proj1-prod (also tried creating them inside traefik's namespace, but the error was the same - maybe there's something else wrong here)

I've checked this https://community.traefik.io/t/how-to-use-my-own-certificates-into-the-configmap-configuration-k8s/2665 but would like to allow the namespace owners to create their own certificates via k8s own certificate/secret objects. The proposed solution would require access to traefik's namespace, which would be a no-go for me.

Here's my code (this is a test minikube 1.25.2/k8s 1.23.3 setup):

helm chart values:

image:
  name: traefik
  tag: ""
  pullPolicy: IfNotPresent
deployment:
  enabled: true
  kind: Deployment
  replicas: 3
  terminationGracePeriodSeconds: 60
  minReadySeconds: 0
  annotations: {}
  labels: {}
  podAnnotations: {}
  podLabels: {}
  additionalContainers: []
  additionalVolumes: []
  initContainers: []
  shareProcessNamespace: false
  imagePullSecrets: []
ingressRoute:
  dashboard:
    enabled: true
providers:
  kubernetesCRD:
    enabled: true
    allowCrossNamespace: false
    allowExternalNameServices: false
    namespaces: []
  kubernetesIngress:
    enabled: true
    allowExternalNameServices: false
    allowEmptyServices: false
    namespaces: []
logs:
  general:
    level: INFO
  access:
    enabled: true
globalArguments:
  - "--global.checknewversion=true"
  - "--global.sendanonymoususage=false"
  - "--api.insecure"
ports:
  traefik:
    port: 8100
    expose: true
    exposedPort: 8100
    protocol: TCP
  web:
    redirectTo: websecure
  websecure:
    port: 443
    expose: true
    exposedPort: 443
    protocol: TCP
    tls:
      enabled: false
      options: ""
      certResolver: ""
      domains: []
  metrics:
    port: 9100
    expose: false
    exposedPort: 9100
    protocol: TCP
tlsOptions:
  default:
    sniStrict: true
    minVersion: VersionTLS12
    cipherSuites:
      - "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"
      - "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"
      - "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256"
      - "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"
      - "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305"
      - "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"
    mintls13:
      minVersion: VersionTLS13
service:
  enabled: true
  type: LoadBalancer
  annotations: {}
  annotationsTCP: {}
  annotationsUDP: {}
  labels: {}
  spec:
    externalTrafficPolicy: Local
    clusterIP: "10.106.17.102"
  loadBalancerSourceRanges: []
  externalIPs: []
affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - topologyKey: failure-domain.beta.kubernetes.io/zone
      labelSelector:
        matchExpressions:
        - key: app
          operator: In
          values: 
          - traefik

self-signed certificate creation (w/ cert-manager) manifest:

---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: homenet-selfsigned-issuer-prod
  namespace: 200-proj1-prod
spec:
  selfSigned: {}

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: selfsigned-cert-prod
  namespace: 200-proj1-prod
spec:
  dnsNames:
    - "whoami-prod.k8s.homenet.local"
  secretName: selfsigned-cert-tls-prod
  issuerRef:
    name: homenet-selfsigned-issuer-prod

middleware + ingressroute manifest:

# Source: https://traefik.io/blog/install-and-configure-traefik-with-helm/
---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: headers-prod
  namespace: 200-proj1-prod

spec:
  headers:
    browserXssFilter: true
    contentTypeNosniff: true
    forceSTSHeader: true
    stsIncludeSubdomains: true
    stsPreload: true
    stsSeconds: 15552000
    customFrameOptionsValue: SAMEORIGIN

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: whoami-prod
  namespace: 200-proj1-prod
spec:
  entryPoints:
    - websecure
  tls:
    secretName: selfsigned-cert-tls-prod
    domains:
    - main: whoami-prod.k8s.homenet.local
      sans:
      - whoami-prod.k8s.homenet.local
  routes:
    - match: Host(`*.k8s.homenet.lan`)
      kind: Rule
      middlewares:
        - name: headers-prod
          namespace: 200-proj1-prod
      services:
        - name: whoami-prod
          namespace: 200-proj1-prod
          port: 80

traefik debug logs could not be added here due to a 32000 char limit

Any ideas?