V1 -> v2 migration | Kubernetes | multiple replica deployment | shared wildcard certificate

Traefik Traefik v2
, ,
1

Hello,

We are facing two issues in the current major version migration.

We are currently running:

Traefik: traefik:v1.7.14-alpine
Kubernetes: v1.16.2
Providers: AWS, GCP, on-prem

We have 2 namespaces, running total of 4 Traefik Deployments (3 replicas in each Deployment)

  • private-ingress namespace
    • Traefik Deployment terminating TLS (x3)
    • Traefik Deployment where ELB terminates TLS (x3)
  • public-ingress namespace
    • Traefik Deployment terminating TLS (x3)
    • Traefik Deployment where ELB terminates TLS (x3)

We managed to achieve a multi replica Deployment using v1 version of Traefik by creating a new namespace, that define a Certificate resource for cert-manager:

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: exp-1-aws-wildcard
  namespace: sys-ingress-certs
spec:
  secretName: exp-1-aws-wildcard-crt
  commonName: '*.exp-1.our.domain'
  acme:
    config:
    - dns01:
        provider: aws-route53
      domains:
        - '*.exp-1.our.domain
  issuerRef:
    name: letsencrypt
    kind: ClusterIssuer
---

We then define an Ingress in the same namespace for each of Traefik Deployments that terminate TLS (private and public) to bind the cert:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: external-ingress-proxy-exp-1
  namespace: sys-ingress-certs
  annotations:
    traefik.frontend.entryPoints: https-proto
    external-dns.alpha.kubernetes.io/target: public-ingress.exp-1.our.domain
  labels:
    kubernetes.io/ingress.class: traefik-public-tls
spec:
  rules:
  - host: public-tls-cert-loader.exp-1.our.domain
    http:
      paths:
      - path: /
        backend:
          serviceName: dummy-cert-loader
          servicePort: 443
  tls:
   - secretName: exp-1-aws-wildcard-crt

And a "dummy" Service, that doesn't point at anything:

apiVersion: v1
kind: Service
metadata:
  name: dummy-cert-loader
  namespace: sys-ingress-certs
spec:
  ports:
  - name: https-proto
    port: 443
  selector:
    k8s-app: app

This works for what we need. We are now able to define Ingress resources in other namespaces, and as long as the hostname matches the cert, Traefik serves that certificate.

We even PRed some code for v1 to make this work. (https://github.com/containous/traefik/pull/4022)

Trying to move this configuration into v2.

We are creating IngressRoute to load the certificate:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: https-ingressroute
  labels:
    kubernetes.io/ingress.class: traefik-private-tls
spec:
  entryPoints:
    - web-secure
  routes:
    - match: Host(`private-tls-cert-loader.exp-1.our.domain`)
      services:
        - name: dummy-cert-loader
          port: 80
          scheme: http
      kind: Rule
  tls:
    secretName: exp-1-aws-wildcard-crt

However, the dummy-cert-loader - now needs to be a "valid" Service to work. We create the simplest Service to satisfy this, but it needs to be 100% available to work:

apiVersion: v1
kind: Service
metadata:
  name: dummy-cert-loader
  namespace: sys-ingress-certs
spec:
  type: ExternalName
  externalName: example.com
  ports:
    - name: web
      port: 80

Lastly our existing Ingress resouces need to explicitely specify tls option in the spec for traefik to register the fact that it needs to service the certificate for these:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: grafana
  labels:
    kubernetes.io/ingress.class: traefik-private-tls
  annotations:
    external-dns.alpha.kubernetes.io/target: private-ingress.exp-1.our.domain
spec:
  rules:
    - host: grafana.exp-1.our.domain
      http:
        paths:
          - path: /
            backend:
              serviceName: grafana
              servicePort: 3000
# Null tls secret to hack tls for the ingress
  tls:
   - secretName: null

This one is not a big deal.

I have seen @daniel.tomcej suggest that this is not desired behaviour, and we should have a look at our architecture (How to use the same wildcard TLS certificate in multiple IngressRoutes in different namespaces?).

But we are running multitenanting clusters for many teams in the department and are trying to provide "Ingress Controller as a Service". That means taking care of dns and TLS for the Ingress resources teams create.

Also our Kubernetes clusters have NATed egress, to enable 3rd parties to whitelist us. This means that if we ask all our teams to run their own Traefik/cert-manager instances, we will be rate limitted by LetsEncrypt.

Finally, we have been advising our teams to run separate namespaces for different concerns within their business domains. That means that a single team can have several namespaces, all of which might require Ingresses / IngressRoutes.

For these reasons we would like to preserve the functionality we managed to get with version 1.

Thank you!

2

V1 -> v2 migration | Kubernetes | multiple replica deployment | global https redirect - resolves this issue as well.