Traefik v2 & cert-manager?

nsteinmetz · October 23, 2019, 8:59pm

Hello,

I'm trying to migration from V1 to V2 with the support of cert-manager which allows to have serveral replicas of traefik and avoid using 1Go volumes for acme.json

What I did : https://www.cerenit.fr/blog/kubernetes-ovh-traefik-cert-manager-secrets/

With annotations and ingress-shim from Cert-Manager, certificates management is so easy.

I tried to reproduce the mechanism but seems that annotation does not have any impact on IngressRoute object.

Trying to use a Certificate object with the traefik ingress class, seems the acme certificate is never issued. CertificateRequest stays on "pending".

If you have any clue to make it work, I'll be happy to test or I'll wait for a later release, hoping it will be possible...

Thanks !

nsteinmetz · October 23, 2019, 10:02pm

My current k8s object in a v2 context:

api-ingress.yml

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik2-web-ui
spec:
  entryPoints:
    - secure
  routes:
    - match: Host(`traefik2.k8s.cerenit.fr`)
      kind: Rule
      services:
        - name: traefik2-ingress-service-clusterip
          port: 8080
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik2-web-ui-http
spec:
  entryPoints:
    - web
  routes:
    - match: Host(`traefik2.k8s.cerenit.fr`)
      kind: Rule
      services:
        - name: traefik2-ingress-service-clusterip
          port: 8080

crd.yml

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutes.traefik.containo.us
spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRoute
    plural: ingressroutes
    singular: ingressroute
  scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutetcps.traefik.containo.us
spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRouteTCP
    plural: ingressroutetcps
    singular: ingressroutetcp
  scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: middlewares.traefik.containo.us
spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: Middleware
    plural: middlewares
    singular: middleware
  scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: tlsoptions.traefik.containo.us
spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TLSOption
    plural: tlsoptions
    singular: tlsoption
  scope: Namespaced

deployment.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: traefik2-ingress-controller
  labels:
    k8s-app: traefik2-ingress-lb
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: traefik2-ingress-lb
  template:
    metadata:
      labels:
        k8s-app: traefik2-ingress-lb
        name: traefik2-ingress-lb
    spec:
      serviceAccountName: traefik2-ingress-controller
      terminationGracePeriodSeconds: 60
      containers:
      - image: traefik:2.0.2
        name: traefik2-ingress-lb
        ports:
          - name: web
            containerPort: 80
          - name: admin
            containerPort: 8080
          - name: secure
            containerPort: 443
        readinessProbe:
          httpGet:
            path: /ping
            port: admin
          failureThreshold: 1
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        livenessProbe:
          httpGet:
            path: /ping
            port: admin
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        args:
          - "--entryPoints.web.address=:80"
          - "--entryPoints.secure.address=:443"
          - "--entryPoints.admin.address=:8080"
          - "--api.dashboard=true"
          - "--api.insecure"
          - "--ping=true"
          - "--providers.kubernetescrd"
          - "--log.level=ERROR"

lb.yml

---
kind: Service
apiVersion: v1
metadata:
  name: traefik-ingress-service-lb
spec:
  selector:
    k8s-app: traefik2-ingress-lb
  ports:
    - protocol: TCP
      port: 80
      name: web
    - protocol: TCP
      port: 443
      name: secure
  type: LoadBalancer

middleware-auth.yml

---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: auth-traefik-webui
spec:
  basicAuth:
    secret: traefik-auth

middleware-redirect-https.yml

---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: https-only
spec:
  redirectScheme:
    scheme: https

rbac.yml

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik2-ingress-controller
  namespace: traefik2
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: traefik2-ingress-controller
  namespace: traefik2
rules:
  - apiGroups:
      - ""
    resources:
      - pods
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - traefik.containo.us
    resources:
      - ingressroutes
      - ingressroutetcps
      - middlewares
      - tlsoptions
    verbs:
      - get
      - list
      - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: traefik2-ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik2-ingress-controller
subjects:
- kind: ServiceAccount
  name: traefik2-ingress-controller
  namespace: traefik2

service.yml

---
kind: Service
apiVersion: v1
metadata:
  name: traefik2-ingress-service-clusterip
spec:
  selector:
    k8s-app: traefik2-ingress-lb
  ports:
    - protocol: TCP
      port: 80
      name: web
    - protocol: TCP
      port: 8080
      name: admin
    - protocol: TCP
      port: 443
      name: secure
  type: ClusterIP

traefik.secret

traefik:REDACTED

nsteinmetz · October 29, 2019, 9:54am

Ok found a first issue, when I create a certificate within the traefik2 namespace, the certificate request is handled by traefik v1 instance as it also matches the ingress class traefik.

As it's not on the same LB and the same IP, certificate is never issued.

rosskevin · October 31, 2019, 2:58pm

I'm doing traefik v1 ingress+cert-manager ingress to v2 today - very interested if you got it ironed out.

nsteinmetz · November 1, 2019, 11:15am

I think I have a clue - I hope it's because I have both traefik v1 and traefik v2 in the same cluster and so they conflicts somehow. I tried to tuned ingressclass for Traefik2 to distinguish them but seems it did not work or I missed something.

I'll try next week by shuttng down traefik v1 and see if it works this way.

I'll keep you posted by end of next week.

nsteinmetz · November 7, 2019, 8:55am

Ok got it - I'll write a detailed blog post by end of the week but, my main findings:

You can't have two traefik instances (v1.x & v2.x) in the same cluster as they would conflict. My certificate request for traefik v2 generated by cert-manager was intercepted by traefik v1. As traefik v2 in my setup has its own LB, it could never match. So I had to turn down traefik v1. I quickly tried to customize the ingress class for traefik v2 but I didn't get it right. This would have allowed me to say cert manager to use the traefikv2 ingress class instead of default one (linked with traefik v1)
As cert-manager will pop-up a traditionnal ingress, you need to enable kubernetes-ingress also on traefik v2 side
As annotations are no longer supported on traefik v2 side, you need to create a Certificate object
As I use a wildcard zone for my k8s cluster, switching from Traefik V1 LB to Traefik v2 LB, it may require you to wait a little bit so that Lets Encrypt certificates can be generated successfully.
Kubernetes Ingress mode in traefik v2 (without annotations) allows you to switch easily from v1 to v2 and then upgrade traditional ingress to traefik IngressRoutes

michael · November 7, 2019, 11:16am

Hi @nsteinmetz,

Just to let you know, I worked on an example available here

This repo explains how to run traefik v2 and cert-manager.

nsteinmetz · November 7, 2019, 1:58pm

Hi @michael,

Indeed I have the same result at the end - except I did not use configmap for traefik.toml this time.

emile · November 8, 2019, 9:26am

FYI, I invite you to follow https://github.com/containous/traefik/issues/5792 to get updated on an official documentation on this

nsteinmetz · November 8, 2019, 5:25pm

Blog post is here: https://www.cerenit.fr/blog/kubernetes-ovh-traefik2-cert-manager-secrets/

guysoft · November 13, 2019, 1:10pm

Hey, I want to apply this on AWS EKS. Have you only tried this on GKE? Anything I should modify in particular? I have helm installed.

guysoft · November 13, 2019, 1:31pm

Hey, your traefik2/deployment.yml is not formatted correctly in the blog post

nsteinmetz · November 13, 2019, 2:49pm

I don't use helm chart so no idea how you can customize it for helm.
Not tested also on AWS nor GKE. Only OVH.

nsteinmetz · November 13, 2019, 2:51pm

If you mean the first space, for apiVersion, it's a CSS issue I can't fix yet :-/

Otherwise, it seems valid yaml - just copy/pasted content in yamllint.com and it works.

guysoft · November 13, 2019, 3:08pm

I get here

kubectl apply -f deployment.yaml 
error: error validating "deployment.yaml": error validating data: [ValidationError(Deployment): unknown field "k8s-app" in io.k8s.api.apps.v1.Deployment, ValidationError(Deployment): unknown field "matchLabels" in io.k8s.api.apps.v1.Deployment, ValidationError(Deployment): unknown field "name" in io.k8s.api.apps.v1.Deployment, ValidationError(Deployment): unknown field "replicas" in io.k8s.api.apps.v1.Deployment, ValidationError(Deployment.spec): unknown field "containers" in io.k8s.api.apps.v1.DeploymentSpec, ValidationError(Deployment.spec): unknown field "serviceAccountName" in io.k8s.api.apps.v1.DeploymentSpec, ValidationError(Deployment.spec): unknown field "terminationGracePeriodSeconds" in io.k8s.api.apps.v1.DeploymentSpec, ValidationError(Deployment.spec): missing required field "selector" in io.k8s.api.apps.v1.DeploymentSpec, ValidationError(Deployment.spec): missing required field "template" in io.k8s.api.apps.v1.DeploymentSpec]; if you choose to ignore these errors, turn validation off with --validate=false

nsteinmetz · November 13, 2019, 8:59pm

Which version of kubernetes do you use ?

nsteinmetz · November 13, 2019, 9:42pm

Oh you're right - my bad !

Correct version is:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: traefik2-ingress-controller
  labels:
    k8s-app: traefik2-ingress-lb
spec:
  replicas: 2
  selector:
    matchLabels:
      k8s-app: traefik2-ingress-lb
  template:
    metadata:
      labels:
        k8s-app: traefik2-ingress-lb
        name: traefik2-ingress-lb
    spec:
      serviceAccountName: traefik2-ingress-controller
      terminationGracePeriodSeconds: 60
      containers:
      - image: traefik:2.0.4
        name: traefik2-ingress-lb
        ports:
          - name: web
            containerPort: 80
          - name: admin
            containerPort: 8080
          - name: secure
            containerPort: 443
        readinessProbe:
          httpGet:
            path: /ping
            port: admin
          failureThreshold: 1
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        livenessProbe:
          httpGet:
            path: /ping
            port: admin
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        args:
          - --entryPoints.web.address=:80
          - --entryPoints.secure.address=:443
          - --entryPoints.traefik.address=:8080
          - --api.dashboard=true
          - --api.insecure=true
          - --ping=true
          - --providers.kubernetescrd
          - --providers.kubernetesingress
          - --log.level=DEBUG

nsteinmetz · November 13, 2019, 10:02pm

Fixed also in the blog post.

Topic		Replies	Views
Can't generate SSL certificate with cert-manager Traefik v2 kubernetes-crd , kubernetes-ingress	1	2476	May 9, 2020
KubernetesCRD config for Cert-Manager Traefik v2 kubernetes-crd , kubernetes-ingress , letsencrypt-acme	10	2640	October 1, 2022
Cert-manager http-01 challenge failing Traefik v2 kubernetes-crd , kubernetes-ingress	2	2392	April 20, 2023
Traefik v2 with Cert-Manager under Kubernetes problem Traefik v2 kubernetes-ingress , letsencrypt-acme	1	843	April 3, 2022
IngressRoute not serving cert from secret Traefik v2 kubernetes-crd , kubernetes-ingress , letsencrypt-acme	0	405	March 4, 2021

Traefik v2 & cert-manager?

Related topics