Traefik v2 & cert-manager?

Hello,

I'm trying to migration from V1 to V2 with the support of cert-manager which allows to have serveral replicas of traefik and avoid using 1Go volumes for acme.json :wink:

What I did : https://www.cerenit.fr/blog/kubernetes-ovh-traefik-cert-manager-secrets/

With annotations and ingress-shim from Cert-Manager, certificates management is so easy.

I tried to reproduce the mechanism but seems that annotation does not have any impact on IngressRoute object.

Trying to use a Certificate object with the traefik ingress class, seems the acme certificate is never issued. CertificateRequest stays on "pending".

If you have any clue to make it work, I'll be happy to test or I'll wait for a later release, hoping it will be possible...

Thanks !

2 Likes

My current k8s object in a v2 context:

api-ingress.yml

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik2-web-ui
spec:
  entryPoints:
    - secure
  routes:
    - match: Host(`traefik2.k8s.cerenit.fr`)
      kind: Rule
      services:
        - name: traefik2-ingress-service-clusterip
          port: 8080
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik2-web-ui-http
spec:
  entryPoints:
    - web
  routes:
    - match: Host(`traefik2.k8s.cerenit.fr`)
      kind: Rule
      services:
        - name: traefik2-ingress-service-clusterip
          port: 8080

crd.yml

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutes.traefik.containo.us
spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRoute
    plural: ingressroutes
    singular: ingressroute
  scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutetcps.traefik.containo.us
spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRouteTCP
    plural: ingressroutetcps
    singular: ingressroutetcp
  scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: middlewares.traefik.containo.us
spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: Middleware
    plural: middlewares
    singular: middleware
  scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: tlsoptions.traefik.containo.us
spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TLSOption
    plural: tlsoptions
    singular: tlsoption
  scope: Namespaced

deployment.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: traefik2-ingress-controller
  labels:
    k8s-app: traefik2-ingress-lb
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: traefik2-ingress-lb
  template:
    metadata:
      labels:
        k8s-app: traefik2-ingress-lb
        name: traefik2-ingress-lb
    spec:
      serviceAccountName: traefik2-ingress-controller
      terminationGracePeriodSeconds: 60
      containers:
      - image: traefik:2.0.2
        name: traefik2-ingress-lb
        ports:
          - name: web
            containerPort: 80
          - name: admin
            containerPort: 8080
          - name: secure
            containerPort: 443
        readinessProbe:
          httpGet:
            path: /ping
            port: admin
          failureThreshold: 1
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        livenessProbe:
          httpGet:
            path: /ping
            port: admin
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        args:
          - "--entryPoints.web.address=:80"
          - "--entryPoints.secure.address=:443"
          - "--entryPoints.admin.address=:8080"
          - "--api.dashboard=true"
          - "--api.insecure"
          - "--ping=true"
          - "--providers.kubernetescrd"
          - "--log.level=ERROR"

lb.yml

---
kind: Service
apiVersion: v1
metadata:
  name: traefik-ingress-service-lb
spec:
  selector:
    k8s-app: traefik2-ingress-lb
  ports:
    - protocol: TCP
      port: 80
      name: web
    - protocol: TCP
      port: 443
      name: secure
  type: LoadBalancer

middleware-auth.yml

---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: auth-traefik-webui
spec:
  basicAuth:
    secret: traefik-auth

middleware-redirect-https.yml

---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: https-only
spec:
  redirectScheme:
    scheme: https

rbac.yml

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik2-ingress-controller
  namespace: traefik2
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: traefik2-ingress-controller
  namespace: traefik2
rules:
  - apiGroups:
      - ""
    resources:
      - pods
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - traefik.containo.us
    resources:
      - ingressroutes
      - ingressroutetcps
      - middlewares
      - tlsoptions
    verbs:
      - get
      - list
      - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: traefik2-ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik2-ingress-controller
subjects:
- kind: ServiceAccount
  name: traefik2-ingress-controller
  namespace: traefik2

service.yml

---
kind: Service
apiVersion: v1
metadata:
  name: traefik2-ingress-service-clusterip
spec:
  selector:
    k8s-app: traefik2-ingress-lb
  ports:
    - protocol: TCP
      port: 80
      name: web
    - protocol: TCP
      port: 8080
      name: admin
    - protocol: TCP
      port: 443
      name: secure
  type: ClusterIP

traefik.secret

traefik:REDACTED

Ok found a first issue, when I create a certificate within the traefik2 namespace, the certificate request is handled by traefik v1 instance as it also matches the ingress class traefik.

As it's not on the same LB and the same IP, certificate is never issued.

I'm doing traefik v1 ingress+cert-manager ingress to v2 today - very interested if you got it ironed out.

I think I have a clue - I hope it's because I have both traefik v1 and traefik v2 in the same cluster and so they conflicts somehow. I tried to tuned ingressclass for Traefik2 to distinguish them but seems it did not work or I missed something.

I'll try next week by shuttng down traefik v1 and see if it works this way.

I'll keep you posted by end of next week.

Ok got it - I'll write a detailed blog post by end of the week but, my main findings:

  • You can't have two traefik instances (v1.x & v2.x) in the same cluster as they would conflict. My certificate request for traefik v2 generated by cert-manager was intercepted by traefik v1. As traefik v2 in my setup has its own LB, it could never match. So I had to turn down traefik v1. I quickly tried to customize the ingress class for traefik v2 but I didn't get it right. This would have allowed me to say cert manager to use the traefikv2 ingress class instead of default one (linked with traefik v1)
  • As cert-manager will pop-up a traditionnal ingress, you need to enable kubernetes-ingress also on traefik v2 side
  • As annotations are no longer supported on traefik v2 side, you need to create a Certificate object
  • As I use a wildcard zone for my k8s cluster, switching from Traefik V1 LB to Traefik v2 LB, it may require you to wait a little bit so that Lets Encrypt certificates can be generated successfully.
  • Kubernetes Ingress mode in traefik v2 (without annotations) allows you to switch easily from v1 to v2 and then upgrade traditional ingress to traefik IngressRoutes
1 Like

Hi @nsteinmetz,

Just to let you know, I worked on an example available here

This repo explains how to run traefik v2 and cert-manager.

2 Likes

Hi @michael,

Indeed I have the same result at the end - except I did not use configmap for traefik.toml this time.

FYI, I invite you to follow https://github.com/containous/traefik/issues/5792 to get updated on an official documentation on this :slight_smile:

2 Likes

Blog post is here: https://www.cerenit.fr/blog/kubernetes-ovh-traefik2-cert-manager-secrets/

Hey, I want to apply this on AWS EKS. Have you only tried this on GKE? Anything I should modify in particular? I have helm installed.

Hey, your traefik2/deployment.yml is not formatted correctly in the blog post

I don't use helm chart so no idea how you can customize it for helm.
Not tested also on AWS nor GKE. Only OVH.

If you mean the first space, for apiVersion, it's a CSS issue I can't fix yet :-/

Otherwise, it seems valid yaml - just copy/pasted content in yamllint.com and it works.

I get here

kubectl apply -f deployment.yaml 
error: error validating "deployment.yaml": error validating data: [ValidationError(Deployment): unknown field "k8s-app" in io.k8s.api.apps.v1.Deployment, ValidationError(Deployment): unknown field "matchLabels" in io.k8s.api.apps.v1.Deployment, ValidationError(Deployment): unknown field "name" in io.k8s.api.apps.v1.Deployment, ValidationError(Deployment): unknown field "replicas" in io.k8s.api.apps.v1.Deployment, ValidationError(Deployment.spec): unknown field "containers" in io.k8s.api.apps.v1.DeploymentSpec, ValidationError(Deployment.spec): unknown field "serviceAccountName" in io.k8s.api.apps.v1.DeploymentSpec, ValidationError(Deployment.spec): unknown field "terminationGracePeriodSeconds" in io.k8s.api.apps.v1.DeploymentSpec, ValidationError(Deployment.spec): missing required field "selector" in io.k8s.api.apps.v1.DeploymentSpec, ValidationError(Deployment.spec): missing required field "template" in io.k8s.api.apps.v1.DeploymentSpec]; if you choose to ignore these errors, turn validation off with --validate=false

Which version of kubernetes do you use ?

Oh you're right - my bad !

Correct version is:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: traefik2-ingress-controller
  labels:
    k8s-app: traefik2-ingress-lb
spec:
  replicas: 2
  selector:
    matchLabels:
      k8s-app: traefik2-ingress-lb
  template:
    metadata:
      labels:
        k8s-app: traefik2-ingress-lb
        name: traefik2-ingress-lb
    spec:
      serviceAccountName: traefik2-ingress-controller
      terminationGracePeriodSeconds: 60
      containers:
      - image: traefik:2.0.4
        name: traefik2-ingress-lb
        ports:
          - name: web
            containerPort: 80
          - name: admin
            containerPort: 8080
          - name: secure
            containerPort: 443
        readinessProbe:
          httpGet:
            path: /ping
            port: admin
          failureThreshold: 1
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        livenessProbe:
          httpGet:
            path: /ping
            port: admin
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        args:
          - --entryPoints.web.address=:80
          - --entryPoints.secure.address=:443
          - --entryPoints.traefik.address=:8080
          - --api.dashboard=true
          - --api.insecure=true
          - --ping=true
          - --providers.kubernetescrd
          - --providers.kubernetesingress
          - --log.level=DEBUG

Fixed also in the blog post.

1 Like