How to configure middleware with kubernetes-ingress? (middleware "xyz" does not exist)

Traefik Traefik v2
,
1

Hi there,

I'm using Traefik 2.2.0-rc3 on a new k8s cluster, basically set up as documented on docs.traefik.io. All in all it works, but I can't figure out how to get middlewares working.

I have also tried the kubernetes-crd setup, where adding middlewares wasn't a problem, but I prefer the kubernetes-ingress way for its smaller deployment manifests.

Adding the router.middlewares annotation in the config shown below only let to an error message:

middleware "traefik-basic-auth@kubernetescrd" does not exist

kind: Ingress
apiVersion: networking.k8s.io/v1beta1
metadata:
  name: traefik-dashboard
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.middlewares: traefik-basic-auth@kubernetescrd
    traefik.ingress.kubernetes.io/router.tls.certresolver: letsencrypt
spec:
  rules:
    - host: traefik.example.com
      http:
        paths:
          - path: /
            backend:
              serviceName: traefik
              servicePort: 8080

Here is the rest of my deployment file that might be relevant:

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: middlewares.traefik.containo.us
spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: Middleware
    plural: middlewares
    singular: middleware
  scope: Namespaced
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - traefik.containo.us
    resources:
      - middlewares
    verbs:
      - get
      - list
      - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
  - kind: ServiceAccount
    name: traefik-ingress-controller
    namespace: default
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress-controller
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: traefik
  labels:
    app: traefik
spec:
  replicas: 1
  selector:
    matchLabels:
      app: traefik
  template:
    metadata:
      labels:
        app: traefik
    spec:
      serviceAccountName: traefik-ingress-controller
      terminationGracePeriodSeconds: 10
      containers:
        - name: traefik
          image: traefik:v2.2
          args:
            - --api.insecure
            - --accesslog
            - --entrypoints.web.address=:80
            - --entrypoints.websecure.address=:443
            - --providers.kubernetesingress
            #- --providers.kubernetescrd TODO is this required to get the middleware running?
            - --certificatesresolvers.letsencrypt.acme.tlschallenge
            - --certificatesresolvers.letsencrypt.acme.email=webmaster@example.com
            - --certificatesresolvers.letsencrypt.acme.storage=acme.json
            # Please note that this is the staging Let's Encrypt server.
            # Once you get things working, you should remove that whole line altogether.
            - --certificatesresolvers.letsencrypt.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
          ports:
            - name: web
              hostPort: 80
              containerPort: 80
            - name: websecure
              hostPort: 443
              containerPort: 443
            - name: admin
              containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: traefik
spec:
  type: LoadBalancer
  selector:
    app: traefik
  ports:
    - name: web
      protocol: TCP
      port: 80
      targetPort: 80
    - name: websecure
      protocol: TCP
      port: 443
      targetPort: 443
    - name: admin
      protocol: TCP
      port: 8080
      targetPort: 8080
---
apiVersion: v1
kind: Secret
metadata:
  name: traefik-basic-auth-secret
data:
  users: <redacted>
---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: traefik-basic-auth
spec:
  basicAuth:
    realm: Traefik
    secret: traefik-basic-auth-secret
    removeHeader: true

I guess @kubernetescrd might be wrong. According to these docs it should be the provider namespace, but I'm not sure what is the correct value in my case.

2

I have a similar situation. I would like to keep a minimal configuration using an Ingress, but I need a Middleware (stripPrefix).

Using Ingress CRD works perfectly, but when I try middleware definition a plain Ingress, like:

apiVersion: apiextensions.k8s.io/v1beta1
metadata:
  name: middlewares.traefik.containo.us
spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: Middleware
    plural: middlewares
    singular: middleware
  scope: Namespaced
---
kind: Middleware
apiVersion: traefik.containo.us/v1alpha1
metadata:
  name: path-prefix
spec:
  stripPrefix:
    prefixes:
      - /play
---
kind: Ingress
apiVersion: networking.k8s.io/v1beta1
metadata:
  name: traefik-basic
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/router.entrypoints: web
    traefik.ingress.kubernetes.io/router.middlewares: path-prefix@kubernatescrd

There is no way to make it working. I get something similar in the log:

msg="middleware \"path-prefix@kubernatescrd\" does not exist" entryPointName=web

My container configuration is similar.

      containers:
      - name: traefik
        image: traefik:v2.2
        args:
          - --log.level=DEBUG
          - --api
          - --api.insecure
          - --entrypoints.web.address=:80
          - --entrypoints.web.http.middlewares=default-striper@kubernetescrd
          - --providers.kubernetesingress
         # - --providers.kubernetescrd ??

If you went through that, please let me know.

S.

3

I've stumbled across that one myself. But fortunately a look at the dashboard helped me. If you define the middleware via the annotations, you have to put the Kubernetes namespace in front of it. That looks like this

traefik.ingress.kubernetes.io/router.middlewares: ingress-traefik-secureheaders@kubernetescrd

My middleware is called "secureheaders" and the namespace is "ingress-traefik"

This should definitely be better documented, because it is very confusing. But at least there is the possibility to access middlewares across the namespace.

2 Likes
4

Thank you @p7k , your reply is very useful and I totally agree there is an "hole"in the documentation.

Could you clarify how can do you define other resources.
For example, if use "default" namespace, shouldn't my definition be

traefik.ingress.kubernetes.io/router.middlewares: default-path-prefix@kubernetescrd

(Don't bother of me keeping mispelling k8s, replacing randomly 'e' with 'a')

Moreover I do you configure configure your traefik CLI in the container?
Could you point me a complete working example snippet?

Because after I make few changes. Dashboard keep on showing:

ERRORS
middleware "default-path-prefix@kubernetescrd" does not exist

Thanks a lot

5

Yes, that's right. In your case, the middleware would be called: default-path-prefix@kubernetescrd

But to be honest. So far I could only access the middlewares via annotations in an ingress object. I also tried using the CLI arguments, because I wanted to configure a default middleware for my entry point.

- --entrypoints.traefik.Address=:8080
- --entrypoints.traefik.http.middlewares=ingress-traefik-traefik-api-auth@kubernetescrd

But in this case I got the same error message as you. I really think this is a bug in Traefik.

Someone else has the same problem here: 2.2 can't see Kubernetes defined entryPoint middlewares? - Traefik v2 (latest) - Traefik Labs Community Forum

So I really don't think that this function works properly yet.

6

Thanks @p7k .
It is not clear to me whether your solution is working for you or not.

Watching my configuration in my first answer, apart from the prefix / provider syntax in "dynamic middleware" configuration in the Ingress definition, how far I am from your solution?
How do you configure traefik in the container in the pod?

S.

7

I'm not quite sure what you mean. You are trying to define a middleware in a normal k8s ingress object (talk about the kubernetesingress provider). In this case only the kubernetes namespace prefix is missing.

If you want to define the middleware as default in an entrypoint (using the CLI arguments), I don't think this works yet. I haven't got it working yet either.

Example for an ingress object with kubernetesingress provider:

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: secureheaders
  namespace: ingress-traefik
spec:
  headers:
    stsSeconds: 15768000
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    traefik.ingress.kubernetes.io/router.middlewares: ingress-traefik-secureheaders@kubernetescrd
    name: <ingress-name>
    namespace: <namespace>
[...]

This example works for me.

2 Likes
8

I finally get it working!
Thanks a lot for your suggetions @p7k

1 Like
9

Thanks a lot @p7k
You save my live. 3 days frustrated on how make traefik ingress v2 to works as expected, from listening on host port 443 until stuck in this middleware things

What left now is how to define middleware in something centralized such as the cli parameter, and refer it from anywhere

10

Unfortunately this cannot be defined in the CLI. This is because there are "dynamic" and "static" configurations. Only static configurations can be adjusted using the CLI. But it is definitely possible to create "ready-made" middlewares in the Traefik namespace, which in turn can be accessed from anywhere (with the appropriate namespace prefix). This is what I do, for example, with a middleware for the HSTS header, which I don't have to redefine in every namespace.

11

May be I should begin a new thread. But at first try to ask here.

I have the similar problem with ingress and middleware.

My ingress looks like:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: gitea-prod-ext
  namespace: ci-tools
  annotations:
    traefik.ingress.kubernetes.io/router.middlewares: ci-tools-https-redirectscheme@kubernetescrd
    name: gitea-prod-ext
    namespace: ci-tools
    kubernetes.io/ingress.class: traefik
    cert-manager.io/cluster-issuer: letsencrypt-prod
    traefik.ingress.kubernetes.io/router.tls: "true"
  labels:
    traffic-type: external
spec:
  tls:
    - secretName: git-xxx-xx-tls
      hosts:
        - git.xxx.xx
  rules:
  - host: git.xxx.xx
    http:
      paths:
      - backend:
          serviceName: gitea-service
          servicePort: 3000
---
# Redirect to https
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: https-redirectscheme
spec:
  redirectScheme:
    scheme: https
    permanent: true

And https redirection in this ingress doesn't work.
curl http://git.xxx.xx - 404
curl https://git.xxx.xx - Ok

If I remove line "traefik.ingress.kubernetes.io/router.tls: "true"" then redirection starts working but I've got 404 in both curl requests.

12

I think we've had the same problem here before. Unfortunately I can't find the link to it. But the solution was probably to set the tls option not in the router (Ingress annotations), but globally in the corresponding entry point definition (EntryPoints | Traefik | v2.2). The tls option should only be set in one entry point, so that this is probably not possible via the annotations, at least not currently. But the global configuration works great here, for example, with a similar setup and a https redirect.

13

I've added option "--entrypoints.websecure.http.tls.options" to my traefik daemonset.
And all is working fine now.
Thank you very much!

1 Like
14

can you please tell me the how to replace this traefik.ingress.kubernetes.io/router.tls: "true"
I had to use this in all of my ingress files . how to use --entrypoints.websecure.http.tls.options can you brief me with example .what changes i had to do