[Bounty Offer] Need to get Traefik V2 working on DigitalOcean k8s

Traefik Traefik v2
,
1

I basically need to get my k8s cluster going, but I can't do that without Traefik functional first (technically I probably could without Traefik, but I'd very much prefer to use it). I would like to be able to try and mess around to figure it out myself, but I don't currently have the time. I'm sure this is probably a small issue or misconfiguration, but would really like some assistance with this and am willing to pay for it to expedite things.

Stack Overflow post with configs, etc

Feel free to make an offer for what you think it would be worth. I'm willing to transfer via PayPal or many of the popular cryptocurrencies. Please allow me time to accept your offer before providing a solution. (Basically, I won't be honoring an offer if I'm sent a fix and then demanded an exorbitant amount for it...)

Thank you!

2
  • Make sure that DO firewall is off, or create rules for HTTP, HTTPS and Custom 8080
  • Make sure DNS of your whoami.domain.tld is pointing to your node IP
  • Make sure this is a single node cluster. distributed acme.json is only supported in Enterprise Edition, it will not work well in the free edition of traefik.
  • Change example config below:
    • replace whoami.domain.tld with your domain
    • replace foo@you.com with your email
    • delete the staging LE url if required

Deploy this yaml kubectl apply -f example.yaml:

example.yaml 
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutes.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRoute
    plural: ingressroutes
    singular: ingressroute
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutetcps.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRouteTCP
    plural: ingressroutetcps
    singular: ingressroutetcp
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: middlewares.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: Middleware
    plural: middlewares
    singular: middleware
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: tlsoptions.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TLSOption
    plural: tlsoptions
    singular: tlsoption
  scope: Namespaced

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller

rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - traefik.containo.us
    resources:
      - middlewares
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - traefik.containo.us
    resources:
      - ingressroutes
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - traefik.containo.us
    resources:
      - ingressroutetcps
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - traefik.containo.us
    resources:
      - tlsoptions
    verbs:
      - get
      - list
      - watch

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller

roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
  - kind: ServiceAccount
    name: traefik-ingress-controller
    namespace: default


---
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: default
  name: traefik-ingress-controller

---
kind: Deployment
apiVersion: apps/v1
metadata:
  namespace: default
  name: traefik
  labels:
    app: traefik

spec:
  replicas: 1
  selector:
    matchLabels:
      app: traefik
  template:
    metadata:
      labels:
        app: traefik
    spec:
      serviceAccountName: traefik-ingress-controller
      dnsConfig:
        nameservers:
          - 173.245.58.51
          - 173.245.59.41
          - 198.41.222.173
      containers:
        - name: traefik
          image: traefik:v2.0.4
          args:
            - --api.insecure
            - --accesslog
            - --entrypoints.web.Address=:80
            - --entrypoints.websecure.Address=:443
            - --providers.kubernetescrd
            - --certificatesresolvers.default.acme.tlschallenge
            - --certificatesresolvers.default.acme.email=foo@you.com
            - --certificatesresolvers.default.acme.storage=acme.json
            # Please note that this is the staging Let's Encrypt server.
            # Once you get things working, you should remove that whole line altogether.
            - --certificatesresolvers.default.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
          ports:
          - name: http
            containerPort: 80
            hostPort: 80
          - name: https
            containerPort: 443
            hostPort: 443
          - name: admin
            containerPort: 8080
            hostPort: 8080
          securityContext:
            capabilities:
              drop:
              - ALL
              add:
              - NET_BIND_SERVICE
---
kind: Deployment
apiVersion: apps/v1
metadata:
  namespace: default
  name: whoami
  labels:
    app: whoami

spec:
  replicas: 2
  selector:
    matchLabels:
      app: whoami
  template:
    metadata:
      labels:
        app: whoami
    spec:
      containers:
        - name: whoami
          image: containous/whoami
          ports:
            - name: web
              containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: whoami

spec:
  ports:
    - protocol: TCP
      name: web
      port: 80
  selector:
    app: whoami

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: simpleingressroute
  namespace: default
spec:
  entryPoints:
    - web
  routes:
  - match: Host(`whoami.domain.tld`) && PathPrefix(`/notls`)
    kind: Rule
    services:
    - name: whoami
      port: 80

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: ingressroutetls
  namespace: default
spec:
  entryPoints:
    - websecure
  routes:
  - match: Host(`whoami.domain.tld`) && PathPrefix(`/tls`)
    kind: Rule
    services:
    - name: whoami
      port: 80
  tls:
    certResolver: default

Examine your logs. I prefer watching them with stern but you can simply use kubectl logs <traefik_pod_name> There could be a few errors on initial deploy, as kubernetes has timing issues when applying several objects. If you observe a bunch if "Cannon create service" messages just kill the pods: kubectl delete pod -l app=traefik and let the deployment re-created it.

You should be able to see:

Additional notes:

  • You should not leave dashboard exposed like this, either password protect it or create a service object in kubernetes for it and use kubectl port-forward to access it.
  • Note that PathPrefix which works well with whoami, will not work for many other appliations, if the application do not have those path prefixes. That is they are not stripped.
3

Thank you for all of that!

Things do work with that setup. I believe not having hostPort on the Traefik deployment was what I needed, but am a little fuzzy on that still.

Is there any way I can give you anything as a thank you?

The biggest things which don't make a whole lot of sense to me are that, when I use kubectl proxy to connect with pods internally, it seems like Traefik still shows the same behavior of responding to the page request, but the entire page is blank. (Using http://localhost:8001/api/v1/namespaces/default/services/http:traefik:8080/proxy/dashboard/#/)

And also, is there anywhere you'd recommend I can research hostPort, or maybe where you learned of it? The only place where I really find it is in the official Kubernetes docs saying how they recommend against using it due to scheduling restrictions it places on the pods.

Thank you again!
And seriously, I'd be happy to give you at least something for your help!

4

This is mostly designed to test API's not pages. Some web applications, traefik dashboard in it's current incarnation included, use absolute url for assets. Since modern SPA consist mostly of assets, if an asset cannot be fetched, blank page is what you get. To elaborate, if you application resides in /main.js then given url above your browser will try http://localhost:8001/main.js to get it, needless to say it's not going to be successful.

How kubernetes exposes applications it hosts to external world is an expansive topic. There are Load Balancer Services and Ingresses, and the way it down by different cloud providers also differ. This is because there is usually a desire to separate IP allocation concern from application container itself.

For example see this and this for Digital Ocean specific configuration.

It is true, that hostPorts are not recommended, but there is also common sense. For example, if you run traefik on your cluster as DaemonSet instead of Deployment, then scheduling restrictions is no longer a concern for you, so this could be a legitimate use case. In fact I did set it up for a customer like this before, when the customer wanted to use a hardware load balancer that they already had to front the cluster.

For your case, I do recommend looking into using the Digital Ocean Load Balancer instead.

For some details on hostPort I refer you to an excellent book by Marko Luksa "Kubernetes in Action". This is discussed in section 13.1.2 "Binding to a host port without using the host’s network

namespace", page 377. Of course it the best to read the whole book, since out of context, some details may not make sense.

If I convert my hourly rate to bitcoin, it comes to 0.01 BTC at today's rate. If you'd like you can use this address: 1BazLpXtmnpMxbPANiHF5eRtSXUKMd14mB. It took me about an hour, to provision a new cluster on DO and test and tune the configuration before posting it to you. Having said that, it is not necessary at all, and this is just in case you would like to do this.

1 Like
5

It looks like I've got a lot of research ahead of me, haha, but thank you very much again!
I do understand the kubectl proxy purpose now better and get why it wasn't really what I was trying to use it for.

I've got to be a little careful with how much I send since I don't have a 'real' job yet, but I definitely will send you something, especially since I'm fully up and running now with my cluster to replace a failing solution.