New to Traefik & K8 questions

Hi, I'm new to K8 and Traefik. It's supposedly supporting DevOps and it does seem like there is a lot of text to do quite simple things so far but it also basically switches manual configuration of NGINX for trusting and copy pasting yaml. However, I do have some questions:

  • should I switch to Traefik v2?
  • how do you issue Let's Encrypt certificates for the three cheeses K8 example?
  • from the Kubernetes website it seemed like Traefik is merely an Ingress for Kubernetes but looking at Traefik's website it isn't. It seems like Traefik is usually used with Docker directly, instead of a scheduler like K8. Is it?
  • given Traefik's role as Ingress for K8, if you would want to manage a node with numerous pods, for what would you use K8's API and for what Traefik's?

Hello,

This community forum is specifically dedicated to Traefik, so it is not really the place to teach you about the whole stack of technologies you're mentioning.
You'd be better off posting to generic forums such as Stack Overflow for this kind of question.
However, when you do have a more precise question about Traefik itself, please do come back and post here.

Regards,
Mathieu

Hello @mpl, I edited my questions to be more specific to Traefik.

Hi @tmpl, you might be interested in https://medium.com/@geraldcroes/kubernetes-traefik-101-when-simplicity-matters-957eeede2cf8 , to get started with Traefik and Kubernetes.
Also https://github.com/containous/traefik/wiki/Awesome-Traefik might help you to get resources :slight_smile: .

About your questions:

from the Kubernetes website it seemed like Traefik is merely an Ingress for Kubernetes but looking at Traefik's website it isn't. It seems like Traefik is usually used with Docker directly, instead of a scheduler like K8. Is it?

Traefik is a valid Kubernetes Ingress Controller, and I recommend you to install it in your Kubernetes cluster using the official Helm Chart, as mentioned in https://docs.traefik.io/v1.7/user-guide/kubernetes/#deploy-traefik-using-helm-chart.

  • given Traefik's role as Ingress for K8, if you would want to manage a node with numerous pods, for what would you use K8's API and for what Traefik's?

You might want to check https://medium.com/@geraldcroes/kubernetes-traefik-101-when-simplicity-matters-957eeede2cf8 to get a better understanding about the role of Traefik Ingress Controller.

TL;DR; In the context of Kubernetes, you must specify a Kubernetes "Ingress Rule" for your application. Traefik Ingress Controller watches Kubernetes API for Ingress rules, and dynamically update itself with these rules's configuration.

  • how do you issue Let's Encrypt certificates for the three cheeses K8 example?

If you followed my advise about using the Helm Chart for installing the Traefik Ingress Controller, then you can customize the installation through helm values, allowing to speicify the ACME/Let's Encrypt settings at installation time (chech the values acme.* in https://github.com/helm/charts/tree/master/stable/traefik#configuration).

Technically, enabling Let's Encrypt in Kubernetes with Traefik v1 requires to change the traefik.toml configuration and add it the right [acme] settings referenced in https://docs.traefik.io/v1.7/configuration/acme/. This traefik.toml is always stored as a Kubernetes ConfigMap, which you can always edit with kubectl edit configmap --namespace <namespace where you installed Traefik> <name of the configmap> if you already installed Traefik.

  • should I switch to Traefik v2?

I would recommend you to first get started on v1 in Kubernetes to get a better understanding of the Kubernetes ecosystem.

The v2 is in beta, and introduces breaking changes, especially by introducing an alternative to the Kubernetes Ingress Rule. Read more here: https://docs.traefik.io/v2.0/providers/kubernetes-crd/ if you are interested.

Thank you @dduportal. I will try this. I don't use the Helm chart because I use minikube and not a cloud provider that can provide an external load balancer...

I hope your answers answer the following questions as well, referring to that tutorial. Please take a look.

I really liked how simple it was to get the things done that did work, but the security features seem more difficult. There are many questions on this board about those. That's a little unnerving because it is the main reason to use Traefik with K8. But that's an aside.

From the tutorial

"Note
For this example to work you need a TLS entrypoint. You don't have to provide a TLS certificate at this point. For more details see here."

  • => What am I looking at "here"?
  • where is the TOML?

(The TOML is thus stored as a ConfigMap in the kube-system namespace, right? Can the arguments be applied before deploying the traefik ingress or pod?)

Then, Q2; is this an alternative? Or does this "need a TLS entrypoint"?/ Or do you still not need a certificate?:
"To setup an HTTPS-protected ingress, you can leverage the TLS feature of the ingress resource."

And Q3: basic authentication is for the dashboard right or are there more things that need access protection? TLS protects the routed hosts right?

"Basic Authentication

It's possible to protect access to Traefik through basic authentication."

I tried the TLS and authentication sections but they leave the traefik dashboard a 404. I can get the dashboard without this section. However, the main point and where I want to go is, to get Let's Encrypt certification for "the 3 cheeses" (my web domains).

I don't really know the difference between the TLS and Let's Encrypt certificates because before, without K8, I always got https with Let's Encrypt and https is TLS to me.

Can Traefik automatically get LE certificates and wire the new cheese if I merely add a new deployment?

After editing the cheeses example, everything looks like it works but the *.minikube URL's give 404. Can you see from this printout why?

$ kubectl get services --all-namespaces
NAMESPACE              NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                       AGE
default                cheddar                     ClusterIP   10.106.204.162   <none>        80/TCP                        8m32s
default                hugo                        ClusterIP   10.107.105.161   <none>        80/TCP                        8m32s
default                kubernetes                  ClusterIP   10.96.0.1        <none>        443/TCP                       107m
default                wensleydale                 ClusterIP   10.109.26.18     <none>        80/TCP                        8m32s
kube-system            default-http-backend        NodePort    10.99.66.176     <none>        80:30001/TCP                  107m
kube-system            kube-dns                    ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP,9153/TCP        107m
kube-system            kubernetes-dashboard        ClusterIP   10.107.27.194    <none>        80/TCP                        107m
kube-system            traefik-ingress-service     NodePort    10.109.100.42    <none>        80:31032/TCP,8080:31214/TCP   29m
kube-system            traefik-web-ui              ClusterIP   10.108.132.74    <none>        80/TCP                        23m
kubernetes-dashboard   dashboard-metrics-scraper   ClusterIP   10.106.176.28    <none>        8000/TCP                      40m
kubernetes-dashboard   kubernetes-dashboard        ClusterIP   10.99.100.7      <none>        443/TCP                       40m
$ kubectl get ingress --all-namespaces
NAMESPACE     NAME             HOSTS                                                 ADDRESS     PORTS   AGE
default       hugo-n-cheese    hugo.minikube,cheddar.minikube,wensleydale.minikube               80      7m41s
kube-system   traefik-web-ui   traefik-ui.minikube                                   10.0.2.15   80      24m
$ kubectl get deployments --all-namespaces
NAMESPACE              NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
default                cheddar                      1/1     1            1           17m
default                hugo-site-local-registry     1/1     1            1           14m
default                wensleydale                  1/1     1            1           17m
kube-system            coredns                      2/2     2            2           112m
kube-system            default-http-backend         1/1     1            1           111m
kube-system            kubernetes-dashboard         1/1     1            1           111m
kube-system            nginx-ingress-controller     1/1     1            1           111m
kube-system            traefik-ingress-controller   1/1     1            1           33m
kubernetes-dashboard   kubernetes-dashboard         1/1     1            1           45m
kubernetes-dashboard   kubernetes-metrics-scraper   1/1     1            1           45m

I come from this https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-18-04
you see.

First of all, I insist on the fact that you should really use the Helm chart: as you are in a learning phase of the Kubernetes world (not Traefik as all your questions are related to pure-kubernetes concepts),
starting with the Helm chart avoid you having to understand everything at first.

Nice for you to know:

I don't really get what do you mean? Traefik can run in a secure way without Kubernetes: wether you're using "Bare metal" with the binary, a Docker container on a single Docker Engine, or in Docker Swarm, or whatever other runtime system supporting Traefik, you can always enable both TLS and authentication. Do you have a specific requirements to explain here?

I'm asking this, because having to learn Kubernetes "only" for securing Traefik seems a bit overkill, so maybe I'm missing some context :slight_smile:

"here" is a link pointing to https://docs.traefik.io/v1.7/configuration/entrypoints. I feel like this could be rephrased to make it more explicit. If you share this feeling, do not hesitate to contribute to this documentation as it is an open project, so as much yours as ours :slight_smile: It's easy: go to the top of the page, and click on the "blue pen" icon, on the right of the title: it will redirect you to a Github web editor so you can edit the Markdown source and propose a change to the project.

I think the link might be better to point directly to the "TLS" section of the page "Entrypoints" (e.g. to https://docs.traefik.io/v1.7/configuration/entrypoints/#tls).

You are right, there is no mention of "how/where" the TOML file should be. Another improvement point for this documentation: feel free to contribute as well, but it's dully noted :slight_smile:

Your understanding is correct: you could define a ConfigMap containing the traefik.toml and mount it in Traefik's pod. Please read carefully https://docs.traefik.io/v1.7/basics/#configuration-file and https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/ before doing so.
=> My recurrent reminder about using Traefik's Helm Chart, as it takes care of this "plumbing" for you to avoid worrying about the details here.

My answers for these 2 questions:

  • An "entrypoint" (or a "TLS entrypoint") is a pure Traefik v1.x feature. You have to define 2 entrypoints for Traefik, aka. telling Traefik to listen on the ports 80 and 443. Defining an "entrypoint" as "TLS" is telling Traefik to encrypt the HTTP communication with TLS (aka. "HTTPS") for any request incoming on this entrypoint. It is a configuration item that you shall have to specify either on the traefik.toml, either with a command line flag. You can see the reference for TOML configuration for this topic on this page: https://docs.traefik.io/v1.7/configuration/entrypoints/#tls.

  • With an entrypoint configured to do HTTPS (e.g. making it a "TLS entrypoint"), Traefik needs a TLS certificates for encrypting the connection. Traefik can be configured to get a collection of TLS certificates (from static files, from Let's Encrypt, etc.), and it also generates a self-signed certificate at startup, under the hood, to use as a "default" certificate.

    • When Traefik receives an incoming request on the "TLS entrypoint", the hostname of the request (precisely the host on the TLS handshake) is used by Traefik to determine which certificate to use, independently of the provenance of certificates. If no certificate's hostname matches the request, then Traefik uses the "default one", which is the "generated and self signed" by default.
  • The "TLS feature of ingress resource" is a pure Kubernetes features, that Traefik can use for retrieving TLS certificates, as an alternative to Let's Encrypt or files mounted on the pod: it is easier to manage in Kubernetes clusters. But you can use Let's Encrypt alternatively.

=> My recurrent reminder about using Traefik's Helm Chart, as it takes care all the https ("http+tls") stuff for you ,based on the installation values. It's way easier.
[/quote]

Authentication and TLS (HTTPS) are totally different things: TLS/HTTPS is about encrypting the exchanges between client and server, while "authentication" is the process of proving your identity as a client, to the server (and eventually add authorization based on this authentication).

In Traefik, Authentication can be defined globally for and endpoints, likewise TLS, as described in https://docs.traefik.io/v1.7/configuration/entrypoints/#authentication. This is what to do for the "dashboard", as the entrypoint for the dashboard (("named traefik on port 8080 by default) is also exposing Traefik's API

But you can also enable authentication per "service" (example: http://example.com should be wide open to the world, but http://example.com/admin and http://admin.example.com ask for user/password).

How enable authentication on a specific Kubernetes Ingress rule is described here: https://docs.traefik.io/v1.7/configuration/backends/kubernetes/#authentication.

To help you on this matter, you need to provide us more informations:

  • Traefik's deployment description (what are the CLI flags used? what are the *.toml file used, what is the version, how was it deployed?)
  • Runtime information (which Kubernetes distribution, which version, which kubernetes objects, etc.)
  • Logs (Traefik logs are "production ready by default", you might want to enable debug logging).

By the way @tmpl, if you are seeking for professional support, you might want to check https://containo.us/traefikee/, as the forum is community based, as Traefik is an Open Source project.

Thank you, I seem to be wasting your time with not using the Helm chart, that however has a [prerequisite](github helm/charts/tree/master/stable/traefik): "You are deploying the chart to a cluster with a cloud provider capable of provisioning an external load balancer (e.g. AWS or GKE)"

I started with minikube delete and then minikube start several times and I do no configuration of Traefik outside the traefik tutorial.

minikube version: v1.2.0

I do have more pods by default than the traefik tutorial though:


kube-system   coredns-5c98db65d4-svgbq                      1/1     Running   0          38m
kube-system   coredns-5c98db65d4-txr59                      1/1     Running   0          38m
kube-system   default-http-backend-59f7ff8999-h64sp         1/1     Running   0          38m
kube-system   etcd-minikube                                 1/1     Running   0          37m
kube-system   kube-addon-manager-minikube                   1/1     Running   0          37m
kube-system   kube-apiserver-minikube                       1/1     Running   0          36m
kube-system   kube-controller-manager-minikube              1/1     Running   0          37m
kube-system   kube-proxy-sp6rs                              1/1     Running   0          38m
kube-system   kube-scheduler-minikube                       1/1     Running   0          37m
kube-system   kubernetes-dashboard-7b8ddcb5d6-44rnr         1/1     Running   0          38m
kube-system   nginx-ingress-controller-7b465d9cf8-ng5lb     1/1     Running   0          38m
kube-system   storage-provisioner                           1/1     Running   0          38m
kube-system   traefik-ingress-controller-668df9b887-nz4fk   1/1     Running   0          32m

I want to combine https://www.linode.com/docs/applications/containers/deploy-container-image-to-kubernetes/#configure-your-kubernetes-cluster with https://docs.traefik.io/user-guide/kubernetes/ .

Since the combination doesn't seem to work, I am looking at the differences (each works separately).

  • The hugo example goes without an Ingress. (I think that the EXTERNAL-IP PORT(S) 80:30304/TCP is not available when the minikube Ingress addon is enabled?)
  • The hugo service uses
spec:
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80

Its deployment uses apiVersion: apps/v1 and different labels.

I use deployment.yaml:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: hugo-site
spec:
  replicas: 2
  selector:
    matchLabels:
      app: hugo-site
  template:
    metadata:
      labels:
        app: hugo-site
    spec:
      containers:
      - name: hugo-site
        image: qkuykuy/hugo-site:v1
        imagePullPolicy: Always
        ports:
        - containerPort: 80

and service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: hugo-site
spec:
  selector:
    app: hugo-site
  ports:
  - name: http
    targetPort: 80
    port: 80

and ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: hugo-site
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  - host: hugo-site.minikube
    http:
      paths:
      - path: /
        backend:
          serviceName: hugo-site
          servicePort: http

It seems like some small thing makes the two definitions non-compatible.

$ kubectl get svc --all-namespaces

NAMESPACE     NAME                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                       AGE
default       hugo-site                 ClusterIP   10.109.198.22   <none>        80/TCP                        37m
default       kubernetes                ClusterIP   10.96.0.1       <none>        443/TCP                       54m
kube-system   default-http-backend      NodePort    10.107.28.13    <none>        80:30001/TCP                  53m
kube-system   kube-dns                  ClusterIP   10.96.0.10      <none>        53/UDP,53/TCP,9153/TCP        54m
kube-system   kubernetes-dashboard      ClusterIP   10.105.209.85   <none>        80/TCP                        53m
kube-system   traefik-ingress-service   NodePort    10.109.138.93   <none>        80:30380/TCP,8080:31655/TCP   48m
kube-system   traefik-web-ui            ClusterIP   10.102.112.72   <none>        80/TCP                        45m

logs from http://127.0.0.1:49486/api/v1/namespaces/kube-system/services/http:kubernetes-dashboard:/proxy/#!/pod/kube-system/traefik-ingress-controller-668df9b887-nz4fk?namespace=kube-system

time="2019-07-29T11:46:31Z" level=info msg="Traefik version v1.7.12 built on 2019-05-29_07:35:02PM"

time="2019-07-29T11:46:31Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.traefik.io/basics/#collected-data\n"

time="2019-07-29T11:46:31Z" level=info msg="Preparing server http &{Address::80 TLS:<nil> Redirect:<nil> Auth:<nil> WhitelistSourceRange:[] WhiteList:<nil> Compress:false ProxyProtocol:<nil> ForwardedHeaders:0xc000656d20} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s"

time="2019-07-29T11:46:31Z" level=info msg="Preparing server traefik &{Address::8080 TLS:<nil> Redirect:<nil> Auth:<nil> WhitelistSourceRange:[] WhiteList:<nil> Compress:false ProxyProtocol:<nil> ForwardedHeaders:0xc000656d40} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s"

time="2019-07-29T11:46:31Z" level=info msg="Starting provider configuration.ProviderAggregator {}"

time="2019-07-29T11:46:31Z" level=info msg="Starting server on :80"

time="2019-07-29T11:46:31Z" level=info msg="Starting server on :8080"

time="2019-07-29T11:46:31Z" level=info msg="Starting provider *kubernetes.Provider {\"Watch\":true,\"Filename\":\"\",\"Constraints\":[],\"Trace\":false,\"TemplateVersion\":0,\"DebugLogGeneratedTemplate\":false,\"Endpoint\":\"\",\"Token\":\"\",\"CertAuthFilePath\":\"\",\"DisablePassHostHeaders\":false,\"EnablePassTLSCert\":false,\"Namespaces\":null,\"LabelSelector\":\"\",\"IngressClass\":\"\",\"IngressEndpoint\":null}"

time="2019-07-29T11:46:31Z" level=info msg="ingress label selector is: \"\""

time="2019-07-29T11:46:31Z" level=info msg="Creating in-cluster Provider client"

time="2019-07-29T11:46:31Z" level=info msg="Server configuration reloaded on :80"

time="2019-07-29T11:46:31Z" level=info msg="Server configuration reloaded on :8080"

Hi @tmpl thanks for this details. Based on this screenshot, everything is good from Traefik point's of view: I see that Traefik correctly autoconfigured itself from the ingress rules: I see a couple of frontend and backends for Traefik's UI itself, which are correctly linked (as you can see the dashboard on your webbrowser), and a couple for the Hugo application you defined, which looks well: 2 servers for the 2 replicas pods, and frontend linked to the backend.

Your next step is to configure your DNS or /etc/hosts to have the domain hugo-site.minikube to the IP of the NodePort service. It looks like you already did it with traefik-ui.minikube as you can see the dashboard.

So what is the result of accessing http://hugo-site.minikube for you?

I did edit hosts, but get default backend - 404

What is the result of the following command:

curl -v http://hugo-site.minikube
echo "=="
curl -v http://traefik-ui.minikube

?

I'm thinking, can I log in to one of the containers if there is an NGINX running? They don't seem to have /bin/ash though (nor ls).

$ curl -v http://hugo-site.minikube echo "==" curl -v http://traefik-ui.minikube

* Rebuilt URL to: http://hugo-site.minikube/
* Trying 192.168.99.120...
* TCP_NODELAY set
* Connected to hugo-site.minikube (192.168.99.120) port 80 (#0)
> GET / HTTP/1.1
> Host: hugo-site.minikube
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Server: nginx/1.15.9
< Date: Mon, 29 Jul 2019 13:03:09 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 21
< Connection: keep-alive
<
* Connection #0 to host hugo-site.minikube left intact
default backend - 404* Rebuilt URL to: echo/
* Could not resolve host: echo
* Closing connection 1
curl: (6) Could not resolve host: echo
* Rebuilt URL to: ==/
* Could not resolve host: ==
* Closing connection 2
curl: (6) Could not resolve host: ==
* Rebuilt URL to: curl/
* Could not resolve host: curl
* Closing connection 3
curl: (6) Could not resolve host: curl
* Rebuilt URL to: http://traefik-ui.minikube/
* Trying 192.168.99.120...
* TCP_NODELAY set
* Connected to traefik-ui.minikube (192.168.99.120) port 80 (#4)
> GET / HTTP/1.1
> Host: traefik-ui.minikube
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 302 Found
< Server: nginx/1.15.9
< Date: Mon, 29 Jul 2019 13:03:09 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 34
< Connection: keep-alive
< Location: /dashboard/
<
<a href="/dashboard/">Found</a>.
* Connection #4 to host traefik-ui.minikube left intact

I find a log message in the K8 NGINX ingress log:
AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15" 394 0.001 [kube-system-traefik-web-ui-web] 172.17.0.9:8080 432 0.000 200 fb06b048b29e48ac3d361c3a9c66e003
I0729 11:58:50.972573 6 store.go:348] ignoring add for ingress hugo-site based on annotation kubernetes.io/ingress.class with value

but it may be me doing minikube apply -f more than once

actually, I did create -f the first time and when I indeed do $ kubectl apply -f ingress.yaml:

Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
ingress.extensions/hugo-site configured

-> Given that the nginx version of the hugo pod is 1.16.0, and that the "HTTP/404 error" you have come from a Nginx server in version 1.15.9, then it means that:

  • Your configuration for the deploymeny of Hugo is not culprit
  • Traefik is not used for the requests from your host.

It's because the DNS in *.minikube are pointing to the external IP of the minikube VM. So the request you are trying are incoming on the port 80 of the interface eth1 of minikube's VM, which are mapped to the port 30001, associated to the service default-http-backend of type NodePort. This is not what you want obviously :slight_smile:

=> Start by deleting the "default nginx controller" of minikube. It can work with it still installed, but let's simplify your environment for the sake of a better learning path.
=> Then, can you try the following request:

curl -v http://hugo-site.minikube: 30380

please?

As you installed Traefik as a service of type "NodePort", then Kubernetes allocated a port on the minikube's VM network interfaces (eth0, eth1 and l0 at least). This port is the public port to be used to reach the service. In the case of your Traefik installation, I see on the output of kubectl get svc --all-namespaces that you gave us earlier, that the allocated external port is 30380.

Let us know if it works with this port.

This should not be a problem as Kubernetes objects are idempotent: you did nothing wrong here.

Should I disable the ingress addon or delete the NGINX pod?
The https://www.linode.com/docs/applications/containers/deploy-container-image-to-kubernetes/#create-a-docker-image Dockerfile also includes NGINX.