Dashboard is not working and unable to access application using https

Manmohan · June 1, 2021, 7:29am

Hi All,

I have deployed traefik ingress controller v2 along with toml file. I am not able to access dashboard unless I mention:

[api]
  insecure = true

Also, I am not able to access my application in https but in http, it's working well. I am getting 404 Not found.

Here is my traefik configuration and toml file:

traefik.yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    k8s-app: traefik-ingress-lb
  name: traefik-ingress-controller
  namespace: public
spec:
  selector:
    matchLabels:
      k8s-app: traefik-ingress-lb
  revisionHistoryLimit: 10
  template:
    metadata:
      creationTimestamp: null
      labels:
        k8s-app: traefik-ingress-lb
        name: traefik-ingress-lb
    spec:
      hostNetwork: true
      containers:
      - args:
        - --configfile=/etc/traefik/traefik.toml
        - --global.checknewversion
        - --global.sendanonymoususage
        - --api.dashboard=true
        - --providers.kubernetescrd
        - --providers.kubernetesingress
        image: traefik:2.4.8
        imagePullPolicy: IfNotPresent
        name: traefik
        ports:
        - containerPort: 9000
          name: traefik
          protocol: TCP
        - containerPort: 80
          name: web
          protocol: TCP
        - containerPort: 443
          name: websecure
          protocol: TCP
        resources:
          requests:
            cpu:  1
            memory: 1G
          limits:
            cpu: 1
            memory: 1G
        env:
         - name: HOST_IP
           valueFrom:
             fieldRef:
               fieldPath: status.hostIP
        securityContext:
          privileged: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - name: traefik-tmpl
          mountPath: /etc/traefik
          readOnly: true
        - mountPath: /ssl
          name: ssl
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/arch: amd64
        role: "mcdr_public"
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 65532
      serviceAccount: traefik-ingress-controller
      serviceAccountName: traefik-ingress-controller
      tolerations:
      - effect: NoSchedule
        key: node-role-kubernetes-io/public
        operator: Exists
      - key: "CriticalAddonsOnly"
        operator: "Exists"
      terminationGracePeriodSeconds: 60
      volumes:
      - name: ssl
        secret:
          secretName: traefik-wildcard-cert
      - name: traefik-tmpl
        configMap:
          name: traefik-tmpl
          items:
          - key: traefik.toml
            path: traefik.toml

======

traefik.toml

data:
traefik.toml: |

[serversTransport]
  insecureSkipVerify = true
  maxIdleConnsPerHost = 0

[entryPoints]
  [entryPoints.web]
    address = ":80"
  [entryPoints.websecure]
    address = ":443"
  [entryPoints.traefik]
    address = ":9000"

[[tls.certificates]]
  CertFile = "/ssl/tls.crt"
  KeyFile = "/ssl/tls.key"

[tls.options]
  [tls.options.default]
     minVersion = "VersionTLS12"
     cipherSuites = [
       "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
       "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
       "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305",
       "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305",
       "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
       "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
     ]

[providers]
  providersThrottleDuration = "2s"
  [providers.kubernetesIngress]
    throttleDuration = "0s"

[api]
  dashboard = true

[log]  
  level = "DEBUG"
  filePath = "/var/log/traefik.log"
  format = "json"

[accessLog]
  format = "json"
  bufferingSize = 0
  [accessLog.fields]
    defaultMode = "drop|keep"
    [accessLog.fields.headers]
       defaultMode = "drop|keep|redact"
       [accessLog.fields.headers.names]
         Authorization = "drop"
         Compressedjwt = "drop"
         Content-Type = "keep"
         Cookie = "drop"
         Jwt = "drop"
         User-Agent = "redact"
         request_Apijwt = "drop"

jakubhajek · June 1, 2021, 8:04am

Hello @Manmohan

You need to add routing for the dashboard, here is the example with Ingressroute CRD:

github.com

traefik-workshops/traefik-workshop/blob/master/exercise-14/ingresscrd-dashboard.yaml

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: dashboard
spec:
  entryPoints:
    - websecure
  routes:
    - kind: Rule
      match: Host(`d.<FIXME>.demo.traefiklabs.tech`) && PathPrefix(`/dashboard`) || PathPrefix(`/api`)
      services:
        - kind: TraefikService
          name: api@internal
      middlewares:
        - name: default-test-auth@kubernetescrd
  tls:
    certResolver: le

Additionally, you have to also create middleware with basic authentication to protect the dashboard, you need also to issue certificates using built-in Lets Encrypt and add TLS termination on the router.

All of the steps that I mentioned are explained in the Github repo I shared. There is also a recording available that will walk you through those examples.

Here is the link to the recording:

Manmohan · June 1, 2021, 8:16am

Thanks for responding back jakub. My earlier configuration with v1 was something like this:

    [api]
      dashboard = true
    [entryPoints]
      [entryPoints.http]
      address = ":80"
      #compress = true
      [entryPoints.https]
      address = ":443"
      #compress = true
      [entryPoints.https.tls]
        minVersion = "VersionTLS12"
        cipherSuites = [
          "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
          "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
          "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305",
          "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305",
          "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
          "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
        ]
        [[entryPoints.https.tls.certificates]]
        CertFile = "/ssl/tls.crt"
        KeyFile = "/ssl/tls.key"

With the above configuration, I was able to access dashboard. So, please correct me for my understanding in regards to v2 is right:

there is an addition of middleware and routing in v2 in order to access dashboard?
Additionally, in order to access application in https, I need to configure router in v2? Is that the only way?

Manmohan · June 1, 2021, 8:34am

Basically, I was wondering, how can I migrate following v1 code to v2:

    [api]
      dashboard = true
    [entryPoints]
      [entryPoints.http]
      address = ":80"
      [entryPoints.https]
      address = ":443"
      [entryPoints.https.tls]
        minVersion = "VersionTLS12"
        cipherSuites = [
          "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
          "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
          "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305",
          "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305",
          "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
          "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
        ]
        [[entryPoints.https.tls.certificates]]
        CertFile = "/ssl/tls.crt"
        KeyFile = "/ssl/tls.key"```

jakubhajek · June 1, 2021, 8:55am

I kindly invite you to have a look at the link that I shared in my previous post. You will find examples of how to achieve the configuration you are looking for and smoothly strat with Traefik v2 on Kubernetes.

In order to access the dashboard, the appropriate routing has to be created, this is also covered in the workshop I shared with you.

There is also an example with creating TLSOption with extra TLS configuration:

github.com

jakubhajek/traefik-workshop/blob/4dc778d4bca02cde46eb4905c1eef88891e5503a/exercise-12/ingresscrd-websecure.yaml#L26

    
      
                  - name: default-security@kubernetescrd
            tls:
              certResolver: le
              domains:
                - main: crd.<FIXME>.demo.traefiklabs.tech
                  sans:
                    - crda.<FIXME>.demo.traefiklabs.tech
              options:
                name: tlsoptions
          
          
---
          apiVersion: traefik.containo.us/v1alpha1
          kind: TLSOption
          metadata:
            name: tlsoptions
          spec:
            minVersion: VersionTLS12
            cipherSuites:
              - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
              - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
              - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

I hope that helps.

Manmohan · June 1, 2021, 9:32am

Yeah, I have gone through your video. It's a nice video with lot of information but unfortunately, I don't see how can I migrate my v1 code to v2 i.e. this code

[api]
      dashboard = true
    [entryPoints]
      [entryPoints.http]
      address = ":80"
      [entryPoints.https]
      address = ":443"
      [entryPoints.https.tls]
        minVersion = "VersionTLS12"
        cipherSuites = [
          "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
          "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
          "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305",
          "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305",
          "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
          "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
        ]
        [[entryPoints.https.tls.certificates]]
        CertFile = "/ssl/tls.crt"
        KeyFile = "/ssl/tls.key"

It is just picking up default traefik certificate and not the one that I provided, if I use below configuration

[entryPoints]
  [entryPoints.web]
    address = ":80"
  [entryPoints.websecure]
    address = ":443"
  [entryPoints.traefik]
    address = ":9000"

[[tls.certificates]]
  CertFile = "/ssl/tls.crt"
  KeyFile = "/ssl/tls.key"

[tls.options]
  [tls.options.default]
     minVersion = "VersionTLS12"
     cipherSuites = [
       "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
       "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
       "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305",
       "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305",
       "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
       "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
     ]

Manmohan · June 1, 2021, 10:21am

@jakubhajek Basically, could you please let me know how can I make sure, it picks my provided certificate and not pick the default certificate.
default certificates

I have my certificates in below path.

[[tls.certificates]]
  CertFile = "/ssl/tls.crt"
  KeyFile = "/ssl/tls.key"

If you see the attached, it is picking up default certificates. This is my current v2 configuration

[entryPoints]
  [entryPoints.web]
    address = ":80"
  [entryPoints.websecure]
    address = ":443"
  [entryPoints.traefik]
    address = ":9000"

[[tls.certificates]]
  CertFile = "/ssl/tls.crt"
  KeyFile = "/ssl/tls.key"

[tls.options]
  [tls.options.default]
     minVersion = "VersionTLS12"
     cipherSuites = [
       "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
       "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
       "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305",
       "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305",
       "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
       "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
     ]

Manmohan · June 1, 2021, 11:21am

@ldez Hi, I saw your comments in Trouble understanding how traefik picks up a custom SSL certificate for a domain

Do you have any suggestions on what possibly I am doing wrong in configuring the custom certificates in kubernetes.

traefiker1 · June 1, 2021, 12:00pm

I think it's best to use Let's Encrypt in Kubernetes. I have not used my own certificate in k8s.

jakubhajek · June 1, 2021, 12:07pm

@Manmohan,
Traefik will present the default certificate only in the case when there is no available certificate for your domain. This is expected behaviour.

If you use your own certificates and don't use Lets Encrypt I suggest you to create Kubernetes secret with
your certificate and than assigned that secret to your Ingressroute configuration in TLS section.

I hope that helps.

Thank you, Jakub

Manmohan · June 1, 2021, 12:54pm

@jakubhajek, I tried configuring the ingressroutes with tls, it's not working. I am sure, I am doing something wrong, which I will try to dig but my question is as below:

We have multiple customers using ingress in traefik v1 and I believe traefik v2 is compatible to handle "kind: Ingress" as well. So, it will be difficult to ask every customer to move immediately to ingressroute with tls. Hence, there should be something within the toml configuration that I can change, for it to work with v2.

For example:

My simple ingress rule is working with my traefik v1 configuration:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ng-ingress
spec:
  rules:
  - host: ng.example.com
    http:
      paths:
      - backend:
          serviceName: ng
          servicePort: 80

my traefik v1 configuration is something like

[api]
      dashboard = true
    [entryPoints]
      [entryPoints.http]
      address = ":80"
      [entryPoints.https]
      address = ":443"
      [entryPoints.https.tls]
        minVersion = "VersionTLS12"
        cipherSuites = [
          "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
          "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
          "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305",
          "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305",
          "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
          "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
        ]
        [[entryPoints.https.tls.certificates]]
        CertFile = "/ssl/tls.crt"
        KeyFile = "/ssl/tls.key"

My setup is something like this.. I have a secret with wildcard certificates.. which is being referenced in my ingress controller daemonset and same has been mentioned in the toml file.. So, when I ssh into the ingress controller pod, I am able to see certificates at following paths:

/ssl/tls.crt
/ssl/tls.key

Hence, the application deployed with ingress are able to get that certificate.

According to the below URL, I have configured my toml file but it is still picking up the default certificate.
https://doc.traefik.io/traefik/v2.0/https/tls/#user-defined

This is how I updated the toml file.

[[tls.certificates]]
  certFile = "/ssl/tls.crt"
  keyFile = "/ssl/tls.key"

So, I am looking for a solution within the toml itself, like I have configured in v1. Sorry for the confusion but I hope, I am able to explain my situation.

@traefiker1 I wish but we have wildcard certificate that we have been using in traefik v1 and so, will have to continue that.

Looking forward to hear from you. Please let me know if you have any questions.

jakubhajek · June 1, 2021, 1:14pm

Is this the configuration that you are looking for :

It allow you to create the default TLS configuration that will be applied to all routers. If you create your own configuration on Ingress level it will have precedence and the default won't be applied.

Manmohan · June 1, 2021, 2:57pm

I don't think so. This is more related to Let's Encrypt.

https://doc.traefik.io/traefik/migration/v1-to-v2/#tls-configuration-is-now-dynamic-per-router

If you check out the above link, this is the exact configuration of my traefik v1 and everything is working well with that but then while configuring it for v2, TLS doesn't work.

Only thing I changed is, I have not configured any router, considering it will accept all kinds of requests and so, my configuration looks like below:

    [[tls.certificates]]
       CertFile = "/ssl/tls.crt"
       KeyFile = "/ssl/tls.key"

    [tls.options]
      [tls.options.default]
        minVersion = "VersionTLS12"
        cipherSuites = [
          "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
          "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
          "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305",
          "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305",
          "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
          "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
        ]

and Ingress is working but https is not detecting the certificate and taking the default traefik certificate.

Manmohan · June 8, 2021, 10:44am

Issue got fixed once, I separated ssl from traefik.toml.. Basically, I included file provider to call ssl.toml from traefik.toml.

#traefik.toml

    [providers]
      providersThrottleDuration = "2s"
      [providers.kubernetesIngress]
        throttleDuration = "0s"
      [providers.file]
        filename = "/ssl-toml/ssl.toml"

    [api]
      dashboard = true

#ssl.toml

kind: ConfigMap
apiVersion: v1
metadata:
  name: traefik-ssl
data:
  ssl.toml: |
    [[tls.certificates]]
      certFile = "/ssl/tls.crt"
      keyFile = "/ssl/tls.key"
      stores = ["default"]

    [tls.stores]
      [tls.stores.default]
        [tls.stores.default.defaultCertificate]
          certFile = "/ssl/tls.crt"
          keyFile = "/ssl/tls.key"

system · June 11, 2021, 10:45am

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Traefik v2 Dashboard 404 - Without insecure=true, dashboard is not working Traefik v2 dashboard-api	5	13574	January 4, 2023
Can't Access Dashboard on HTTPS Traefik v2 dashboard-api	5	1317	September 29, 2022
404 Not Found on Traefik Dashboard via http on local Kubernetes Cluster Traefik v2 kubernetes-ingress	2	2071	November 23, 2021
404 on Dashboard but endpoints work Traefik v2 dashboard-api	0	775	June 1, 2020
Unable use kubernetes Ingress to visit dashboard Traefik v2 kubernetes-ingress , dashboard-api	0	454	August 26, 2022

Dashboard is not working and unable to access application using https

#traefik.toml

#ssl.toml

Related topics