Basic helm chart cli usage and ingress example with let's encrypt

I'm coming from Traefik v1 and trying to get Traefik v2 up and running via Helm. I am running via the official chart and the following values.yaml:

logs:
  general:
    level: INFO
  access:
    enabled: true

additionalArguments:
  - "--log.level=INFO"
  - "--global.checknewversion=false"
  - "--global.sendanonymoususage=false"
  - "--entrypoints.websecure.http.tls.domains[0].main=example.com"
  - "--entrypoints.websecure.http.tls.domains[0].sans=*.example.com"
  - "--providers.kubernetesingress=true"
  - "--certificatesresolvers.le.acme.storage=/data/acme.json"
  - "--certificatesresolvers.le.acme.email=bighead@hooli.xyz"

ports:
  web:
    redirectTo: websecure

persistence:
  enabled: true
  path: /data
  size: 1Gi

resources:
  requests:
    cpu: 100m
    memory: 32Mi
  limits:
    cpu: 1

Then the following example Kubernetes ingress definition (note setting ingressClassName: traefik):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/tls-acme: "true"
    meta.helm.sh/release-name: wg-easy
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2022-02-03T05:08:32Z"
  generation: 1
  labels:
    app.kubernetes.io/instance: wg-easy
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: wg-easy
    app.kubernetes.io/version: "5"
    helm.sh/chart: wg-easy-0.1.14
  name: wg-easy
  namespace: default
  resourceVersion: "11374475"
  uid: df6cd791-bbc4-4baf-b56d-9d314e11b411
spec:
  ingressClassName: traefik
  rules:
  - host: vpn.hooli.xyz
    http:
      paths:
      - backend:
          service:
            name: wg-easy-web
            port:
              number: 51821
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - vpn.hooli.xyz
status:
  loadBalancer: {}

But nothing in the logs about attempting to retrieve a let's encrypt certificate and the web portal U/I for traefik is not even showing the service bound to http.

Finally, here are the logs from Traefik:

pi@kube-master:helm/charts/traefik $ kubectl logs traefik-7d7d5649b4-hnnxc
time="2022-02-03T05:15:15Z" level=info msg="Configuration loaded from flags."
time="2022-02-03T05:15:15Z" level=info msg="Traefik version 2.6.0 built on 2022-01-24T17:08:39Z"
time="2022-02-03T05:15:15Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\n"
time="2022-02-03T05:15:15Z" level=info msg="Starting provider aggregator.ProviderAggregator"
time="2022-02-03T05:15:15Z" level=info msg="Starting provider *traefik.Provider"
time="2022-02-03T05:15:15Z" level=info msg="Starting provider *acme.ChallengeTLSALPN"
time="2022-02-03T05:15:15Z" level=info msg="Starting provider *ingress.Provider"
time="2022-02-03T05:15:15Z" level=info msg="ingress label selector is: \"\"" providerName=kubernetes
time="2022-02-03T05:15:15Z" level=info msg="Creating in-cluster Provider client" providerName=kubernetes
time="2022-02-03T05:15:15Z" level=info msg="Starting provider *acme.Provider"
time="2022-02-03T05:15:15Z" level=info msg="Testing certificate renew..." ACME CA="https://acme-v02.api.letsencrypt.org/directory" providerName=le.acme
time="2022-02-03T05:15:15Z" level=info msg="Starting provider *crd.Provider"
time="2022-02-03T05:15:15Z" level=info msg="label selector is: \"\"" providerName=kubernetescrd
time="2022-02-03T05:15:15Z" level=info msg="Creating in-cluster Provider client" providerName=kubernetescrd
10.244.2.1 - - [03/Feb/2022:05:15:33 +0000] "GET /ping HTTP/1.1" 200 2 "-" "-" 1 "ping@internal" "-" 0ms
10.244.2.1 - - [03/Feb/2022:05:15:33 +0000] "GET /ping HTTP/1.1" 200 2 "-" "-" 2 "ping@internal" "-" 0ms
10.244.2.1 - - [03/Feb/2022:05:15:43 +0000] "GET /ping HTTP/1.1" 200 2 "-" "-" 3 "ping@internal" "-" 0ms
10.244.2.1 - - [03/Feb/2022:05:15:43 +0000] "GET /ping HTTP/1.1" 200 2 "-" "-" 4 "ping@internal" "-" 0ms
10.244.2.1 - - [03/Feb/2022:05:15:53 +0000] "GET /ping HTTP/1.1" 200 2 "-" "-" 5 "ping@internal" "-" 0ms
10.244.2.1 - - [03/Feb/2022:05:15:53 +0000] "GET /ping HTTP/1.1" 200 2 "-" "-" 6 "ping@internal" "-" 0ms

Bump, any ideas? I am completely stumped. Unfortunately the official example does not include let's encrypt and uses IngressRoute CRD, where I am trying to use ingress directly.

Well you either haven't posted all your config or you are missing key item like your resolver config. My strong suggestions is don't mix treaefik.yml, commandline and envvars for you static config - that's not supported and by that i mean its documented it wont work - as soon as you specify anything in traefik yml all command line options appear to be ignored. (i hit same issue, took me an a couple hours of head banging to solve :frowning: )

For reference here is my traaefik.yml

api:
  dashboard: true
  debug: true

log:
  level: FATAL

entryPoints:
  http:
    address: ":80"
  https:
    address: ":443"
    http:
      tls:
        certResolver: letsencrypt

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    swarmMode: true
    exposedByDefault: false
  file:
    directory: /data/sites-enabled
    watch: true

certificatesResolvers:
  letsencrypt:
    acme:    
      email: email@mydomain.com
      storage: /data/acme.json
      certificatesDuration: 72
      dnsChallenge:
        provider: cloudflare
        resolvers:
          - "1.1.1.1:53"
          - "8.8.8.8:53"
          - "9.9.9.9:53"

and i don't know K8S - but this is what a docker provider looks like - can you use label syntax in K8S?

version: "3.3"

services:
  traefik:
    image: "traefik:latest"
    restart: always 
    ports:                  #these 3 ports must be published (8080 is for the dashboard reverse proxy)
      - "80:80"
      - "443:443"
      - "8081:8080"
    networks:
      - traefik-public
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - data:/data         # i user glusterfs plugin volume driver instead of regaulr bind mounts
    configs:               # i use configs to store configs and make avasilabel to entire swarm cluster
      - source: traefik_config
        target: /etc/traefik/traefik.yml
    environment:
      - CF_DNS_API_TOKEN=<foo>
    deploy:
      placement:
        constraints:
          - node.role == manager
      labels:
        - "traefik.enable=true"
        #Traefik Router Setup
        - "traefik.http.routers.traefik.rule=Host(`traefik.mydomain.com`)"
        - "traefik.http.routers.traefik.service=api@internal"
        - "traefik.http.routers.traefik.entrypoints=https"
        # - "traefik.http.routers.traefik.middlewares=oauth2-redirect@file"
        - "traefik.http.routers.traefik.middlewares=basic-auth@file"
        
        # Dashboard Service Setup
        - "traefik.http.services.dashboard.loadbalancer.server.port=8081" # this isn't really used but is required, in reality the port can be anything and the services.name can be anyting note 8081 must be published, this for dashboard only


        # global redirect to https
        - "traefik.http.routers.http-catchall.rule=hostregexp(`{host:.+}`)"
        - "traefik.http.routers.http-catchall.entrypoints=http"
        - "traefik.http.routers.http-catchall.middlewares=redirect-to-https"

        # middleware redirect
        - "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"

configs:
  traefik_config:
    external: true

volumes:
  data:
    driver: gluster-vol1

networks:
  traefik-public:

maybe that will have something that help. good luck

Sorry the delay responding. Where are you getting certificatesResolvers and entryPoints keys from the Helm chart? If you look at the default Helm chart values.yaml file those keys are not even specified. Thus, why I thought I had to use the command line flags to configure everything with Traefik v2.

Default Helm chart values:
https://github.com/traefik/traefik-helm-chart/blob/master/traefik/values.yaml

Here are my custom values.yaml, note I removed a few of the cli arguments as they did not work and added to confusion.

logs:
  general:
    level: INFO
  access:
    enabled: true

additionalArguments:
  - "--log.level=INFO"
  - "--global.checknewversion=false"
  - "--global.sendanonymoususage=false"
  - "--certificatesresolvers.le.acme.storage=/data/acme.json"
  - "--certificatesresolvers.le.acme.email=justin@watchn.co"

ports:
  web:
    redirectTo: websecure

persistence:
  enabled: true
  path: /data
  size: 1Gi

resources:
  requests:
    cpu: 100m
    memory: 32Mi
  limits:
    cpu: 1

Sorry i don't know about helm etc but i don't think it changes the fundamentals of how traefik works?

All i was pointing out is this from the docs (as you were asking for ideas why commands didnt seem to work) - the static config locations in v2.6 are mutually exclusive. Took me ages to figure this out - i couldn't fathom out for a couple of days why command line and env vars were not working for me - it was because i had used traefik.yml. then i found this....

The Static Configuration

There are three different, mutually exclusive (i.e. you can use only one at the same time), ways to define static configuration options in Traefik:

  1. In a configuration file
  2. In the command-line arguments
  3. As environment variables

another suggestion are you sure the /data directory and /acme.json have the right chown/chmod - i think i got silent failures when acme couldn't write to the location / file....

one last suggestion (take with grain of slat as i don't use command line) but you don't seem be matching the canonical as i don't see where either file defines websecure or binds the resolver (but i am new to this so maybe i am missing something?) This seems to be the syntax?

--entrypoints.websecure.address=:443
--entrypoints.websecure.http.tls.options=foobar
--entrypoints.websecure.http.tls.certResolver=leresolver
--entrypoints.websecure.http.tls.domains[0].main=example.com
--entrypoints.websecure.http.tls.domains[0].sans=foo.example.com,bar.example.com
--entrypoints.websecure.http.tls.domains[1].main=test.com
--entrypoints.websecure.http.tls.domains[1].sans=foo.test.com,bar.test.com

Hey @nodesocket

Seems you are missing the crucial annotations that are required for the Traefik Kubernetes Ingress provider. - here is the link to the docs with all available annotation.

This is the example Kubernetes Ingress resource:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: whoami-https
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.tls: "true"
    traefik.ingress.kubernetes.io/router.tls.certresolver: le
spec:
  rules:
    - host: whoami.<FIXME>.demo.traefiklabs.tech
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: whoami-svc
                port:
                  number: 80

Please note the minimum required annotations that have been added.
You mentioned spec.ingressClassName, so please ensure that follwing resoruce is created;

apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  labels:
    app.kubernetes.io/instance: traefik
  name: traefik
spec:
  controller: traefik.io/ingress-controller

The ingressClass name should match spec.ingressClassName in Ingress object.

Some time ago I recorded the following webinar and there are some examples that should help starting with Traefik on Kubernetes. I use Helm to deploy Traefik since it is the most easy way to deploy it and to start using it.

Here is the link to the recording:

And here is the source code:

I hope it will help you :slight_smile:

@jakubhajek thank you very much for the reply. That helped tremendously. I added the following annotations to my ingress definition:

  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.tls: "true"
    traefik.ingress.kubernetes.io/router.tls.certresolver: le

And also updated my Traefik values to be:

logs:
  general:
    level: INFO
  access:
    enabled: true

globalArguments: []

additionalArguments:
  - "--log.level=INFO"
  - "--global.checknewversion=false"
  - "--global.sendanonymoususage=false"
  - "--certificatesresolvers.le.acme.storage=/data/acme.json"
  - "--certificatesresolvers.le.acme.dnschallenge=true"
  - "--certificatesresolvers.le.acme.dnschallenge.provider=cloudflare"
  - "--certificatesresolvers.le.acme.email=justin@watchn.co"

ports:
  web:
    redirectTo: websecure

persistence:
  enabled: true
  path: /data
  size: 1Gi

resources:
  requests:
    cpu: 100m
    memory: 32Mi
  limits:
    cpu: 1

The last and missing piece is how to specify the CloudFlare credentials for use with the Let's Encrypt DNS challenge type. I know I can set the envars CF_API_EMAIL and CF_API_KEY directly in the Traefik values Helm chart but is there a way I can create a native Kubernetes secret and reference that secret in the Helm chart for Traefik instead? That way sensitive CloudFlare credentials are not stored in the chart configuration.

Finally, creating the custom IngressClass does not seem to be needed for my use-case.

Hello @nodesocket

Regarding the Cloudflare configuration and its sensitive data you can create the following array in the values.yaml

# Environment variables to be passed to Traefik's binary
env: 
 - name: CF_API_KEY
   valueFrom:
     secretKeyRef:
       name: cloudflare-credentials
       key: CF_API_KEY
 - name: CF_API_EMAIL
   valueFrom:
     secretKeyRef:
       name: cloudflare-credentials
       key: CF_API_EMAIL

Before deploying Traefik please ensure the cloudflare-credentials secret exists in the same namespace where Traefik will be deployed.

Those few lines will create environment variables that are being read from Kubernetes secret. Then its values will be accessible by Traefik inside a running pod.

Thanks!