Tlsstore not working with kubernetes-crd for self-signed certificate

Use helm to install traefik. The value file is this:

# Default values for Traefik
image:
  name: traefik
  tag: 2.2.8
  pullPolicy: IfNotPresent

#
# Configure the deployment
#
deployment:
  enabled: true
  # Number of pods of the deployment
  replicas: 3
  # Additional deployment annotations (e.g. for jaeger-operator sidecar injection)
  annotations: {}
  # Additional pod annotations (e.g. for mesh injection or prometheus scraping)
  podAnnotations: {}
  # Additional containers (e.g. for metric offloading sidecars)
  additionalContainers: []
  # Additional initContainers (e.g. for setting file permission as shown below)
  initContainers: []
    # The "volume-permissions" init container is required if you run into permission issues.
    # Related issue: https://github.com/containous/traefik/issues/6972
    # - name: volume-permissions
    #   image: busybox:1.31.1
    #   command: ["sh", "-c", "chmod -Rv 600 /data/*"]
    #   volumeMounts:
    #     - name: data
    #       mountPath: /data
  # Custom pod DNS policy. Apply if `hostNetwork: true`
  # dnsPolicy: ClusterFirstWithHostNet

# Pod disruption budget
podDisruptionBudget:
  enabled: false
  # maxUnavailable: 1
  # minAvailable: 0

# Create an IngressRoute for the dashboard
ingressRoute:
  dashboard:
    enabled: true
    # Additional ingressRoute annotations (e.g. for kubernetes.io/ingress.class)
    annotations: {}
    # Additional ingressRoute labels (e.g. for filtering IngressRoute by custom labels)
    labels: {}

rollingUpdate:
  maxUnavailable: 1
  maxSurge: 1


#
# Configure providers
#
providers:
  kubernetesCRD:
    enabled: true
  kubernetesIngress:
    enabled: true
    # IP used for Kubernetes Ingress endpoints
    publishedService:
      enabled: false
      # Published Kubernetes Service to copy status from. Format: namespace/servicename
      # By default this Traefik service
      # pathOverride: ""

#
# Add volumes to the traefik pod.
# This can be used to mount a cert pair or a configmap that holds a config.toml file.
# After the volume has been mounted, add the configs into traefik by using the `additionalArguments` list below, eg:
# additionalArguments:
# - "--providers.file.filename=/config/dynamic.toml"
volumes:
- name: traefik-secret-tls
  mountPath: "/certs"
  type: secret
# - name: configs
#   mountPath: "/config"
#   type: configMap

# Logs
# https://docs.traefik.io/observability/logs/
logs:
  # Traefik logs concern everything that happens to Traefik itself (startup, configuration, events, shutdown, and so on).
  general:
    # By default, the logs use a text format (common), but you can
    # also ask for the json format in the format option
    # format: json
    # By default, the level is set to ERROR. Alternative logging levels are DEBUG, PANIC, FATAL, ERROR, WARN, and INFO.
    level: ERROR
  access:
    # To enable access logs
    enabled: true
    # By default, logs are written using the Common Log Format (CLF).
    # To write logs in JSON, use json in the format option.
    # If the given format is unsupported, the default (CLF) is used instead.
    # format: json
    # To write the logs in an asynchronous fashion, specify a bufferingSize option.
    # This option represents the number of log lines Traefik will keep in memory before writing
    # them to the selected output. In some cases, this option can greatly help performances.
    # bufferingSize: 100
    # Filtering https://docs.traefik.io/observability/access-logs/#filtering
    filters: {}
      # statuscodes: "200,300-302"
      # retryattempts: true
      # minduration: 10ms
    # Fields
    # https://docs.traefik.io/observability/access-logs/#limiting-the-fieldsincluding-headers
    fields:
      general:
        defaultmode: keep
        names: {}
          # Examples:
          # ClientUsername: drop
      headers:
        defaultmode: drop
        names: {}
          # Examples:
          # User-Agent: redact
          # Authorization: drop
          # Content-Type: keep

globalArguments:
  - "--global.checknewversion"
  - "--global.sendanonymoususage"

# Enable persistence using Persistent Volume Claims
# ref: http://kubernetes.io/docs/user-guide/persistent-volumes/
# After the pvc has been mounted, add the configs into traefik by using the `additionalArguments` list below, eg:
# additionalArguments:
#
# Configure Traefik static configuration
# Additional arguments to be passed at Traefik's binary
# All available options available on https://docs.traefik.io/reference/static-configuration/cli/
## Use curly braces to pass values: `helm install --set="additionalArguments={--providers.kubernetesingress.ingressclass=traefik-internal,--log.level=DEBUG}"`
additionalArguments: 
- "--providers.consulcatalog.endpoint.address=http://consul-server:8500"
- "--serversTransport.insecureSkipVerify=true"
- "--accesslog.fields.headers.defaultmode=keep"
- "--accesslog"
# - "--providers.file.tls.certificates.keyFile=/certs/tls.key"
# - "--providers.file.tls.certificates.certFile=/certs/tls.crt"
# - "--certificatesresolvers.le.acme.storage=/data/acme.json"
#  - "--providers.kubernetesingress.ingressclass=traefik-internal"
#  - "--log.level=DEBUG"

# Environment variables to be passed to Traefik's binary
env: []
# - name: SOME_VAR
#   value: some-var-value
# - name: SOME_VAR_FROM_CONFIG_MAP
#   valueFrom:
#     configMapRef:
#       name: configmap-name
#       key: config-key
# - name: SOME_SECRET
#   valueFrom:
#     secretKeyRef:
#       name: secret-name
#       key: secret-key

envFrom: []
# - configMapRef:
#     name: config-map-name
# - secretRef:
#     name: secret-name

# Configure ports
ports:
  # The name of this one can't be changed as it is used for the readiness and
  # liveness probes, but you can adjust its config to your liking
  traefik:
    port: 9000
    # Use hostPort if set.
    # hostPort: 9000
    #
    # Use hostIP if set. If not set, Kubernetes will default to 0.0.0.0, which
    # means it's listening on all your interfaces and all your IPs. You may want
    # to set this value if you need traefik to listen on specific interface
    # only.
    # hostIP: 192.168.100.10

    # Defines whether the port is exposed if service.type is LoadBalancer or
    # NodePort.
    #
    # You SHOULD NOT expose the traefik port on production deployments.
    # If you want to access it from outside of your cluster,
    # use `kubectl port-forward` or create a secure ingress
    expose: false
    # The exposed port for this service
    exposedPort: 9000
    # The port protocol (TCP/UDP)
    protocol: TCP
  web:
    port: 8000
    # hostPort: 8000
    expose: true
    exposedPort: 80
    # The port protocol (TCP/UDP)
    protocol: TCP
    # Use nodeport if set. This is useful if you have configured Traefik in a
    # LoadBalancer
    nodePort: 32080
    # Port Redirections
    # Added in 2.2, you can make permanent redirects via entrypoints.
    # https://docs.traefik.io/routing/entrypoints/#redirection
    # redirectTo: websecure
  websecure:
    port: 8443
    # hostPort: 8443
    expose: true
    exposedPort: 443
    # The port protocol (TCP/UDP)
    protocol: TCP
    nodePort: 32443

# Options for the main traefik service, where the entrypoints traffic comes
# from.
service:
  enabled: true
  type: NodePort
  # Additional annotations (e.g. for cloud provider specific config)
  annotations: {}
  # Additional entries here will be added to the service spec. Cannot contains
  # type, selector or ports entries.
  spec: {}
    # externalTrafficPolicy: Cluster
    # loadBalancerIP: "1.2.3.4"
    # clusterIP: "2.3.4.5"
  loadBalancerSourceRanges: []
    # - 192.168.0.1/32
    # - 172.16.0.0/16
  externalIPs: []
    # - 1.2.3.4

## Create HorizontalPodAutoscaler object.
##
autoscaling:
  enabled: false
#   minReplicas: 1
#   maxReplicas: 10
#   metrics:
#   - type: Resource
#     resource:
#       name: cpu
#       targetAverageUtilization: 60
#   - type: Resource
#     resource:
#       name: memory
#       targetAverageUtilization: 60

# It will persist TLS certificates.
persistence:
  enabled: false
#  existingClaim: ""
  accessMode: ReadWriteOnce
  size: 128Mi
  # storageClass: ""
  path: /data
  annotations: {}
  # subPath: "" # only mount a subpath of the Volume into the pod

# If hostNetwork is true, runs traefik in the host network namespace
# To prevent unschedulabel pods due to port collisions, if hostNetwork=true
# and replicas>1, a pod anti-affinity is recommended and will be set if the
# affinity is left as default.
hostNetwork: false

# Whether Role Based Access Control objects like roles and rolebindings should be created
rbac:
  enabled: true

  # If set to false, installs ClusterRole and ClusterRoleBinding so Traefik can be used across namespaces.
  # If set to true, installs namespace-specific Role and RoleBinding and requires provider configuration be set to that same namespace
  namespaced: false

# The service account the pods will use to interact with the Kubernetes API
serviceAccount:
  # If set, an existing service account is used
  # If not set, a service account is created automatically using the fullname template
  name: ""

# Additional serviceAccount annotations (e.g. for oidc authentication)
serviceAccountAnnotations: {}

resources: {}
  # requests:
  #   cpu: "100m"
  #   memory: "50Mi"
  # limits:
  #   cpu: "300m"
  #   memory: "150Mi"
affinity: {}
# # This example pod anti-affinity forces the scheduler to put traefik pods
# # on nodes where no other traefik pods are scheduled.
# # It should be used when hostNetwork: true to prevent port conflicts
#   podAntiAffinity:
#     requiredDuringSchedulingIgnoredDuringExecution:
#     - labelSelector:
#         matchExpressions:
#         - key: app
#           operator: In
#           values:
#           - {{ template "traefik.name" . }}
#       topologyKey: failure-domain.beta.kubernetes.io/zone
nodeSelector: {}
tolerations: []

# Pods can have priority.
# Priority indicates the importance of a Pod relative to other Pods.
priorityClassName: ""

# Set the container security context
# To run the container with ports below 1024 this will need to be adjust to run as root
securityContext:
  capabilities:
    drop: [ALL]
  readOnlyRootFilesystem: true
  runAsGroup: 65532
  runAsNonRoot: true
  runAsUser: 65532

podSecurityContext:
  fsGroup: 65532

Use the following command to generate a self-signed certificate:

 openssl req -x509 -sha256 -nodes -days 3650 -newkey rsa:2048 -keyout traefik-k8s-test.key -out traefik-k8s-test.crt \
  -subj "/CN=*.k8sgate.io" \
  -addext "subjectAltName=IP:192.168.102.124,IP:192.168.102.125,IP:192.168.102.126"

The secret and tilsstore configuration file:

---
apiVersion: v1
kind: Secret
metadata:
  name: traefik-secret-tls
type: kubernetes.io/tls
data:
  tls.crt: |
    MIIDLDCCAhSgAwIBAgIUeOwMWkdB5moeNgXy3HAWCAVeMVgwDQYJKoZIhvcNAQEL
    BQAwFzEVMBMGA1UEAwwMKi5rOHNnYXRlLmlvMB4XDTIxMDgyNTAxMTgyNFoXDTMx
    MDgyMzAxMTgyNFowFzEVMBMGA1UEAwwMKi5rOHNnYXRlLmlvMIIBIjANBgkqhkiG
    9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxNN1+36XxU/OonjHLUhMS6JwUKCI8aBj9KRw
    20UMfdnViY/zPLwiVPJhZSCXX3eUE+j8Kiuzf8bM6EVdLUnceqCtkrQvtCGwLPOs
    hmmzWLDOJERh4ySyuEur6l7oVDtELrDeUZL1vJ5BdKwQ02PK8GXZTlyKZkiWPbBO
    9akbGLAMYZTjtiUj8zyBly609Q0GAv8S68yfEDIlSYBdH/LarF4yq8C1iNOjELr4
    Fu7f6IePpVO9FFsfTed8vGDSWoRmJWMG57dmNLu52Xlln30MR7glYwOnJmLYi7ZV
    qyq2vki9Hk8ydH7HK1i9mopry3dPR/VOckL7Tq7h4l9vGN7USQIDAQABo3AwbjAd
    BgNVHQ4EFgQUHNlgC27pzPhTymQLE60v/BREZ8MwHwYDVR0jBBgwFoAUHNlgC27p
    zPhTymQLE60v/BREZ8MwDwYDVR0TAQH/BAUwAwEB/zAbBgNVHREEFDAShwTAqGZ8
    hwTAqGZ9hwTAqGZ+MA0GCSqGSIb3DQEBCwUAA4IBAQCXCWGaaVR3tN//MivZJKIX
    Mi5kh33OzWczsgl97FeSbWEjErsCn14vu5jRx7W/nS1cKKz3vrFXOAaPbjVRkAHS
    GIS4YFzt6PucEA8adDf1FjD9loRavdqJgSlEuIXaoeyND8Uy7R5XXJbtyt5S4kyi
    8FDcKUoVTW3Wjepwr1LS87opOsfHczKWRAbQ4qQ22nwyNikka8Ve1TR4VbuO5ByS
    s5ysHndB3/sFJMxkuHPyHpTZfDEvVACj/5YOSseehrdQAr+sNuhladOCLUzPdihJ
    2Z56tgsNjm6/RE5bSMCYP1OZy9T/GdTyFDHb4QCDMsXY4upw991lF4ipF71xQ74U

  tls.key: |
    MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDE03X7fpfFT86i
    eMctSExLonBQoIjxoGP0pHDbRQx92dWJj/M8vCJU8mFlIJdfd5QT6PwqK7N/xszo
    RV0tSdx6oK2StC+0IbAs86yGabNYsM4kRGHjJLK4S6vqXuhUO0QusN5RkvW8nkF0
    rBDTY8rwZdlOXIpmSJY9sE71qRsYsAxhlOO2JSPzPIGXLrT1DQYC/xLrzJ8QMiVJ
    gF0f8tqsXjKrwLWI06MQuvgW7t/oh4+lU70UWx9N53y8YNJahGYlYwbnt2Y0u7nZ
    eWWffQxHuCVjA6cmYtiLtlWrKra+SL0eTzJ0fscrWL2aimvLd09H9U5yQvtOruHi
    X28Y3tRJAgMBAAECggEBAKfi62cawyBlg47oUJSab6GwK/dKB6V6tufs/+q4igAr
    N85/v+1VWD93A4Z790JSQDd/auAZqKFcFFonGGRR9UOQHJ3fHjWa6qhdLRL1xCnp
    ndwnjAtwuRUkTr66N5fpea4N5YRVQ1Cg+JopTqzRGUBWz7Xcj5zRW33BDojStq8l
    tCw19FioXhdI7eGqDInt0rPF5LxuuNe6VJLo6dPG5SzixmSYr57CkpmZBquZ+Wbw
    e9OOHAIq2dIV5MTvDKW/zYsq7gOhtHhaLY63v4VvbqJpx0ua61oDC0pwW/1ZqrEw
    viEy0WV8cOWb/iCcFawLpT3j7Rso4WyBeqgZ98IlrcUCgYEA6IgZYrf98EQQo1qe
    pNcdf/hLfVjjFA4ICRMYf1rsFeZzYc+i2Sj70cqJxPvWMFahjL1z++1eeHntMBFD
    moA4GWJltO+PS3VO9feNmn+a6aGEvACHLPty03vzQTSW0R6eimwRx3gnhMQyIDrQ
    hkPodTAhv9AKLMHYetvf5bVjSP8CgYEA2LDWrNITlsS49mZmz0aKZnVrheto1GJV
    bl47S4PIgXIpnRT/oWIsUmEd7q4n3xsUIyxD1Al3y45dncXIEq+soiWk4SxQbBIY
    lt7vDrxdrBq3PTtUj/GU8NhoBfYPPLMNdAOnqiwvE66xUz4OeE/VOWLNNb/RxiZH
    jXaXylzEWrcCgYBbuMuiqAbvzjlt5iHrzbqiLtPAJt6zm3FJ+xFPsxKq9NMQgvGu
    jT7/ms/LNrdfK9zfgh+NAS2hgh6wNi8fc3IteSW7QD+auTQjS7m8dpcpjv+nlOG9
    OZYqm5rxXogbBuPsoAJHVzrnYOiQ1AJK4+B+0TSOz8Yjr3bvx2lC19DJCwKBgBJC
    BDSXltLzTGpmgm8gkHTFnAOyrCiJCI2+HtzkA7b0eZEkvc1JSla+nBkKT4266F12
    U1jAW3mpSzYJ/iuk5bTb2yKl6zjP5YaAzj+9m17u+VWNdDiUsGoCsfEepHYG2DFY
    RX5PLsBHzeUnq2xzvZGSQ5wtx20cuJaw0WVZn6nNAoGBANe5wxU+PObf+D4cRirJ
    4NEMCe2rT+Lp6zucabz9oIlfOUwGKT4JUb7GNTgBit4e1ULHEB6qrr7gonGMS0cY
    Fhsv16aXnSyYNIiPEO2PhAZHupIbicEWxeJ/zxBuRquC3NnYoXkhjrFTsCHn/AJ+
    IYskw/pOTSPIhSsY00ebD/ki

---
apiVersion: traefik.containo.us/v1alpha1
kind: TLSStore
metadata:
  name: default
  namespace: default
spec:
  defaultCertificate:
    secretName:  traefik-secret-tls

Service configuration file:

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: minio
spec:
  entryPoints:
  - websecure
  routes:
  - match: Host(`minio.k8sgate.io`) 
    kind: Rule
    services:
    - name: minio
      namespace: minio-tenant-default
      port: 443
      kind: Service
      scheme: https
  tls:
    store:
      name: default
      namespace: default

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: minio-console
spec:
  entryPoints:
  - websecure
  routes:
  - match: Host(`minioc.k8sgate.io`) 
    kind: Rule
    services:
    - name: minio-tenant-default-console
      namespace: minio-tenant-default
      port: 9443
      kind: Service
      scheme: https
  tls:
    store:
      name: default
      namespace: default

Then I got the following error:

time="2021-08-25T01:32:48Z" level=error msg="Error while creating certificate store: failed to load X509 key pair: tls: failed to find any PEM data in certificate input" tlsStoreName=default

What is wrong? I cannot figure out.

Trey to change this:

  tls:
    store:
      name: traefik-secret-tls
      namespace: default

The official documentation says that "default" is the only tlsstore name that can be used.

Where is it in the documentation? Can you share the link?
I think I would be worth trying what I suggested. It worked for me.

https://doc.traefik.io/traefik/routing/providers/kubernetes-crd/#kind-tlsstore

Have you tried renaming your secret traefik-secret-tls to default.

Have you considered using Ingress instead of IngressRoute?

For example:

# HTTPS ingress
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: myapp-ingress
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.tls: "true"
spec:
  rules:
    - host: myapp.localhost
      http:
        paths:
          - backend:
              serviceName: myapp
              servicePort: 80
  tls:
    - secretName: myapp-cert
---
# Ingresses
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: redirect
spec:
  redirectScheme:
    scheme: https
    permanent: true
---
# http ingress for http->https redirection
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: myapp-redirect
  annotations:
    traefik.ingress.kubernetes.io/router.middlewares: default-redirect@kubernetescrd
    traefik.ingress.kubernetes.io/router.entrypoints: web
spec:
  rules:
    - host: myapp.localhost
      http:
        paths:
          - backend:
              serviceName: myapp
              servicePort: 80

The secret:


kubectl create secret tls myapp-cert \
--key ./cert/tls.key \
--cert ./cert/tls.crt