"tls: failed to find any PEM data in certificate input" when using TLSStore CRD

FabianNiehaus · January 23, 2021, 1:12pm

Since I have not been able to find any threads containing this specific setup, I decided to create a new one.

I have the following situation:

Traefik-specific namespace: traefik-system

CA-issued certificate, added to the traefik-system namespace via TLS-Secret:

apiVersion: v1
kind: Secret
metadata:
  name: tls-ingress
type: kubernetes.io/tls

data:
  tls.crt: |
    <redacted, certificate in .pem format without BEGIN and END lines>
  tls.key: |
    <redacted, key in .pem format without BEGIN and END lines>

Traefik installed to the traefik-system namespace via HelmChart:

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
name: traefik
spec:
repo: https://helm.traefik.io/traefik
chart: traefik
version: 9.12.3
valuesContent: |-
    ports:
      web:
        redirectTo: websecure
      websecure:
        tls:
          enabled: "true"

    # Only run traefik on ingress nodes
    nodeSelector:
      node-role.kubernetes.io/ingress: "true"
    tolerations:
      - key: "role"
        operator: "Equal"
        value: "ingress"
        effect: "NoSchedule"

    service:
      type: NodePort
      externalIPs:
        - 192.168.255.30

    additionalArguments:
      - "--log.level=DEBUG"

TLSStore CRD for the certificate, also in the traefik-system namespace:

apiVersion: traefik.containo.us/v1alpha1
kind: TLSStore
metadata:
  name: default
spec:
  defaultCertificate:
    secretName: tls-ingress

IngressRoute CRD for the Dashboard and API:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: traefik-dashboard
spec:
  routes:
    - match: Host(`domain.redacted.com`)
      kind: Rule
      services:
        - name: api@internal
          kind: TraefikService
      middlewares:
        - name: internal-whitelist

Looking into the traefik-pod's logs outputs these errors:

[...]

time="2021-01-23T12:32:49Z" level=debug msg="Creating middleware" entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2021-01-23T12:32:49Z" level=debug msg="No default certificate, generating one"
time="2021-01-23T12:32:49Z" level=info msg="Starting provider *crd.Provider {\"allowCrossNamespace\":true}"
time="2021-01-23T12:32:49Z" level=info msg="label selector is: \"\"" providerName=kubernetescrd
time="2021-01-23T12:32:49Z" level=info msg="Creating in-cluster Provider client" providerName=kubernetescrd
time="2021-01-23T12:32:49Z" level=warning msg="Cross-namespace reference between IngressRoutes and resources is enabled, please ensure that this is expected (see AllowCrossNamespace option)" providerName=kubernetescrd
time="2021-01-23T12:32:49Z" level=debug msg="Start TCP Server" entryPointName=traefik
time="2021-01-23T12:32:49Z" level=debug msg="Start TCP Server" entryPointName=web
time="2021-01-23T12:32:49Z" level=debug msg="Configuration received from provider kubernetescrd: {\"http\":{\"routers\":{\"traefik-system-traefik-dashboard-593b437cae31df1cf4ea\":{\"entryPoints\":[\"traefik\"],\"middlewares\":[\"traefik-system-internal-whitelist\"],\"service\":\"api@internal\",\"rule\":\"Host(`domain.redacted.com`)\"}},\"middlewares\":{\"traefik-system-internal-whitelist\":{\"ipWhiteList\":{\"sourceRange\":[\"192.168.255.0/24\"]}}}},\"tcp\":{},\"udp\":{},\"tls\":{\"stores\":{\"default\":{}}}}" providerName=kubernetescrd
time="2021-01-23T12:32:49Z" level=error msg="Error while creating certificate store: failed to load X509 key pair: tls: failed to find any PEM data in certificate input" tlsStoreName=default
time="2021-01-23T12:32:49Z" level=debug msg="Added outgoing tracing middleware api@internal" routerName=traefik-system-traefik-dashboard-593b437cae31df1cf4ea@kubernetescrd middlewareName=tracing middlewareType=TracingForwarder entryPointName=traefik
time="2021-01-23T12:32:49Z" level=debug msg="Creating middleware" entryPointName=traefik routerName=traefik-system-traefik-dashboard-593b437cae31df1cf4ea@kubernetescrd middlewareName=traefik-system-internal-whitelist@kubernetescrd middlewareType=IPWhiteLister
time="2021-01-23T12:32:49Z" level=debug msg="Setting up IPWhiteLister with sourceRange: [192.168.255.0/24]" entryPointName=traefik routerName=traefik-system-traefik-dashboard-593b437cae31df1cf4ea@kubernetescrd middlewareName=traefik-system-internal-whitelist@kubernetescrd middlewareType=IPWhiteLister
time="2021-01-23T12:32:49Z" level=debug msg="Adding tracing to middleware" entryPointName=traefik routerName=traefik-system-traefik-dashboard-593b437cae31df1cf4ea@kubernetescrd middlewareName=traefik-system-internal-whitelist@kubernetescrd
time="2021-01-23T12:32:49Z" level=debug msg="Added outgoing tracing middleware ping@internal" entryPointName=traefik routerName=ping@internal middlewareName=tracing middlewareType=TracingForwarder
time="2021-01-23T12:32:49Z" level=debug msg="Creating middleware" entryPointName=traefik middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2021-01-23T12:32:49Z" level=debug msg="Added outgoing tracing middleware noop@internal" middlewareType=TracingForwarder routerName=web-to-443@internal entryPointName=web middlewareName=tracing
time="2021-01-23T12:32:49Z" level=debug msg="Creating middleware" routerName=web-to-443@internal middlewareName=redirect-web-to-443@internal middlewareType=RedirectScheme entryPointName=web
time="2021-01-23T12:32:49Z" level=debug msg="Setting up redirection to https 443" routerName=web-to-443@internal middlewareName=redirect-web-to-443@internal middlewareType=RedirectScheme entryPointName=web
time="2021-01-23T12:32:49Z" level=debug msg="Adding tracing to middleware" entryPointName=web routerName=web-to-443@internal middlewareName=redirect-web-to-443@internal
time="2021-01-23T12:32:49Z" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=web
time="2021-01-23T12:32:49Z" level=debug msg="No default certificate, generating one"
time="2021-01-23T12:32:49Z" level=debug msg="Configuration received from provider kubernetes: {\"http\":{},\"tcp\":{}}" providerName=kubernetes
time="2021-01-23T12:32:49Z" level=error msg="Error while creating certificate store: failed to load X509 key pair: tls: failed to find any PEM data in certificate input" tlsStoreName=default

[...]

Accessing the Domain (mapped locally via /etc/hosts) simply results in a server not available error, which is to be expected since the Pod is in an error loop.

cakiwi · January 23, 2021, 2:11pm

Why are you removing the PEM headers and footers ? This is almost certainly a cause for error.

FabianNiehaus · January 23, 2021, 4:20pm

I tried to add the header and footer back to the file:

apiVersion: v1
kind: Secret
metadata:
    name: tls-ingress
type: kubernetes.io/tls

data:
tls.crt: |
  -----BEGIN CERTIFICATE-----
  <redacted, base64 .pem certificate string>
  -----END CERTIFICATE-----
tls.key: |
  -----BEGIN RSA PRIVATE KEY-----
  <redacted, base64 .pem key string>
  -----END RSA PRIVATE KEY-----

Applying that configuration does not work though:

$ kubectl apply -f  tls-ingress.yaml

Error from server (BadRequest): error when creating "tls-ingress.yaml": Secret in version "v1" cannot be handled as a Secret: v1.Secret.Data: decode base64: illegal base64 data at input byte 0, error found in #10 byte of ...|TE-----\n","tls.key"|..., bigger context ...|<redacted>\n-----END CERTIFICATE-----\n","tls.key":"-----BEGIN RSA PRIVATE KEY-----\<redacted>|...

cakiwi · January 23, 2021, 5:31pm

Sorry, I'm not up on kubernetes tls secrets, it seems like a weird thing to do, but looks like you were correct in doing it that way according to k8s docs.

I'm gonna shut up about stuff I don't know

FabianNiehaus · January 23, 2021, 5:47pm

you were correct in doing it that way according to k8s docs

Yeah, that is what confuses me so much about this

I'm gonna shut up about stuff I don't know

No worries, thanks for the input

FabianNiehaus · January 25, 2021, 9:30pm

I figured it out.

Copying the contents of the .pem-based key and certificate files into the tls-ingress Secret .yaml was not the correct way.

Apparently, kubectl create secret applies some form of encryption to the inputs. With the old definition, the secrets mounted inside the pod were not base64 strings, but rather byte mash. Since the secret held the original base64 strings, they were decrypted without ever being encrypted.

Comparing the secret after creating it the old way, and creating it from the same certificate and key with kubectl create secret also gave differing strings as results.

system · January 28, 2021, 9:30pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
TLSStore not used and no error messages in the logs Traefik v2 kubernetes-crd	1	274	November 27, 2023
Error logged for non-existent tls certResolver across namespaces Traefik v2 kubernetes-crd	3	781	February 8, 2021
"Error while creating certificate store: failed to load X509 key pair: tls: failed to find any PEM data Traefik v2 docker	4	2537	January 15, 2021
Traefik configuration issue with TLS certificates Traefik v3 (latest) kubernetes-crd , kubernetes-ingress	0	134	October 13, 2024
Cert Manager with Traefik 2 error Traefik v2 kubernetes-ingress	1	1021	February 17, 2021

"tls: failed to find any PEM data in certificate input" when using TLSStore CRD

Related topics