Failed to download plugin due to tls error

I have deployed Traefik on K3S using helm and the following values.yml

  - "--global.sendanonymoususage=false"
  - "--global.checknewversion=false"

  - "--serversTransport.insecureSkipVerify=true"
  - "--log.level=WARN"
  - "--entrypoints.gitlab-ssh.address=:22/tcp"
  - "--entryPoints.web.proxyProtocol.trustedIPs=,,"
  - "--entryPoints.websecure.proxyProtocol.trustedIPs=,,"
  - "--entryPoints.web.forwardedHeaders.trustedIPs=,,"
  - "--entryPoints.websecure.forwardedHeaders.trustedIPs=,,"
  - ""
  - "--experimental.plugins.traefik-themepark.version=v1.3.0"

  enabled: true
  replicas: 2
  annotations: {}
  podAnnotations: {}
  additionalContainers: []
  initContainers: []
  - name: plugins

- name: plugins
  mountPath: /plugins-storage

    redirectTo: websecure
      enabled: true

# Enable experimental features
    enabled: true

    enabled: false

    enabled: true
    ingressClass: traefik
    allowExternalNameServices: true
    enabled: true
    allowExternalNameServices: true
      enabled: false

  enabled: true

  enabled: true
  type: LoadBalancer
  annotations: {}
  labels: {}
    loadBalancerIP: # This should be an IP within the MetalLB range
  loadBalancerSourceRanges: []
  externalIPs: []

However the pods are failing to download the plugins due to a tls error.

time="2023-04-21T05:11:34Z" level=error msg="Plugins are disabled because an error has occurred." error="failed to download plugin failed to call service: Get \"\": tls: failed to verify certificate: x509: certificate is valid for 7f45f56f42100bf88aee3f0ab3adde12.cf03e4201f51fc5b39e11448d034ebd6.traefik.default, not"

Is there a way to disable this check for plugins or is there genuinely a tls problem with traefik?


I'm not able to reproduce it maybe it was just a temporary problem. states: "Secure Renegotiation: OpenSSL handshake didn't succeed" shows "green" for all 6 IPs.

Maybe you just try again or try to run a wget in the Traefik container sh.

Thank you both for checking, it may be a local network issue then, it did seem persist for several hours. I'll investigate local network issues.

I restored all my VM's to a clean state, re-installed K3S using ansible and tried again today. Same problem is occurring.

❯ k exec -it --namespace traefik traefik-5777fd4fff-45m26 sh        
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
/ $ wget -O /dev/null
Connecting to (
48BB7C5E1A7F0000:error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1889:
ssl_client: SSL_connect
wget: error getting response: Connection reset by peer

But from a metalb pod it's working

❯ k exec -it --namespace metallb-system controller-c6c466d64-9cqkk sh  
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
~ $ curl
sh: curl: not found
~ $ wget > /dev/null
Connecting to (
wget: can't open 'v1.3.0': Read-only file system
~ $ wget -O /dev/null
Connecting to (
saving to '/dev/null'
null                 100% |************************************************************************************************************************************************************|  508k  0:00:00 ETA
'/dev/null' saved

Interestingly the traefik pod is trying to connect to traefik that's running on baremetal. But the metallb pod connects to the real ip. Both have the same resolv.conf file, so I'm not sure why.

/ $ cat /etc/resolv.conf 
search traefik.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

I am completely new to K8/K3S and this is my attempts to move some services into Kubernetes.

The above investigation led me to a DNS issue. resolv.conf on traefik's pods had as a search target, but metallb's did not. Digging deeper it looks like a wildcard * entry to on my Bind9 setup was causing this.

For now, I've removed that and manually listed out all the dns entries manually instead of the wildcard and it's working as expected now.

Modifying DNS zones feels hacky, but then forcing the exclusion of from resolv.conf (if even possible) also feels hacky. Honestly I have no idea why that entry exists in traefik's pods but not in metallb's pods.

Open to better solutions/advice on a cleaner resolution.

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.