Failed to download plugin due to tls error

Traefik Traefik v2
1

I have deployed Traefik on K3S using helm and the following values.yml

globalArguments:
  - "--global.sendanonymoususage=false"
  - "--global.checknewversion=false"

additionalArguments:
  - "--serversTransport.insecureSkipVerify=true"
  - "--log.level=WARN"
  - "--entrypoints.gitlab-ssh.address=:22/tcp"
  - "--entryPoints.web.proxyProtocol.trustedIPs=10.23.2.0/24,172.16.0.0/12,10.42.0.0/16"
  - "--entryPoints.websecure.proxyProtocol.trustedIPs=10.23.2.0/24,172.16.0.0/12,10.42.0.0/16"
  - "--entryPoints.web.forwardedHeaders.trustedIPs=10.23.2.0/24,172.16.0.0/12,10.42.0.0/16"
  - "--entryPoints.websecure.forwardedHeaders.trustedIPs=10.23.2.0/24,172.16.0.0/12,10.42.0.0/16"
  - "--experimental.plugins.traefik-themepark.modulename=github.com/packruler/traefik-themepark"
  - "--experimental.plugins.traefik-themepark.version=v1.3.0"

deployment:
  enabled: true
  replicas: 2
  annotations: {}
  podAnnotations: {}
  additionalContainers: []
  initContainers: []
  additionalVolumes:
  - name: plugins

additionalVolumeMounts:
- name: plugins
  mountPath: /plugins-storage

ports:
  web:
    redirectTo: websecure
  websecure:
    tls:
      enabled: true

# Enable experimental features
experimental:
  plugins:
    enabled: true

ingressRoute:
  dashboard:
    enabled: false

providers:
  kubernetesCRD:
    enabled: true
    ingressClass: traefik
    allowExternalNameServices: true
  kubernetesIngress:
    enabled: true
    allowExternalNameServices: true
    publishedService:
      enabled: false

rbac:
  enabled: true

service:
  enabled: true
  type: LoadBalancer
  annotations: {}
  labels: {}
  spec:
    loadBalancerIP: 10.23.2.100 # This should be an IP within the MetalLB range
  loadBalancerSourceRanges: []
  externalIPs: []

However the pods are failing to download the plugins due to a tls error.

time="2023-04-21T05:11:34Z" level=error msg="Plugins are disabled because an error has occurred." error="failed to download plugin github.com/packruler/traefik-themepark: failed to call service: Get \"https://plugins.traefik.io/public/download/github.com/packruler/traefik-themepark/v1.3.0\": tls: failed to verify certificate: x509: certificate is valid for 7f45f56f42100bf88aee3f0ab3adde12.cf03e4201f51fc5b39e11448d034ebd6.traefik.default, not plugins.traefik.io"

Is there a way to disable this check for plugins or is there genuinely a tls problem with traefik?

2

Hello,

I'm not able to reproduce it maybe it was just a temporary problem.

3

TestTLS.com states: "Secure Renegotiation: OpenSSL handshake didn't succeed"

SSLLabs.com shows "green" for all 6 IPs.

Maybe you just try again or try to run a wget in the Traefik container sh.

4

Thank you both for checking, it may be a local network issue then, it did seem persist for several hours. I'll investigate local network issues.

5

I restored all my VM's to a clean state, re-installed K3S using ansible and tried again today. Same problem is occurring.

❯ k exec -it --namespace traefik traefik-5777fd4fff-45m26 sh        
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
/ $ wget https://plugins.traefik.io/public/download/github.com/packruler/traefik-themepark/v1.3.0 -O /dev/null
Connecting to plugins.traefik.io (10.23.2.5:443)
48BB7C5E1A7F0000:error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1889:
ssl_client: SSL_connect
wget: error getting response: Connection reset by peer

But from a metalb pod it's working

❯ k exec -it --namespace metallb-system controller-c6c466d64-9cqkk sh  
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
~ $ curl
sh: curl: not found
~ $ wget https://plugins.traefik.io/public/download/github.com/packruler/traefik-themepark/v1.3.0 > /dev/null
Connecting to plugins.traefik.io (104.26.3.101:443)
wget: can't open 'v1.3.0': Read-only file system
~ $ wget https://plugins.traefik.io/public/download/github.com/packruler/traefik-themepark/v1.3.0 -O /dev/null
Connecting to plugins.traefik.io (104.26.2.101:443)
saving to '/dev/null'
null                 100% |************************************************************************************************************************************************************|  508k  0:00:00 ETA
'/dev/null' saved

Interestingly the traefik pod is trying to connect to traefik that's running on baremetal. But the metallb pod connects to the real ip. Both have the same resolv.conf file, so I'm not sure why.

/ $ cat /etc/resolv.conf 
search traefik.svc.cluster.local svc.cluster.local cluster.local mydomain.com
nameserver 10.43.0.10
options ndots:5

I am completely new to K8/K3S and this is my attempts to move some services into Kubernetes.

6

The above investigation led me to a DNS issue. resolv.conf on traefik's pods had mydomain.com as a search target, but metallb's did not. Digging deeper it looks like a wildcard *.mydomain.com entry to 10.23.2.5 on my Bind9 setup was causing this.

For now, I've removed that and manually listed out all the dns entries manually instead of the wildcard and it's working as expected now.

Modifying DNS zones feels hacky, but then forcing the exclusion of mydomain.com from resolv.conf (if even possible) also feels hacky. Honestly I have no idea why that entry exists in traefik's pods but not in metallb's pods.

Open to better solutions/advice on a cleaner resolution.

1 Like
7

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.