Upgrade from k3s v1.25.3+k3s1 to v1.25.4+k3s1 broke traefik ingresses

Traefik Traefik v2 (latest)
1

Hello forum

Yesterday I attempted an upgrade of my k3s cluster to v1.25.4+k3s1. when I did all my ingresses stopped working. ( Getting a connection refused )

I then attempted a downgrade back to v1.25.3+k3s1 and still getting the connection refused.

Upon looking at the traefik pod the version is currently 2.9.1 and I am seeing the following in the pod logs repeated over and over

time="2022-11-24T05:51:04Z" level=error msg="Skipping service: no endpoints found" providerName=kubernetes ingress=prometheus-server serviceName=prometheus-server namespace=prometheus servicePort="&ServiceBackendPort{Name:,Number:80,}"
time="2022-11-24T05:51:05Z" level=error msg="Skipping service: no endpoints found" namespace=prometheus serviceName=prometheus-server servicePort="&ServiceBackendPort{Name:,Number:80,}" providerName=kubernetes ingress=prometheus-server
time="2022-11-24T05:51:05Z" level=error msg="Skipping service: no endpoints found" serviceName=opensearch-dashboards servicePort="&ServiceBackendPort{Name:,Number:5601,}" namespace=opensearch providerName=kubernetes ingress=opensearch-dashboards
time="2022-11-24T05:51:05Z" level=error msg="Skipping service: no endpoints found" namespace=prometheus servicePort="&ServiceBackendPort{Name:,Number:80,}" serviceName=prometheus-server providerName=kubernetes ingress=prometheus-server
time="2022-11-24T05:51:05Z" level=error msg="Skipping service: no endpoints found" namespace=opensearch providerName=kubernetes serviceName=opensearch-dashboards servicePort="&ServiceBackendPort{Name:,Number:5601,}" ingress=opensearch-dashboards
time="2022-11-24T05:51:05Z" level=error msg="Skipping service: no endpoints found" providerName=kubernetes serviceName=prometheus-server servicePort="&ServiceBackendPort{Name:,Number:80,}" ingress=prometheus-server namespace=prometheus
time="2022-11-24T05:51:05Z" level=error msg="Skipping service: no endpoints found" servicePort="&ServiceBackendPort{Name:,Number:5601,}" namespace=opensearch providerName=kubernetes serviceName=opensearch-dashboards ingress=opensearch-dashboards
time="2022-11-24T05:51:05Z" level=error msg="Skipping service: no endpoints found" providerName=kubernetes ingress=prometheus-server namespace=prometheus serviceName=prometheus-server servicePort="&ServiceBackendPort{Name:,Number:80,}"
time="2022-11-24T05:51:05Z" level=error msg="Skipping service: no endpoints found" ingress=opensearch-dashboards namespace=opensearch serviceName=opensearch-dashboards servicePort="&ServiceBackendPort{Name:,Number:5601,}" providerName=kubernetes
time="2022-11-24T05:51:06Z" level=error msg="Skipping service: no endpoints found" servicePort="&ServiceBackendPort{Name:,Number:80,}" providerName=kubernetes ingress=prometheus-server namespace=prometheus serviceName=prometheus-server
time="2022-11-24T05:51:06Z" level=error msg="Skipping service: no endpoints found" ingress=opensearch-dashboards namespace=opensearch providerName=kubernetes servicePort="&ServiceBackendPort{Name:,Number:5601,}" serviceName=opensearch-dashboards

can someone please point me in the right direction to restore function, before the upgrade all was working well.

I will be happy to post any output of any command, just let me know what is needed.

Thank you for helping to support

2

I figured out the issue. It was a conflict between k3s native loadbalancer and metallb. To fix I had to use the correct disable switch on the install script and ensure my internal dns was pointing to the correct loadbalancer IP.

3

Hi @handbrianc, we are facing similar problems with our cert-manager acme configuration and the new traefik in k3s 1.25.4-k3s1. Could you please share some details what exactly you needed to specify? I understand that the --disable servicelb when upgrading k3s with the installation script, is required when using metallb (what we also do).