Diagnosing "Skipping service: no endpoints found" error message in logs

I've installed Traefik 2.5.1 into an EKS cluster running Kubernetes 1.21 version. When I add an ingress for an nginx service running in the nginx namespace, I see the following error:

$ kubectl logs -n traefik traefik-556b7776d4-gskbn -f
time="2021-09-27T04:29:13Z" level=info msg="Configuration loaded from flags."
{"level":"info","msg":"Traefik version 2.5.1 built on 2021-08-20T16:32:51Z","time":"2021-09-27T04:29:13Z"}
{"level":"info","msg":"Starting provider aggregator.ProviderAggregator {}","time":"2021-09-27T04:29:13Z"}
{"level":"info","msg":"Starting provider *traefik.Provider {}","time":"2021-09-27T04:29:13Z"}
{"level":"info","msg":"Starting provider *ingress.Provider {}","time":"2021-09-27T04:29:13Z"}
{"level":"info","msg":"ingress label selector is: \"\"","providerName":"kubernetes","time":"2021-09-27T04:29:13Z"}
{"level":"info","msg":"Creating in-cluster Provider client","providerName":"kubernetes","time":"2021-09-27T04:29:13Z"}
{"level":"info","msg":"Starting provider *crd.Provider {}","time":"2021-09-27T04:29:13Z"}
{"level":"info","msg":"label selector is: \"\"","providerName":"kubernetescrd","time":"2021-09-27T04:29:13Z"}
{"level":"info","msg":"Creating in-cluster Provider client","providerName":"kubernetescrd","time":"2021-09-27T04:29:13Z"}
{"level":"info","msg":"Starting provider *acme.Provider {\"email\":\"vendor@example.com\",\"caServer\":\"https://acme-staging-v02.api.letsencrypt.org/directory\",\"storage\":\"/data/acme.json\",\"keyType\":\"RSA4096\",\"dnsChallenge\":{\"provider\":\"route53\",\"delayBeforeCheck\":\"5s\"},\"ResolverName\":\"staging\",\"store\":{},\"TLSChallengeProvider\":{\"Timeout\":4000000000},\"HTTPChallengeProvider\":{}}","time":"2021-09-27T04:29:13Z"}
{"level":"info","msg":"Testing certificate renew...","providerName":"staging.acme","time":"2021-09-27T04:29:13Z"}
{"level":"info","msg":"Starting provider *acme.ChallengeTLSALPN {\"Timeout\":4000000000}","time":"2021-09-27T04:29:13Z"}
{"level":"info","msg":"Starting provider *acme.Provider {\"email\":\"vendor@example.com\",\"caServer\":\"https://acme-v02.api.letsencrypt.org/directory\",\"storage\":\"/data/acme.json\",\"keyType\":\"RSA4096\",\"dnsChallenge\":{\"provider\":\"route53\",\"delayBeforeCheck\":\"5s\"},\"ResolverName\":\"production\",\"store\":{},\"TLSChallengeProvider\":{\"Timeout\":4000000000},\"HTTPChallengeProvider\":{}}","time":"2021-09-27T04:29:13Z"}
{"level":"info","msg":"Testing certificate renew...","providerName":"production.acme","time":"2021-09-27T04:29:13Z"}
{"ingress":"nginx","level":"error","msg":"Skipping service: no endpoints found","namespace":"nginx","providerName":"kubernetes","serviceName":"nginx","servicePort":"\u0026ServiceBackendPort{Name:http,Number:0,}","time":"2021-09-27T04:44:02Z"}
{"ingress":"nginx","level":"error","msg":"Skipping service: no endpoints found","namespace":"nginx","providerName":"kubernetes","serviceName":"nginx","servicePort":"\u0026ServiceBackendPort{Name:http,Number:0,}","time":"2021-09-27T04:44:04Z"}

If I look at the service with kubectl, I do see an endpoint associated with the service.

$ kubectl describe svc -n traefik 
Name:                     traefik
Namespace:                traefik
Labels:                   app.kubernetes.io/instance=traefik
                        app.kubernetes.io/managed-by=Helm
                        app.kubernetes.io/name=traefik
                        helm.sh/chart=traefik-10.3.2
Annotations:              meta.helm.sh/release-name: traefik
                        meta.helm.sh/release-namespace: traefik
Selector:                 app.kubernetes.io/instance=traefik,app.kubernetes.io/name=traefik
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.100.199.183
IPs:                      10.100.199.183
LoadBalancer Ingress:     some-valid-elb-hostname.us-east-2.elb.amazonaws.com
Port:                     web  80/TCP
TargetPort:               web/TCP
NodePort:                 web  30045/TCP
Endpoints:                172.31.27.227:8000
Port:                     websecure  443/TCP
TargetPort:               websecure/TCP
NodePort:                 websecure  31380/TCP
Endpoints:                172.31.27.227:8443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
Type    Reason                Age   From                Message
----    ------                ----  ----                -------
Normal  EnsuringLoadBalancer  48m   service-controller  Ensuring load balancer
Normal  EnsuredLoadBalancer   48m   service-controller  Ensured load balancer

One thing that looks suspect to me is the "servicePort":"\u0026ServiceBackendPort{Name:http,Number:0,}"
in the errors logged.

Does anyone have any advice on what this might be or how to diagnose this?

Many thanks in advance,
-ag

2 Likes

Have you managed to fix the issue? I have a similar problem with traefik 2.5.4 and Kubernetes 1.21.

Having the same issue on traefik v2.9.10 kubernetes 1.26.3, having followed the instructions on the quickstart guide (Traefik Getting Started With Kubernetes - Traefik). I am also seeing the slightly confusing: servicePort="&ServiceBackendPort{Name:web,Number:0,}"

Hi @Sycrosity, thanks for your interest in Traefik!

Could you provide your configs and tell us what are you trying to achieve?