I have installed a Kubernetes cluster on 4 Raspberry Pis 4 for educational purposes. Now I am trying to setup an ingress using Traefik 1.7. As a starting point I want to put my ingress to an Nginx pod. Trying to connect to the ingress on my cluster results in a connection refused response.
> curl 192.168.178.31 -H "HOST: nginx"
curl: (7) Failed to connect to 192.168.178.31 port 80: Connection refused
> kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
kube01 Ready master 14h v1.17.3 192.168.178.31 <none> Raspbian GNU/Linux 10 (buster) 4.19.97-v7l+ docker://19.3.6
kube02 Ready <none> 14h v1.17.3 192.168.178.46 <none> Raspbian GNU/Linux 10 (buster) 4.19.97-v7l+ docker://19.3.6
kube03 Ready <none> 14h v1.17.3 192.168.178.47 <none> Raspbian GNU/Linux 10 (buster) 4.19.97-v7l+ docker://19.3.6
kube04 Ready <none> 14h v1.17.3 192.168.178.48 <none> Raspbian GNU/Linux 10 (buster) 4.19.97-v7l+ docker://19.3.6
Regarding troubleshooting I followed the information provided on the Kubernetes site
What works:
- Exposing nginx deployment as nodeport and connecting to it
- Kubernetes Dashboard
- Curl into service from Master node (curl 10.104.113.207:80)
- From ingress controller (sh in Traefik; installed alpine image): curl -H "HOST: nginx" localhost
- From nginx pod: 'nslookup nginx-deploy-main'
- From node kube01: "nslookup nginx-deploy-main.default.svc.cluster.local 10.38.0.0"
- From node kube01: curl 10.104.113.207
I have no glue what else I can check. Given all the things that works, it seem to be really related to the ingress not processing the request. I am really stuck and appreciate any help.
> kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default nginx-deploy-main-7cc547b6f7-tls5s 1/1 Running 0 13h
kube-system coredns-6955765f44-tddgb 1/1 Running 0 14h
kube-system coredns-6955765f44-wq986 1/1 Running 0 14h
kube-system etcd-kube01 1/1 Running 0 14h
kube-system kube-apiserver-kube01 1/1 Running 0 14h
kube-system kube-controller-manager-kube01 1/1 Running 0 14h
kube-system kube-proxy-58d2r 1/1 Running 0 14h
kube-system kube-proxy-kxcsg 1/1 Running 0 14h
kube-system kube-proxy-n5vjd 1/1 Running 0 14h
kube-system kube-proxy-sqnrm 1/1 Running 0 14h
kube-system kube-scheduler-kube01 1/1 Running 0 14h
kube-system traefik-ingress-controller-57b54878b6-d7q4z 1/1 Running 0 12h
kube-system weave-net-brd4s 2/2 Running 0 14h
kube-system weave-net-jkrsh 2/2 Running 0 14h
kube-system weave-net-nk6nr 2/2 Running 0 14h
kube-system weave-net-tbbh4 2/2 Running 0 14h
kubernetes-dashboard dashboard-metrics-scraper-7b8b58dc8b-w8dzl 1/1 Running 0 14h
kubernetes-dashboard kubernetes-dashboard-866f987876-bzwnv 1/1 Running 0 14h
My Service:
> kubectl describe svc nginx-deploy-main
Name: nginx-deploy-main
Namespace: default
Labels: run=nginx
Annotations: <none>
Selector: run=nginx-main
Type: ClusterIP
IP: 10.104.113.207
Port: <unset> 80/TCP
TargetPort: 80/TCP
Endpoints: 10.38.0.2:80
Session Affinity: None
Events: <none>
The Nginx pod:
> kubectl describe pod nginx-deploy-main-7cc547b6f7-tls5s
Name: nginx-deploy-main-7cc547b6f7-tls5s
Namespace: default
Priority: 0
Node: kube04/192.168.178.48
Start Time: Thu, 05 Mar 2020 20:27:20 +0100
Labels: pod-template-hash=7cc547b6f7
run=nginx-main
Annotations: <none>
Status: Running
IP: 10.38.0.2
IPs:
IP: 10.38.0.2
Controlled By: ReplicaSet/nginx-deploy-main-7cc547b6f7
Containers:
nginx:
Container ID: docker://e05485ef423dfa370eb1e20933cdd44a792ce78b3c1d9246f7e32e0d958e891a
Image: nginx
Image ID: docker-pullable://nginx@sha256:2539d4344dd18e1df02be842ffc435f8e1f699cfc55516e2cf2cb16b7a9aea0b
Port: <none>
Host Port: <none>
State: Running
Started: Thu, 05 Mar 2020 20:27:36 +0100
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-56nt9 (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-56nt9:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-56nt9
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
The ingress:
> kubectl describe ingress
Name: ingress-resource-1
Namespace: default
Address:
Default backend: default-http-backend:80 (<none>)
Rules:
Host Path Backends
---- ---- --------
nginx
/ nginx-deploy-main:80 (10.38.0.2:80)
Annotations:
kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"networking.k8s.io/v1beta1","kind":"Ingress","metadata":{"annotations":{},"name":"ingress-resource-1","namespace":"default"},"spec":{"rules":[{"host":"nginx","http":{"paths":[{"backend":{"serviceName":"nginx-deploy-main","servicePort":80},"path":"/"}]}}]}}
Events: <none>
Log file of ingress controller:
> kubectl logs traefik-ingress-controller-57b54878b6-d7q4z -n kube-system
time="2020-03-05T19:58:38Z" level=info msg="Traefik version v1.7.14 built on 2019-08-14_09:46:58AM"
time="2020-03-05T19:58:38Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.traefik.io/basics/#collected-data\n"
time="2020-03-05T19:58:38Z" level=info msg="Preparing server http &{Address::80 TLS:<nil> Redirect:<nil> Auth:<nil> WhitelistSourceRange:[] WhiteList:<nil> Compress:false ProxyProtocol:<nil> ForwardedHeaders:0x621da80} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s"
time="2020-03-05T19:58:38Z" level=info msg="Preparing server traefik &{Address::8080 TLS:<nil> Redirect:<nil> Auth:<nil> WhitelistSourceRange:[] WhiteList:<nil> Compress:false ProxyProtocol:<nil> ForwardedHeaders:0x621daa0} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s"
time="2020-03-05T19:58:38Z" level=info msg="Starting provider configuration.ProviderAggregator {}"
time="2020-03-05T19:58:38Z" level=info msg="Starting server on :8080"
time="2020-03-05T19:58:38Z" level=info msg="Starting provider *kubernetes.Provider {\"Watch\":true,\"Filename\":\"\",\"Constraints\":[],\"Trace\":false,\"TemplateVersion\":0,\"DebugLogGeneratedTemplate\":false,\"Endpoint\":\"\",\"Token\":\"\",\"CertAuthFilePath\":\"\",\"DisablePassHostHeaders\":false,\"EnablePassTLSCert\":false,\"Namespaces\":null,\"LabelSelector\":\"\",\"IngressClass\":\"\",\"IngressEndpoint\":null}"
time="2020-03-05T19:58:38Z" level=info msg="ingress label selector is: \"\""
time="2020-03-05T19:58:38Z" level=info msg="Creating in-cluster Provider client"
time="2020-03-05T19:58:38Z" level=info msg="Starting server on :80"
time="2020-03-05T19:58:38Z" level=info msg="Server configuration reloaded on :8080"
time="2020-03-05T19:58:38Z" level=info msg="Server configuration reloaded on :80"
time="2020-03-05T20:08:40Z" level=warning msg="A new release has been found: 2.1.6. Please consider updating."
time="2020-03-05T20:23:45Z" level=info msg="Server configuration reloaded on :8080"
time="2020-03-05T20:23:45Z" level=info msg="Server configuration reloaded on :80"
time="2020-03-05T21:10:00Z" level=info msg="Server configuration reloaded on :8080"
time="2020-03-05T21:10:00Z" level=info msg="Server configuration reloaded on :80"
time="2020-03-05T21:10:49Z" level=info msg="Server configuration reloaded on :8080"
time="2020-03-05T21:10:49Z" level=info msg="Server configuration reloaded on :80"
Endpoints:
> kubectl get endpoints nginx-deploy-main
NAME ENDPOINTS AGE
nginx-deploy-main 10.38.0.2:80 14h
Kubeproxy Log on kube01:
> journalctl -u kubeproxy
-- Logs begin at Tue 2020-03-03 17:24:11 GMT, end at Fri 2020-03-06 10:15:36 GMT. --
-- No entries --