Hi!
I'm trying to serve an application using NestJS but I'm not being able to do so.
I've already configured Traefik IngressRoutes to serve both Traefik Dashboard and also ArgoCD (and a couple more test apps), but I've been trying to deploy this new application for almost 2 days, without success.
The error is the following:
Bad Gateway
And this is the log Traefik outputs upon a request:
[traefik-c88c9f869-b8cm8] 10.0.1.122 - - [11/Dec/2020:03:13:20 +0000] "GET /graphql HTTP/2.0" 502 11 "-" "-" 764 "develop-business-app-64fa6977f85a45bb4625@kubernetescrd" "http://10.0.3.86:8080" 1ms
I don't know if there is any custom configuration I need to do in my app to use HTTP/2.0 or handle Traefik SSL (since the entry point is websecure). I've followed the docs over and over but I always get the same error (I've already tried to remove and installed Traefik again entirely)
Also, if I run kubectl port-forward I can use the application as expected.
Here are my configuration files:
This is my Traefik deployment:
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: traefik
labels:
app.kubernetes.io/name: traefik-proxy
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/component: infrastructure
app.kubernetes.io/part-of: traefik
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: traefik-proxy
template:
metadata:
labels:
app.kubernetes.io/name: traefik-proxy
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/component: infrastructure
app.kubernetes.io/part-of: traefik
spec:
serviceAccountName: traefik-ingress-controller
volumes:
- name: acme-certificates
emptyDir: {}
containers:
- name: traefik
image: traefik:v2.3
args:
- --accesslog
- --providers.kubernetescrd
- --ping
- --api.dashboard
- --entrypoints.traefik.address=:8080
- --entrypoints.web.address=:80
- --entrypoints.websecure.address=:443
- --entrypoints.web.http.redirections.entrypoint.to=websecure
- --entrypoints.websecure.http.tls.certResolver=letsencrypt
- --certificatesresolvers.letsencrypt.acme.email=accounts+letsencrypt@getbud.co
- --certificatesresolvers.letsencrypt.acme.storage=/etc/acme/letsencrypt.json
- --certificatesResolvers.letsencrypt.acme.dnsChallenge.provider=route53
- --certificatesResolvers.letsencrypt.acme.dnsChallenge.delayBeforeCheck=0
volumeMounts:
- name: acme-certificates
mountPath: /etc/acme
ports:
- containerPort: 8080
name: admin
protocol: TCP
- containerPort: 80
name: web
protocol: TCP
- containerPort: 443
name: websecure
protocol: TCP
livenessProbe:
failureThreshold: 3
httpGet:
path: /ping
port: 8080
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 2
readinessProbe:
failureThreshold: 1
httpGet:
path: /ping
port: 8080
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 2
This is my application deployment:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: business-app
labels:
app.kubernetes.io/name: business-app
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/component: business
app.kubernetes.io/part-of: application-layer
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: business-app
template:
metadata:
labels:
app.kubernetes.io/name: business-app
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/component: business
app.kubernetes.io/part-of: application-layer
spec:
containers:
- name: business-app
image: 904333181156.dkr.ecr.sa-east-1.amazonaws.com/business:$ECR_TAG <- this is updated with the latest tag using envsubst
ports:
- containerPort: 8080
name: web
protocol: TCP
This is my application service:
---
kind: Service
apiVersion: v1
metadata:
name: business-app
spec:
selector:
app.kubernetes.io/name: business-app
ports:
- name: web
port: 80
targetPort: 8080
And this is my IngressRoute:
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: business-app
labels:
app.kubernetes.io/name: business-app
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/component: business
app.kubernetes.io/part-of: application-layer
spec:
entryPoints:
- websecure
routes:
- match: Host(`api.develop.getbud.co`)
kind: Rule
services:
- name: business-app
port: 80
tls:
certResolver: letsencrypt
options: {}
Can someone give me a hint on what am I doing wrong?
Just an update, I've changed the loglevel of traefik to debug, and here is what it logs upon request:
[traefik-55888dfd67-r8b2c] time="2020-12-11T04:54:31Z" level=debug msg="Error while Peeking first byte: read tcp 10.0.3.86:80->10.0.1.122:44996: read: connection reset by peer"
[traefik-55888dfd67-r8b2c] time="2020-12-11T04:54:31Z" level=debug msg="Error while Peeking first byte: read tcp 10.0.3.86:8080->10.0.3.100:6380: read: connection reset by peer"
[traefik-55888dfd67-r8b2c] time="2020-12-11T04:54:32Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/graphql\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,pt-BR;q=0.5\"],\"Cache-Control\":[\"no-cache\"],\"Pragma\":[\"no-cache\"],\"Te\":[\"trailers\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (X11; Linux x86_64; rv:83.0) Gecko/20100101 Firefox/83.0\"],\"X-Forwarded-Host\":[\"api.develop.getbud.co\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"traefik-55888dfd67-r8b2c\"],\"X-Real-Ip\":[\"10.0.1.122\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"api.develop.getbud.co\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.0.1.122:27473\",\"RequestURI\":\"/graphql\",\"TLS\":null}"
[traefik-55888dfd67-r8b2c] time="2020-12-11T04:54:32Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/graphql\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,pt-BR;q=0.5\"],\"Cache-Control\":[\"no-cache\"],\"Pragma\":[\"no-cache\"],\"Te\":[\"trailers\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (X11; Linux x86_64; rv:83.0) Gecko/20100101 Firefox/83.0\"],\"X-Forwarded-Host\":[\"api.develop.getbud.co\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"traefik-55888dfd67-r8b2c\"],\"X-Real-Ip\":[\"10.0.1.122\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"api.develop.getbud.co\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.0.1.122:27473\",\"RequestURI\":\"/graphql\",\"TLS\":null}" ForwardURL="http://10.0.1.158:8080"
[traefik-55888dfd67-r8b2c] time="2020-12-11T04:54:32Z" level=debug msg="'502 Bad Gateway' caused by: dial tcp 10.0.1.158:8080: connect: connection refused"
[traefik-55888dfd67-r8b2c] time="2020-12-11T04:54:32Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/graphql\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,pt-BR;q=0.5\"],\"Cache-Control\":[\"no-cache\"],\"Pragma\":[\"no-cache\"],\"Te\":[\"trailers\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (X11; Linux x86_64; rv:83.0) Gecko/20100101 Firefox/83.0\"],\"X-Forwarded-Host\":[\"api.develop.getbud.co\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"traefik-55888dfd67-r8b2c\"],\"X-Real-Ip\":[\"10.0.1.122\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"api.develop.getbud.co\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.0.1.122:27473\",\"RequestURI\":\"/graphql\",\"TLS\":null}"
[traefik-55888dfd67-r8b2c] 10.0.1.122 - - [11/Dec/2020:04:54:32 +0000] "GET /graphql HTTP/2.0" 502 11 "-" "-" 754 "develop-business-app-64fa6977f85a45bb4625@kubernetescrd" "http://10.0.1.158:8080" 2ms
[traefik-55888dfd67-r8b2c] time="2020-12-11T04:54:32Z" level=debug msg="Error while Peeking first byte: read tcp 10.0.3.86:443->10.0.3.75:35314: read: connection reset by peer"
[traefik-55888dfd67-r8b2c] time="2020-12-11T04:54:32Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/favicon.ico\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"image/webp,*/*\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,pt-BR;q=0.5\"],\"Cache-Control\":[\"no-cache\"],\"Pragma\":[\"no-cache\"],\"Referer\":[\"https://api.develop.getbud.co/graphql\"],\"Te\":[\"trailers\"],\"User-Agent\":[\"Mozilla/5.0 (X11; Linux x86_64; rv:83.0) Gecko/20100101 Firefox/83.0\"],\"X-Forwarded-Host\":[\"api.develop.getbud.co\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"traefik-55888dfd67-r8b2c\"],\"X-Real-Ip\":[\"10.0.1.122\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"api.develop.getbud.co\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.0.1.122:27473\",\"RequestURI\":\"/favicon.ico\",\"TLS\":null}"
[traefik-55888dfd67-r8b2c] time="2020-12-11T04:54:32Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" ForwardURL="http://10.0.1.158:8080" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/favicon.ico\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"image/webp,*/*\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,pt-BR;q=0.5\"],\"Cache-Control\":[\"no-cache\"],\"Pragma\":[\"no-cache\"],\"Referer\":[\"https://api.develop.getbud.co/graphql\"],\"Te\":[\"trailers\"],\"User-Agent\":[\"Mozilla/5.0 (X11; Linux x86_64; rv:83.0) Gecko/20100101 Firefox/83.0\"],\"X-Forwarded-Host\":[\"api.develop.getbud.co\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"traefik-55888dfd67-r8b2c\"],\"X-Real-Ip\":[\"10.0.1.122\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"api.develop.getbud.co\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.0.1.122:27473\",\"RequestURI\":\"/favicon.ico\",\"TLS\":null}"
[traefik-55888dfd67-r8b2c] time="2020-12-11T04:54:32Z" level=debug msg="'502 Bad Gateway' caused by: dial tcp 10.0.1.158:8080: connect: connection refused"
[traefik-55888dfd67-r8b2c] time="2020-12-11T04:54:32Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/favicon.ico\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"image/webp,*/*\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,pt-BR;q=0.5\"],\"Cache-Control\":[\"no-cache\"],\"Pragma\":[\"no-cache\"],\"Referer\":[\"https://api.develop.getbud.co/graphql\"],\"Te\":[\"trailers\"],\"User-Agent\":[\"Mozilla/5.0 (X11; Linux x86_64; rv:83.0) Gecko/20100101 Firefox/83.0\"],\"X-Forwarded-Host\":[\"api.develop.getbud.co\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"traefik-55888dfd67-r8b2c\"],\"X-Real-Ip\":[\"10.0.1.122\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"api.develop.getbud.co\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.0.1.122:27473\",\"RequestURI\":\"/favicon.ico\",\"TLS\":null}"
[traefik-55888dfd67-r8b2c] 10.0.1.122 - - [11/Dec/2020:04:54:32 +0000] "GET /favicon.ico HTTP/2.0" 502 11 "-" "-" 755 "develop-business-app-64fa6977f85a45bb4625@kubernetescrd" "http://10.0.1.158:8080" 1ms
So, it seems Traefik is receiving a connection refused from the pod. I've opened a shell inside Traefik's container and tried to run wget directly in the Pod IP and indeed I received the same error (connection refused).
Any other working pod whenever I ran wget it works.
Any ideas?
