Hi!
I have a setup I am deploying on k8s cluster, there are traefik v2 itself, cert manager, backend (laravel) and frontend (nuxt). Everything was all good for a half of the year for this setup and I've used it on multiple projects, but a week ago I've deployed a new project, and frontend pods are working just fine as always, but laravel pods - I can't reach them from the browser, it's just stuck on loading most of the time, multiple times it gave me 502 EOF, no more logs when it's loading. In dashboard everthing is green. I just can't understand what's wrong, I've never changed anything even.
I've tried to redeploy, to upgrade traefik to v3, to disable cert manager, to remove probes, like everything, I am stuck on this thing for entire week, I can't find anything similar in google, I guess I just need help 
traefik.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: traefik-ingress-controller
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: traefik
labels:
app: traefik
spec:
replicas: {{ .Values.replicaCount }}
selector:
matchLabels:
app: traefik
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 50%
template:
metadata:
labels:
app: traefik
spec:
serviceAccountName: traefik-ingress-controller
containers:
- image: traefik:v3.0
imagePullPolicy: IfNotPresent
name: traefik
readinessProbe:
httpGet:
path: /ping
port: 80
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 2
livenessProbe:
httpGet:
path: /ping
port: 80
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 2
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443
- name: ws
containerPort: 6001
resources:
requests:
memory: 50Mi
cpu: 100m
limits:
memory: 50Mi
cpu: 500m
securityContext:
capabilities:
add:
- NET_BIND_SERVICE
drop:
- ALL
volumeMounts:
- mountPath: /etc/traefik/traefik.yml
name: config-static
subPath: traefik.yml
volumes:
- name: config-static
configMap:
name: traefik-static
---
apiVersion: v1
kind: Service
metadata:
name: traefik
annotations:
service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"
spec:
externalTrafficPolicy: Local
selector:
app: traefik
ports:
- port: 80
protocol: TCP
name: http
- port: 443
protocol: TCP
name: https
- port: 6001
protocol: TCP
name: ws
type: LoadBalancer
traefik-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: traefik-static
labels:
app: traefik
data:
traefik.yml: |
global:
checkNewVersion: false
sendAnonymousUsage: true
entryPoints:
http:
address: :80
http:
redirections:
entryPoint:
to: https
scheme: https
permanent: false
priority: 1
proxyProtocol:
trustedIPs:
{{- range .Values.loadBalancerTrustedIps }}
- {{ . }}
{{- end }}
forwardedHeaders:
trustedIPs:
{{- range .Values.loadBalancerTrustedIps }}
- {{ . }}
{{- end }}
https:
address: :443
http:
tls: {}
proxyProtocol:
trustedIPs:
{{- range .Values.loadBalancerTrustedIps }}
- {{ . }}
{{- end }}
forwardedHeaders:
trustedIPs:
{{- range .Values.loadBalancerTrustedIps }}
- {{ . }}
{{- end }}
ws:
address: :6001
http:
tls: {}
proxyProtocol:
trustedIPs:
{{- range .Values.loadBalancerTrustedIps }}
- {{ . }}
{{- end }}
forwardedHeaders:
trustedIPs:
{{- range .Values.loadBalancerTrustedIps }}
- {{ . }}
{{- end }}
providers:
kubernetesCRD: {}
api:
dashboard: true
ping:
entryPoint: http
log:
level: DEBUG
backend.yaml
kind: Deployment
apiVersion: apps/v1
metadata:
name: backend
namespace: default
labels:
app: backend
name: backend
spec:
replicas: {{ .Values.replicaCount }}
selector:
matchLabels:
app: backend
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 50%
template:
metadata:
labels:
app: backend
annotations:
rollme: {{ randAlphaNum 5 | quote }}
spec:
volumes:
- name: laravel-env
secret:
secretName: laravel-env
containers:
- image: {{ .Values.image }}
name: backend
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
ports:
- name: http
containerPort: 8000
volumeMounts:
- name: laravel-env
mountPath: /var/www/app/.env
subPath: env
restartPolicy: Always
imagePullSecrets:
- name: gitlab-registry
---
apiVersion: v1
kind: Service
metadata:
name: backend
namespace: default
spec:
ports:
- port: 80
protocol: TCP
targetPort: http
name: http
selector:
app: backend
backend-ingress-route.yaml
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: backend
namespace: default
spec:
entryPoints:
- https
routes:
- kind: Rule
match: Host(`{{ .Values.apiHost }}`) && (Path(`/`) || PathPrefix(`/api`) || PathPrefix(`/livewire`))
services:
- name: backend
port: 80
- kind: Rule
match: Host(`{{ .Values.apiHost }}`) && (PathPrefix(`/horizon`) || PathPrefix(`/pulse`) || PathPrefix(`/swagger`))
middlewares:
- name: vpn-access-development
namespace: default
services:
- name: backend
port: 80
tls:
secretName: tls
I've also tried to not start the server inside the pod on startup, in this case it gives solid 502 Bad Gateway with log 502 Bad Gateway error="dial tcp 10.244.0.152:8000: connect: connection refused", but the moment I am starting the server on the 8000 port and 0.0.0.0 host, so the traefik now CAN reach the server inside pod it's stuck in browser and no logs.
Also I've tried to reach the server from inside the pod by it's ip using curl, and I am getting the page, from other pods as well!
If you have ANY ideas of why it's happening, PLEASE share!