ACME http challenge always returns 404

Traefik Traefik v3 (latest)
,
1

I have a pretty basic setup where I deploy traefik using kustomize (based on the latest helm release).

This is the manifest I am using:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: traefik
  namespace: traefik
spec:
  releaseName: traefik
  interval: 10m
  chart:
    spec:
      chart: traefik
      version: 37.1.2
      sourceRef:
        kind: HelmRepository
        name: traefik
        namespace: flux-system
  install:
    createNamespace: true
  upgrade:
    remediation:
      retries: 3
  values:
    service:
      type: LoadBalancer
    ports:
      web:
        redirections:
          entryPoint:
            to: websecure
            scheme: https
            permanent: true
    ingressRoute:
      dashboard:
        enabled: true
        matchRule: <redacted>
        entryPoints: ["websecure"]
        middlewares:
          - name: traefik-dashboard-auth
    logs:
      access:
        enabled: true
    certificatesResolvers:
      le:
        acme:
          email: <redacted>
          storage: /data/acme.json
          httpChallenge:
            entryPoint: web
    persistence:
      enabled: true
      name: acme-storage
      accessMode: ReadWriteOnce
      size: 128Mi
      path: /data
      storageClass: local-path
    podSecurityContext:
      runAsGroup: 65532
      runAsNonRoot: true
      runAsUser: 65532
      fsGroup: 65532
      fsGroupChangePolicy: "Always"
    deployment:
      initContainers:
        - name: volume-permissions
          image: busybox:latest
          command: ["sh", "-c", "ls -la /; touch /data/acme.json; chmod -v 600 /data/acme.json"]
          volumeMounts:
          - mountPath: /data
            name: acme-storage
    extraObjects:
      - apiVersion: v1
        kind: Secret
        metadata:
          name: traefik-dashboard-auth-secret
        type: kubernetes.io/basic-auth
        stringData:
          username: <redacted>
          password: <redacted>
      - apiVersion: traefik.io/v1alpha1
        kind: Middleware
        metadata:
          name: traefik-dashboard-auth
        spec:
          basicAuth:
            secret: traefik-dashboard-auth-secret

I am then deploying my app with an ingress that has this manifest (with the service):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: blabla
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/router.entrypoints: web,websecure
    traefik.ingress.kubernetes.io/router.tls: "true"
    traefik.ingress.kubernetes.io/router.tls.certresolver: le
spec:
  rules:
    - host: host.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: laravel-service
                port:
                  number: 80
  tls:
    - hosts:
        - host.example.com
---
apiVersion: v1
kind: Service
metadata:
  name: laravel-service
  labels:
    app: laravel
spec:
  selector:
    app: laravel
  ports:
    - port: 80
      targetPort: 80

However, the when trying to access http://host.example.com/.well-known/acme-challenge/<token> I always get a 404, even though I do see the rule that should catch it from the dashboard, so the certificate never gets issued.

One other thing I noticed from the dashboard is that my host is exposed both from web and from websecure which is fine, but the web config shows the TLS badge.

Any idea on how to fix this?

Thanks a lot.