404 responses to everything on HTTP (web) including ACME challenges but HTTPS (websecure) is fine

[couling]

I’m trying to get ACME HTTP challenge to work. But no matter what I do, I can’t persuade Traefik to accept HTTP requests. It accepts the HTTPS requests just fine (albeit with an invalid certificate) but rejects all HTTP requests with:

404 page not found

I first noticed that the ACME requests were failing

"GET /.well-known/acme-challenge/<snip> HTTP/1.1" 404 19 "-" "-" 1 "-" "-" 0ms
2025-12-16T08:26:05Z ERR Unable to obtain ACME certificate for domains error="unable to generate a certificate for the domains [hello.example.com]: error: one or more domains had a problem:\n[hello.example.com] invalid authorization: acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: <snip>: Invalid response from http://hello.example.com/.well-known/acme-challenge/<snip>: 404\n" ACME CA=https://acme-staging-v02.api.letsencrypt.org/directory acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory domains=["hello.example.com"] providerName=default.acme routerName=websecure-app-hello-world-hello-world-hello-example-com@kubernetes rule="Host(`hello.example.com`) && PathPrefix(`/`)"

But then when I tried a browser accessing via HTTP, I realised all requests were failing to get through to the service. I’m deploying using helm (Traefik Charts | charts)

ports:
  web:
    port: 8089
    # Have tried with and without this section
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
          permanent: true
  websecure:
    port: 8443

providers:
  kubernetesIngress:
    ingressEndpoint:
      publishedService: traefik/traefik
    enabled: true
  kubernetesGateway:
    enabled: false

service:
  enabled: true
  type: LoadBalancer

logs:
  general:
    level: INFO
  access:
    enabled: true

persistence:
  enabled: true
  accessMode: ReadWriteOnce
  size: 128Mi
  path: "/data"

# The "volume-permissions" init container is required if you run into permission issues.
# Related issue: https://github.com/traefik/traefik-helm-chart/issues/396
deployment:
  initContainers:
   - name: volume-permissions
     image: docker.io/busybox:latest
     command: ["sh", "-c", "touch /data/acme.json ; chown 65532:65532 /data /data/acme.json ; chmod -v 600 /data/acme.json"]
     volumeMounts:
       - name: data
         mountPath: /data

resources:
  requests:
    cpu: "100m"
    memory: "50Mi"
  limits:
    cpu: "300m"
    memory: "150Mi"

certificatesResolvers:
  default:
    acme:
      email: "example@example.com"
      caserver: https://acme-staging-v02.api.letsencrypt.org/directory
      storage: "/data/acme.json"
      httpChallenge:
        entryPoint: "web"

Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello-world
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/router.entrypoints: websecure,web
    traefik.ingress.kubernetes.io/router.tls: "true"
    traefik.ingress.kubernetes.io/router.tls.certresolver: default
spec:
  rules:
  - host: hello.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: hello-world
            port:
              name: http

Any suggestions on what I’m missing?

[bluepuma77]

You have a redirect on entrypoint. All http requests will automatically be answered with a redirect response. Client will retry with https, but your target service is probably not listening on websecure.

[couling]

Not true. Just to highlight 3 points from my first post:

• I get 404 not found. I do not get a 301 or 303. I have also confirmed this from the access log.
• HTTPS is working just fine. IE: the service behind websecure is there, the only issue is the default Traefik certificate.
• I’ve tried with and without that block. It actually changes nothing.

[couling]

Well that’s weird. I noticed that the dashboard shows a shield symbol next to the web router rule for the sites. So I tried to connect to it through HTTPS on port 80 and discovered that it gets routed correctly:

> curl https://hello.example.com/ --connect-to hello.example.com:192.168.0.2:80 --insecure
<content from service>

But when I try a near identical command without TLS it fails:

> curl http://hello.example.com/ --connect-to hello.example.com:192.168.0.2:80 --insecure
404 page not found

This results in hitting the service page. So it seems something’s misconfigured somewhere such that it’s refusing to serve without TLS encryption.

[couling]

So I’ve managed to get HTTPs working with valid certificates by switching to tlsChallenge and switching off httpChallenge.

I had a lot of problems getting that to work because the Traefik helm chart is a bit flaky passing options through. Despite the settings in values.yaml looking like traefik config, it tries to reformat settings as commandline arguments. It will silently ignore anything it doesn’t recognise or know how to format.

So to get this working I’ve used Kustomize invoking the helm chart. I patched command: traefik and args: [] to override passing config in through the commandline. Then I used a configMapGenerator to write my static config as a yaml file and finally added that as a volume mount into /etc/traefik/traefik.yaml.

This bypassed all of helm’s madness trying to reformat yaml config (that looks like trafik config but isn’t) into command line arguments and allowed me to just pass in plain old traefik config yaml file.

Still not fixed:

Traefik is still responding 404 to everything on web if it’s not TLS encrypted. Since the whole point of web (port 80) is to be unencrypted, that means it’s just not working and breaks ACME httpChallenge.

If anyone knows how to persuade Traefik to accept unencrypted traffic be routed on web that’d be great.

[couling]

Okay I think I’ve cracked it. Looks like the redirect wasn’t working but was very necessary.

The problem with helm is that it can silently drop settings if you make a mistake. It will not warn you at all of unknown settings. And the values.yaml looks similar but is not the same as the standard yaml configuration.

The kubernetes setup guide disagrees on how to add the redirect: Traefik EntryPoints Documentation - Traefik

My advice to any future reader is to look at arguments passed into deployment / pod and make sure that the redirect is actually present in the arguments. If it’s not you’ll run into trouble.