Getting ERR_INVALID_RESPONSE on 404 page

Traefik Traefik v2
1

I'm using Kamal Deploy (https://kamal-deploy.org) to deploy my application. It utilizes Docker and Traefik as the proxy layer in front of your application. My web app (SvelteKit) related configuration in my Kamal config look's like so:

servers:
  web:
    hosts:
      - 10.0.0.5 # web-us-east-1
      - 10.0.0.3 # web-us-east-2
      - 10.0.0.2 # web-us-east-3
    labels:
      traefik.http.routers.code_secure.entrypoints: websecure
      traefik.http.routers.code_secure.rule: Host(`snoculars.com`)
      traefik.http.routers.code_secure.tls: true
      traefik.http.routers.code_secure.tls.certresolver: letsencrypt
      traefik.http.routers.code_secure.tls.domains[0].main: snoculars.com
      traefik.http.routers.code_secure.tls.domains[0].sans: '*.snoculars.com'
      traefik.http.routers.code_secure.middlewares: custom404@docker
      # Custom 404 middleware
      traefik.http.middlewares.custom404.errors.status: '404'
      traefik.http.middlewares.custom404.errors.service: snoculars-web
      traefik.http.middlewares.custom404.errors.query: '/error/404'

And my Traefik container is configured like so:

traefik:
  image: traefik:v3.1.0
  options:
    publish:
      - '443:443'
      - '8080:8080'
    volume:
      - '/letsencrypt/acme.json:/letsencrypt/acme.json'
  args:
    api.dashboard: true
    api.insecure: true
    log.level: DEBUG
    accesslog.format: json
    accesslog.filters.statusCodes: '400-599'
    accesslog.filters.retryAttempts: true
    accesslog.filters.minDuration: 101ms
    entryPoints.websecure.address: ':443'
    certificatesResolvers.letsencrypt.acme.email: 'email@example.com'
    certificatesResolvers.letsencrypt.acme.storage: '/letsencrypt/acme.json'
    certificatesResolvers.letsencrypt.acme.dnschallenge: true
    certificatesResolvers.letsencrypt.acme.dnschallenge.provider: cloudflare
  env:
    secret:
      - CLOUDFLARE_API_KEY
    clear:
      CLOUDFLARE_EMAIL: email@example.com

When you go to my site it loads fine:

https://snoculars.com

If you go to a static asset URL it loads fine:

https://snoculars.com/_app/immutable/entry/start.CpW3_ZYD.js)

But if you load a URL that doesn't exist, it will return a ERR_INVALID_RESPONSE error in the browser (and it won't even load my SSL certificate)

https://snoculars.com/_app/immutable/entry/start.CpW3_ZYD.jsssssssss

The weird thing is that i'm using a docker-compose.yml file like so locally..and it does exactly what it should when it can't load a static asset (it should hit the custom 404 middleware)

name: snoculars
version: '3.1'

services:
  traefik:
    image: traefik:v3.1
    command:
      - '--api.insecure=true'
      - '--providers.docker=true'
      - '--providers.docker.exposedbydefault=false'
      - '--entrypoints.web.address=:80'
      - '--log.level=DEBUG'
    ports:
      - '80:80' # Expose port 80
      - '8080:8080' # The Web UI
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    labels:
      - 'traefik.enable=true'
      - 'traefik.http.routers.traefik.rule=Host(`traefik.localhost`)'
      - 'traefik.http.routers.traefik.service=api@internal'
      - 'traefik.http.routers.traefik.entrypoints=web'
      # Custom 404 middleware
      - 'traefik.http.middlewares.custom404.errors.status=404'
      - 'traefik.http.middlewares.custom404.errors.service=web'
      - 'traefik.http.middlewares.custom404.errors.query=/error/404'
  web:
    env_file: './.env'
    labels:
      - 'traefik.enable=true'
      - 'traefik.http.routers.web.rule=Host(`localhost`)'
      - 'traefik.http.routers.web.entrypoints=web'
      - 'traefik.http.services.web.loadbalancer.server.port=3000'
      # Apply the custom 404 middleware
      - 'traefik.http.routers.web.middlewares=custom404@docker'
    build:
      context: .
      dockerfile: ./Dockerfile.dev.web

Does anyone know why I might be getting this ERR_INVALID_RESPONSE error?

2

It's called "debugging" when you try to figure out why something is not working.

It can be an art form :wink:

Some ideas to become an expert:

  1. Reduce complexity: maybe remove the middlewares for testing
  2. Get more information: enable Traefik debug log and Traefik access log in JSON
  3. Challenge current assumptions: how can you see an error if "it won't even load my SSL certificate"
  4. If it's open source, you can grep* the source for ERR_INVALID_RESPONSE, to find it's not there, so must be something else
  5. If you use further systems down the line, check their logs

Your friendly Traefik and SvelteKit colleague

* For the shell enthusiasts:

$ git clone https://github.com/traefik/traefik --depth=1
$ grep -r "ERR_INVALID_RESPONSE" ./traefik