Adding a waf middleware causes 404 on startup

Traefik
1

I have two docker compose projects:

  • traefik: contains traefik 3.6.1 (with v2 ruleset), waf (an instance of modsecurity-crs) and dummy (an instance of whoami used by modsecurity). Also contains traefiknet with bridge driver. Traefik uses the docker provider.
  • app: contains a web application and database. The web application is in external network traefik_traefiknet. This contains labels for traefik, causing it to expose the web application, generate a letsencrypt certificate and supply a http basic auth.

This setup has been working flawlessly for years. However, as soon as I add another middleware to the app pointing to waf, traefik will respond with 404 when attempting to connect to the web app. The reason is: error="invalid middleware "waf@docker" configuration: invalid middleware type or middleware does not exist".

I’ve managed to boil this down to what is likely a race condition. It occurs when:

  1. Have traefik and the waf (and whoami) running already
  2. Recreate the docker container of the web app

The issue sometimes also occurs on machine restart, which causes all docker containers to be started automatically (restart: always).

The issue can be fixed by restarting traefik, which causes it to see both the waf and web application.

This may be related to Middleware is sometimes not found when defined in the traefik service's docker labels - #12 by kevinpollet but I’m seeing a slightly different reproduction pattern and I don’t have healthchecks.

I really like the docker provider and switching the stack to a file provider would be a lot of work, as I have many such setups running.

Is there a way to make this more solid, i.e. have traefik automatically retry detecting the waf middleware, or defining a dependency order?

Best,
Kalsan

EDIT: as requested, here are the relevant config files:

The app:

services:
  web:
    image: hub.me.com/myapp:0.2.14
    restart: always
    env_file:
      - .env
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.myinstance.rule=Host(`myapp.com`) || Host(`www.myapp.com`)"
      - "traefik.http.routers.myinstance.entrypoints=https"
      - "traefik.http.routers.myinstance.tls.certresolver=letsencryptresolver"
      - "traefik.http.middlewares.waf.plugin.traefik-modsecurity-plugin.modSecurityUrl=http://waf:8080"
      - "traefik.http.middlewares.waf.plugin.traefik-modsecurity-plugin.maxBodySize=10485760"
      - "traefik.http.routers.myinstance.middlewares=waf@docker"
    depends_on:
      - db
    networks:
      - internal
      - traefik_traefiknet
    volumes:
      - ./web-storage:/app/storage

  db:
    image: postgres:18.1
    restart: always
    env_file:
      - .env
    networks:
      - internal
    volumes:
      - ./db-data:/var/lib/postgresql/18/docker

networks:
  internal:
    driver: bridge
  traefik_traefiknet:
    external: true

Traefik:


services:
  traefik:
    image: "traefik:v3.6.1"
    command:
      - "--log.level=INFO"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--providers.docker.network=traefik_traefiknet"
      - "--entrypoints.http.address=:80"
      - "--entrypoints.https.address=:443"
      - "--entrypoints.http.http.redirections.entrypoint.to=https"
      - "--entrypoints.http.http.redirections.entrypoint.scheme=https"
      - "--entrypoints.https.proxyprotocol.trustedips=192.168.1.1"
      - "--certificatesresolvers.letsencryptresolver.acme.httpchallenge=true"
      - "--certificatesresolvers.letsencryptresolver.acme.httpchallenge.entrypoint=http"
      - "--certificatesresolvers.letsencryptresolver.acme.email=me@me.com"
      - "--certificatesresolvers.letsencryptresolver.acme.storage=/letsencrypt/acme.json"
      - "--experimental.plugins.traefik-modsecurity-plugin.modulename=github.com/acouvreur/traefik-modsecurity-plugin"
      - "--experimental.plugins.traefik-modsecurity-plugin.version=v1.3.0"
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - "./letsencrypt:/letsencrypt"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    restart: always
    networks:
      - traefiknet
      - wafnet

  waf:
    image: owasp/modsecurity-crs:3.3.9-apache-202604040104
    environment:
      - PARANOIA=1
      - ANOMALY_INBOUND=10
      - ANOMALY_OUTBOUND=5
      - BACKEND=http://dummy
    volumes:
      - './RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf:/etc/modsecurity.d/owasp-crs/rules/RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf'
    restart: always
    networks:
      - wafnet

  dummy:
    image: containous/whoami
    restart: always
    networks:
      - traefiknet
      - wafnet

networks:
  traefiknet:
    driver: bridge
  wafnet:
    driver: bridge
2

You are looking for support, but don’t post your config files?

3

Thank you for your suggestion, I've updated the original post with an edit on the bottom.

4

In the example they define the middleware on Traefik and assign it on the target service, maybe that helps.

5

So do I, or am I misunderstanding?

6

Try to define the middleware in Traefik labels:

services:
  traefik:
    image: traefik:3.0.1
    ports:
      - "8000:80"
      - "8080:8080"
    command:
      - --api.dashboard=true
      - --api.insecure=true
      - --experimental.plugins.traefik-modsecurity-plugin.modulename=github.com/acouvreur/traefik-modsecurity-plugin
      - --experimental.plugins.traefik-modsecurity-plugin.version=v1.3.0
      - --providers.docker=true
      - --entrypoints.http.address=:80
    volumes:
      - '/var/run/docker.sock:/var/run/docker.sock'
    labels:
      - traefik.enable=true
      - traefik.http.services.traefik.loadbalancer.server.port=8080
      - traefik.http.middlewares.waf.plugin.traefik-modsecurity-plugin.modSecurityUrl=http://waf:8080
      - traefik.http.middlewares.waf.plugin.traefik-modsecurity-plugin.maxBodySize=10485760

Then assign it on your target service.

7

Ah, I think I understand now, thank you for your elaboration.

If I understood correctly, the following lines that are now present in the application should be moved to traefik's docker-compose.yml:

- "traefik.http.middlewares.waf.plugin.traefik-modsecurity-plugin.modSecurityUrl=http://waf:8080"
- "traefik.http.middlewares.waf.plugin.traefik-modsecurity-plugin.maxBodySize=10485760"

I'll perform this in a lab ASAP and post how it went within the next few days.

8

@bluepuma77 thanks a lot, that did indeed help!

Important: for this to work, the label traefik.enable.true MUST also be present in traefik's docker-compose.yml. Thus:

In traefik's docker compose:

labels:
      - "traefik.enable=true"
      - "traefik.http.middlewares.waf.plugin.traefik-modsecurity-plugin.modSecurityUrl=http://waf:8080"
      - "traefik.http.middlewares.waf.plugin.traefik-modsecurity-plugin.maxBodySize=10485760"

Then the two latter lines can be removed from all web services using that traefik instance, leaving only the line:

- "traefik.http.routers.myinstance.middlewares=waf@docker"