Log file spammed after unknown malicious POST request - 499 errors

Traefik Traefik v2
1

I am using traefik v2.10.5 in docker to permanently redirect two domains and do HTTP to HTTPS. This is running fine for the last 3 years almost. Recently, I started to see traefik generating over 200GB of log with 499 errors (within about 24 hours!) which resulted in my server/service going offline.

The second time around, I caught it and tracked the request back to where it started. It looks to me like a malicious/malformed request is sent to traefik, which results in traefik seemingly talking to itself, time-outing, retrying until oblivion. Looks like a DoS attack to me, but I can't take out enough to be sure. Or could this be a simple configuration issue or bug?

---snip
149.28.xxx.xxx - - [28/Nov/2023:07:10:17 +0000] "POST /wp-login.php HTTP/1.1" 308 18 "-" "-" 17390 "website-redirect-http@docker" "-" 0ms
149.28.xxx.xxx - - [28/Nov/2023:07:10:17 +0000] "POST /wp-login.php HTTP/1.1" 499 21 "-" "-" 17391 "website-redirect-http@docker" "ht tp://172.19.0.2:80" 14676ms
172.19.0.2 - - [28/Nov/2023:07:10:17 +0000] "POST /wp-login.php HTTP/1.1" 499 21 "-" "-" 17392 "website-redirect-http@docker" "ht tp://172.19.0.2:80" 14675ms
172.19.0.2 - - [28/Nov/2023:07:10:17 +0000] "POST /wp-login.php HTTP/1.1" 499 21 "-" "-" 17393 "website-redirect-http@docker" "ht tp://172.19.0.2:80" 14675ms
172.19.0.2 - - [28/Nov/2023:07:10:17 +0000] "POST /wp-login.php HTTP/1.1" 499 21 "-" "-" 17394 "website-redirect-http@docker" "ht tp:://172.19.0.2:80" 14674ms
172.19.0.2 - - [28/Nov/2023:07:10:17 +0000] "POST /wp-login.php HTTP/1.1" 499 21 "-" "-" 17395 "website-redirect-http@docker" "ht tp:://172.19.0.2:80" 14674ms
172.19.0.2 - - [28/Nov/2023:07:10:17 +0000] "POST /wp-login.php HTTP/1.1" 499 21 "-" "-" 17396 "website-redirect-http@docker" "ht tp://172.19.0.2:80" 14674ms
172.19.0.2 - - [28/Nov/2023:07:10:17 +0000] "POST /wp-login.php HTTP/1.1" 499 21 "-" "-" 17397 "website-redirect-http@docker" "ht tp://172.19.0.2:80" 14674ms
172.19.0.2 - - [28/Nov/2023:07:10:17 +0000] "POST /wp-login.php HTTP/1.1" 499 21 "-" "-" 17398 "website-redirect-http@docker" "ht tp://172.19.0.2:80" 14674ms
172.19.0.2 - - [28/Nov/2023:07:10:17 +0000] "POST /wp-login.php HTTP/1.1" 499 21 "-" "-" 17399 "website-redirect-http@docker" "ht tp://172.19.0.2:80" 14674ms
172.19.0.2 - - [28/Nov/2023:07:10:17 +0000] "POST /wp-login.php HTTP/1.1" 499 21 "-" "-" 17400 "website-redirect-http@docker" "ht tp://172.19.0.2:80" 14673ms
172.19.0.2 - - [28/Nov/2023:07:10:17 +0000] "POST /wp-login.php HTTP/1.1" 499 21 "-" "-" 17401 "website-redirect-http@docker" "ht tp://172.19.0.2:80" 14673ms
172.19.0.2 - - [28/Nov/2023:07:10:17 +0000] "POST /wp-login.php HTTP/1.1" 499 21 "-" "-" 17402 "website-redirect-http@docker" "ht tp://172.19.0.2:80" 14673ms
172.19.0.2 - - [28/Nov/2023:07:10:17 +0000] "POST /wp-login.php HTTP/1.1" 499 21 "-" "-" 17403 "website-redirect-http@docker" "ht tp://172.19.0.2:80" 14673ms
172.19.0.2 - - [28/Nov/2023:07:10:17 +0000] "POST /wp-login.php HTTP/1.1" 499 21 "-" "-" 17404 "website-redirect-http@docker" "ht tp://172.19.0.2:80" 14673ms
172.19.0.2 - - [28/Nov/2023:07:10:17 +0000] "POST /wp-login.php HTTP/1.1" 499 21 "-" "-" 17405 "website-redirect-http@docker" "ht tp://172.19.0.2:80" 14672ms
172.19.0.2 - - [28/Nov/2023:07:10:17 +0000] "POST /wp-login.php HTTP/1.1" 499 21 "-" "-" 17406 "website-redirect-http@docker" "ht tp://172.19.0.2:80" 14672ms
[...]
log hammering on from here with the same message until server fills up it's storage...
---snap

Thank you!

2

499 is a client error. Can you reproduce it, while enabling access log in JSON?

Did you change something in the app, Traefik version or Traefik config?

Share your full Traefik static and dynamic config, and docker-compose.yml if used.

Note that you can limit the Docker log file size and number of files to keep:

docker run --log-opt max-size=100m --log-opt max-file=3 <image_name>
3

Thank you for the quick reply.

I can't reproduce it myself (yet). But I have started to increase the logging (+switch to JSON) should it happen again. It usually starts during the night.

I do also run NextCloud on the same system in another Docker image. Occasionally, I also see HTTP 499 for some requests there. Not sure whether it is related to my reported issue. The changes I can recall: software updated has been traefik itself to v2.10.5. NextCloud v.25 (latest subversion) and also the normal OS host packages have been update recently.

Thank you for providing the log file limit settings. That should help with my mitigations in place now :).

4

Traefik has been upgraded to v2.10.6, but still, almost every day (except weekend at this point) the same sort of attack / issue happens:

Time frame when the issue starts:

2023-12-04T13:39:50+11:00
...
2023-12-04T13:44:08+11:00

Initial request within that time frame (HTTP 308):

{
  "ClientAddr": "35.188.22.1XX:52498",
  "ClientHost": "35.188.22.1XX",
  "ClientPort": "52498",
  "ClientUsername": "-",
  "DownstreamContentSize": 18,
  "DownstreamStatus": 308,    ************
  "Duration": 146803,
  "GzipRatio": 0,
  "OriginContentSize": 0,
  "OriginDuration": 0,
  "OriginStatus": 0,    ************
  "Overhead": 146803,
  "RequestAddr": "the_REDACTED.com.au",
  "RequestContentSize": 0,
  "RequestCount": 15916,
  "RequestHost": "the_REDACTED.com.au",
  "RequestMethod": "POST",
  "RequestPath": "/wp-login.php",
  "RequestPort": "-",
  "RequestProtocol": "HTTP/1.1",
  "RequestScheme": "http",
  "RetryAttempts": 0,
  "RouterName": "the_REDACTED-redirect-http@docker",
  "StartLocal": "2023-12-04T13:43:53.379161919+11:00",
  "downstream_Location": "http://DOWNSTREAM-SITE-REDACTED.com.au/the_REDACTED/wp-login.php",
  "entryPointName": "web",
  "level": "info",
  "msg": "",
  "request_Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
  "request_Accept-Encoding": "gzip, deflate",
  "request_Accept-Language": "en-US,en;q=0.5",
  "request_Content-Length": "138",
  "request_Content-Type": "application/x-www-form-urlencoded",
  "request_Cookie": "wordpress_test_cookie=WP Cookie check;",
  "request_Referer": "http://the_REDACTED.com.au/wp-login.php",
  "request_User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:109.0) Gecko/20100101 Firefox/111.0",
  "request_X-Forwarded-Host": "the_REDACTED.com.au",
  "request_X-Forwarded-Port": "80",
  "request_X-Forwarded-Proto": "http",
  "request_X-Forwarded-Server": "d0f1ce6e2bf4",
  "request_X-Real-Ip": "35.188.22.1XX",
  "time": "2023-12-04T13:43:53+11:00"
}

Next request (HTTP 499):

{
  "ClientAddr": "35.188.22.1XX:52482",
  "ClientHost": "35.188.22.1XX",
  "ClientPort": "52482",
  "ClientUsername": "-",
  "DownstreamContentSize": 21,
  "DownstreamStatus": 499,    ************
  "Duration": 14698508773,
  "OriginContentSize": 21,
  "OriginDuration": 14698218785,
  "OriginStatus": 499,    ************
  "Overhead": 289988,
  "RequestAddr": "THE_REDACTED.COM.AU",
  "RequestContentSize": 117,
  "RequestCount": 15917,
  "RequestHost": "THE_REDACTED.COM.AU",
  "RequestMethod": "POST",
  "RequestPath": "/wp-login.php",
  "RequestPort": "-",
  "RequestProtocol": "HTTP/1.1",
  "RequestScheme": "http",
  "RetryAttempts": 0,
  "RouterName": "the_REDACTED-redirect-http@docker",
  "ServiceAddr": "172.19.0.2:80",
  "ServiceName": "traefik-traefik@docker",
  "ServiceURL": {
    "Scheme": "http",
    "Opaque": "",
    "User": null,
    "Host": "172.19.0.2:80",
    "Path": "",
    "RawPath": "",
    "OmitHost": false,
    "ForceQuery": false,
    "RawQuery": "",
    "Fragment": "",
    "RawFragment": ""
  },
  "StartLocal": "2023-12-04T13:43:53.381521314+11:00",
  "entryPointName": "web",
  "level": "info",
  "msg": "",
  "request_Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
  "request_Accept-Encoding": "gzip, deflate",
  "request_Accept-Language": "en-US,en;q=0.5",
  "request_Content-Length": "117",
  "request_Content-Type": "application/x-www-form-urlencoded",
  "request_Cookie": "wordpress_test_cookie=WP Cookie check;",
  "request_Referer": "HTTP://THE_REDACTED.COM.AU/wp-login.php",
  "request_User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:109.0) Gecko/20100101 Firefox/111.0",
  "request_X-Forwarded-Host": "THE_REDACTED.COM.AU",
  "request_X-Forwarded-Port": "80",
  "request_X-Forwarded-Proto": "http",
  "request_X-Forwarded-Server": "d0f1ce6e2bf4",
  "request_X-Real-Ip": "35.188.22.1XX",
  "time": "2023-12-04T13:44:08+11:00"
}

...then the log flooding begins (all HTTP 499, but ClientAddr[ess] replaced with the internal docker image IP:

{
  "ClientAddr": "172.19.0.2:41844",
  "ClientHost": "172.19.0.2",
  "ClientPort": "41844",
  "ClientUsername": "-",
  "DownstreamContentSize": 21,
  "DownstreamStatus": 499,
  "Duration": 14698659506,
  "OriginContentSize": 21,
  "OriginDuration": 14698065932,
  "OriginStatus": 499,
  "Overhead": 593574,
  "RequestAddr": "THE_REDACTED.COM.AU",
  "RequestContentSize": 117,
  "RequestCount": 15918,
  "RequestHost": "THE_REDACTED.COM.AU",
  "RequestMethod": "POST",
  "RequestPath": "/wp-login.php",
  "RequestPort": "-",
  "RequestProtocol": "HTTP/1.1",
  "RequestScheme": "http",
  "RetryAttempts": 0,
  "RouterName": "the_REDACTED-redirect-http@docker",
  "ServiceAddr": "172.19.0.2:80",
  "ServiceName": "traefik-traefik@docker",
  "ServiceURL": {
    "Scheme": "http",
    "Opaque": "",
    "User": null,
    "Host": "172.19.0.2:80",
    "Path": "",
    "RawPath": "",
    "OmitHost": false,
    "ForceQuery": false,
    "RawQuery": "",
    "Fragment": "",
    "RawFragment": ""
  },
  "StartLocal": "2023-12-04T13:43:53.381951984+11:00",
  "entryPointName": "web",
  "level": "info",
  "msg": "",
  "request_Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
  "request_Accept-Encoding": "gzip, deflate",
  "request_Accept-Language": "en-US,en;q=0.5",
  "request_Content-Length": "117",
  "request_Content-Type": "application/x-www-form-urlencoded",
  "request_Cookie": "wordpress_test_cookie=WP Cookie check;",
  "request_Referer": "HTTP://THE_REDACTED.COM.AU/wp-login.php",
  "request_User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:109.0) Gecko/20100101 Firefox/111.0",
  "request_X-Forwarded-Host": "THE_REDACTED.COM.AU",
  "request_X-Forwarded-Port": "80",
  "request_X-Forwarded-Proto": "http",
  "request_X-Forwarded-Server": "d0f1ce6e2cf4",
  "request_X-Real-Ip": "172.19.0.2",
  "time": "2023-12-04T13:44:08+11:00"
}

Config files:

docker-compose.yaml for traefik:

version: '3.7'

services:
  traefik:
    image: traefik:2.10.6
    container_name: traefik
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - $PWD/config:/etc/traefik
      - /var/log/traefik:/var/log/traefik
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.rule=Host(`proxy.REDACTED.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
      - "traefik.http.routers.traefik.entrypoints=web-secure"
      - "traefik.http.routers.traefik.tls.certresolver=default"
      - "traefik.http.routers.traefik.service=api@internal"
      - "traefik.http.routers.traefik.middlewares=auth@docker"
      - "traefik.http.middlewares.auth.basicauth.users=********:******************REDACTED*****************************"
      - "traefik.http.routers.another-redirect-http.rule=Host(`another.com.au`) || Host(`www.another.com.au`)"
      - "traefik.http.routers.another-redirect-http.entrypoints=web"
      - "traefik.http.routers.another-redirect-http.middlewares=another-redirectregex-http"
      - "traefik.http.middlewares.another-redirect-http.ratelimit.average=100"
      - "traefik.http.middlewares.another-redirect-http.ratelimit.burst=50"
      - "traefik.http.middlewares.another-redirect-https.ratelimit.average=100"
      - "traefik.http.middlewares.another-redirect-https.ratelimit.burst=50"
      - "traefik.http.middlewares.another-redirectregex-http.redirectregex.regex=^http://(?:www.)?another.com.au/(.*)"
      - "traefik.http.middlewares.another-redirectregex-http.redirectregex.replacement=http://DOWNSTREAM-SITE-REDACTED.com.au/another/$${1}"
      - "traefik.http.middlewares.another-redirectregex-http.redirectregex.permanent=true"
      - "traefik.http.routers.another-redirect-https.rule=Host(`another.com.au`) || Host(`www.another.com.au`)"
      - "traefik.http.routers.another-redirect-https.entrypoints=web-secure"
      - "traefik.http.routers.another-redirect-https.tls=true"
      - "traefik.http.routers.another-redirect-https.middlewares=another-redirectregex-https"
      - "traefik.http.middlewares.another-redirectregex-https.redirectregex.regex=^https://(?:www.)?another.com.au/(.*)"
      - "traefik.http.middlewares.another-redirectregex-https.redirectregex.replacement=https://DOWNSTREAM-SITE-REDACTED.com.au/another/$${1}"
      - "traefik.http.middlewares.another-redirectregex-https.redirectregex.permanent=true"
      - "traefik.http.routers.the_REDACTED-redirect-http.rule=Host(`the_REDACTED.com.au`) || Host(`www.the_REDACTED.com.au`)"
      - "traefik.http.routers.the_REDACTED-redirect-http.entrypoints=web"
      - "traefik.http.routers.the_REDACTED-redirect-http.middlewares=the_REDACTED-redirectregex-http"
      - "traefik.http.middlewares.the_REDACTED-redirect-http.ratelimit.average=100"
      - "traefik.http.middlewares.the_REDACTED-redirect-http.ratelimit.burst=50"
      - "traefik.http.middlewares.the_REDACTED-redirect-https.ratelimit.average=100"
      - "traefik.http.middlewares.the_REDACTED-redirect-https.ratelimit.burst=50"
      - "traefik.http.middlewares.the_REDACTED-redirectregex-http.redirectregex.regex=^http://(?:www.)?the_REDACTED.com.au/(.*)"
      - "traefik.http.middlewares.the_REDACTED-redirectregex-http.redirectregex.replacement=http://DOWNSTREAM-SITE-REDACTED.com.au/the_REDACTEDilion/$${1}"
      - "traefik.http.middlewares.the_REDACTED-redirectregex-http.redirectregex.permanent=true"
      - "traefik.http.routers.the_REDACTED-redirect-https.rule=Host(`the_REDACTED.com.au`) || Host(`www.the_REDACTED.com.au`)"
      - "traefik.http.routers.the_REDACTED-redirect-https.entrypoints=web-secure"
      - "traefik.http.routers.the_REDACTED-redirect-https.tls=true"
      - "traefik.http.routers.the_REDACTED-redirect-https.middlewares=the_REDACTED-redirectregex-https"
      - "traefik.http.middlewares.the_REDACTED-redirectregex-https.redirectregex.regex=^https://(?:www.)?the_REDACTED.com.au/(.*)"
      - "traefik.http.middlewares.the_REDACTED-redirectregex-https.redirectregex.replacement=https://DOWNSTREAM-SITE-REDACTED.com.au/the_REDACTEDilion/$${1}"
      - "traefik.http.middlewares.the_REDACTED-redirectregex-https.redirectregex.permanent=true"
    networks:
      - proxy
      - default
    ports:
      - "80:80"
      - "443:443"
    restart: always
    logging:
      driver: "json-file"
      options:
        max-size: 10m
        max-file: "10"

networks:
  proxy:
    external:
      name: proxy
  default:
    driver: bridge

traefik.toml:

[log]
  level = "DEBUG"
  filePath = "/var/log/traefik/traefik.log"

[accesslog]
  filePath = "/var/log/traefik/json.log"
  format = "json"
  bufferingSize = 1000 
  [accessLog.fields]
    defaultMode = "keep"
    [accessLog.fields.names]
      StartUTC = "drop"
    [accessLog.fields.headers]
      defaultMode = "keep"
      [accessLog.fields.headers.names]
        "User-Agent" = "keep"
        "Authorization" = "keep"
        "Content-Type" = "keep"
	"X-Forwarded-Host" = "keep"
	"Cookie" = "keep"
  [accessLog.filters]
    statusCodes = ["110-199", "201-299", "400-499", "300-308", "500-511"]
    retryAttempts = true

[providers]
  [providers.docker]
    exposedByDefault = false
    #endpoint = "unix:///var/run/docker.sock"
    endpoint = "tcp://socket-proxy:****"
    network = "proxy"
  [providers.file]
    filename = "/etc/traefik/dynamic.yml"
    
[api]
  dashboard = true

[entryPoints]
  [entryPoints.web]
    address = ":80"
  [entryPoints.web-secure]
    address = ":443"

[certificatesResolvers]
  [certificatesResolvers.default.acme]
    email = "REDACTED@REDACTED"
    storage = "/etc/traefik/ACME/acme.json"
    [certificatesResolvers.default.acme.tlsChallenge]
    
[pilot]
    token = "**************REDACTED******************"

dynamic.yml

tls:
options:
  default:
    minVersion: VersionTLS12
    cipherSuites:
      - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
      - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
      - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
      - TLS_AES_128_GCM_SHA256
      - TLS_AES_256_GCM_SHA384
      - TLS_CHACHA20_POLY1305_SHA256
  curvePreferences:
    - CurveP521
    - CurveP384
  sniStrict: true

http:
  middlewares:
    secHeaders:
      headers:
        browserXssFilter: true
        contentTypeNosniff: true
        frameDeny: true
        #obsolete from 2.5: sslRedirect: true
        #HSTS Configuration
        stsIncludeSubdomains: true
        stsPreload: true
        stsSeconds: 31536000
        customRequestHeaders:
          X-Frame-Options: "SAMEORIGIN"
        customFrameOptionsValue: "SAMEORIGIN"
    https-redirect:
        redirectScheme:
          scheme: https
5

Well, it seems like yet another day in Internet, you are under constant attack.

You have an OriginStatus of 499, so the error comes from your target service, not from Traefik itself.

Make sure that your target service is updated as well, especially if it is WordPress.

6

All done re WordPress. That was one of my first action, to work with the Website peeps to update all plugins, clean-up loose ends and scan for Malware. Nothing found of course...

7

Based on my presented config, is there anything I can do within traefik to prevent those requests or prevent the HTTP 308/499 flood? Is there a header I can add or rate limit I could set? Traefik should be able to prevent these situations, no?!

8

Sure, you can use a middleware like ratelimit (doc).