Poor performance when using Traefik as reverse proxy

Traefik Traefik v2
1

I am using Traefik in Docker and seeing a dramatic performance impact in benchmarks. I'm using the cheapest Droplet on Digital Ocean so I'm not expecting excellent performance overall, but in benchmarking I am seeing around 30% performance when going through Traefik compared to connecting directly to my app.

My setup follows the basic Traefik docker-compose example of a Traefik container and a Whoami container.

docker-compose.yml

version: '3.7'

services:
  traefik:
    image: traefik:v2.2
    command:
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
    ports:
      - "80:80"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

  whoami:
    image: containous/whoami
    ports:
      - 8000:80
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami.rule=Host(`mydomain.com`)"
      - "traefik.http.routers.whoami.entrypoints=web"
      - "traefik.port=8000"

I'm setting the whoami port to 8000 to allow me to connect directly to it bypassing Traefik for testing. Connecting to port 80 will instead route through Traefik.

For benchmarking I am using a local docker container running Fedora with wrk. I have repeated the benchmark 3 times for both routes and selected the middle result.

Results for bypassing Traefik:

[root@8d7ccd188fd1 /]# wrk -t20 -c1000 -d60s --latency http://mydomain.com:8000/bench
Running 1m test @ http://mydomain.com:8000/bench
  20 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   117.19ms   97.77ms   1.94s    97.86%
    Req/Sec   449.79    102.31     1.40k    86.52%
  Latency Distribution
     50%  105.28ms
     75%  110.66ms
     90%  125.46ms
     99%  621.08ms
  521423 requests in 1.00m, 62.66MB read
  Socket errors: connect 0, read 94, write 0, timeout 504
Requests/sec:   8676.09
Transfer/sec:      1.04MB

Results using Traefik:

[root@8d7ccd188fd1 /]# /usr/bin/wrk -t20 -c1000 -d60s --latency http://mydomain.com/bench
Running 1m test @ http://mydomain.com/bench
  20 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   363.67ms  125.90ms   1.99s    83.98%
    Req/Sec   137.35     59.59     1.12k    72.39%
  Latency Distribution
     50%  352.11ms
     75%  423.98ms
     90%  482.60ms
     99%  704.71ms
  160486 requests in 1.00m, 15.61MB read
  Socket errors: connect 0, read 92, write 0, timeout 114
Requests/sec:   2670.66
Transfer/sec:    266.02KB