Slow upload speeds at docker swarm

Traefik Traefik v2
1

Hi, Im getting really desperate figuring out, why my uploads through traefik proxy are limited to 60-80Mbit/s on our portal app. I tested the speed also with librespeed container, and while upload is slow, download has no problem to reach 400-500Mbit/s through traefik. What is also interesting, if I do 2 uploads simultaneously, they both can reach 60-80Mbit...

The setup in development is 2 VPS systems in docker swarm - one manager and one worker-node. Traefik runs on the swarm manager node and the target apps run on the swarm worker node.

Prod have the same setup, but 3 swarm managers and 2 worker nodes. Prod have about twice beefier VPS systems as dev, but both cap out around the same upload speeds - this basically rules out some HW resource bottleneck. Also traefik doesnt use more than 25% of CPU during speed tests.

docker swarm labels for libre speed test:

    - traefik.enable=true
    - traefik.http.routers.speedtest.rule=(Host(`some.domain`) && PathPrefix(`/speedtest`) || PathPrefix(`/speedtest`))
    - traefik.http.routers.speedtest.tls=true
    - traefik.http.services.speedtest.loadbalancer.server.port=86
    - traefik.http.routers.speedtest.entryPoints=websecure
    - traefik.http.routers.speedtest.tls.certresolver=myresolver
    - traefik.http.middlewares.speedtest-stripprefix.stripprefix.prefixes=/speedtest
    - traefik.http.routers.speedtest.middlewares=speedtest-stripprefix

Also on dev, I spin up a nginx proxy with self signed certs for testing on the same swarm manager node as traefik, and that has no issues to reach ~500Mbit/s both ways - so that rules out a bottleneck within the swarm network itself...

nginx.conf:

 server {
    listen 88 ssl;
    server_name "" $_;
        ssl_certificate /etc/nginx/nginx.crt;
        ssl_certificate_key /etc/nginx/nginx.key;
    resolver 127.0.0.11 ipv6=off valid=15s;
    set $upstream http://speedtest:86;
    location / {
        proxy_pass $upstream;
    }
 }

Traefik config:

entryPoints:
  web:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: websecure
  websecure:
    address: ":443"
log:
  level: warn
certificatesResolvers:
  myresolver:
    acme:
      email: some@mail.com
      storage: /letsencrypt/letsencrypt.json
      httpChallenge:
        entryPoint: web
providers:
  docker: 
    exposedbydefault: false
    swarmMode: true
    network: traefik

Any idea what could be causing this ?

2

How did you setup the ports in Docker Swarm?

3

For librespeed I have it listening on port 86, to which traefik connects via traefik overlay swarm network. nginx is connecting to it the same way, so it doesnt seems to be network related.

Even tried tcpdump in the traefik container, no retransmissions or resets or anything is visible

version: '3'

services:
  speedtest:
    image: adolfintel/speedtest
    networks:
      - traefik
    environment:
      - MODE=standalone
      - WEBPORT=86
    ports:
      - 86:86
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints: [node.role != manager]
      labels:
        - traefik.enable=true
        - traefik.http.routers.speedtest.rule=(Host(`some.domain`) && PathPrefix(`/speedtest`) || PathPrefix(`/speedtest`))
          #- traefik.http.routers.speedtest.tls=true
        - traefik.http.services.speedtest.loadbalancer.server.port=86
        - traefik.http.routers.speedtest.entryPoints=websecure
        - traefik.http.routers.speedtest.tls.certresolver=myresolver
        - traefik.http.middlewares.speedtest-stripprefix.stripprefix.prefixes=/speedtest
        - traefik.http.routers.speedtest.middlewares=speedtest-stripprefix

networks:
  traefik:
    external: true
    driver_opts:
      com.docker.network.driver.mtu: 1400

version: "3.9"
services:
  traefik:
    image: traefik:latest
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "/mnt/swarmfs/traefik/letsencrypt:/letsencrypt"
      - "/mnt/swarmfs/traefik/config:/etc/traefik/"
      - "/mnt/swarmfs/traefik/log:/var/log"
    deploy:
      replicas: 1
      placement:
        constraints: [node.role == manager]
    networks:
      - traefik

networks:
  traefik:
    external: true
    driver_opts:
      com.docker.network.driver.mtu: 1400
4

Try setting up the ports in host mode:

    ports:
      # listen on host ports without Docker ingress network
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
      - target: 8080
        published: 8080
        protocol: tcp
        mode: host
5

still the same... 522/67 DL/UP Mbit/s