Hello World
I'm self-hosting MinIO server using Docker w/ expose 9000, 9001 (console) ports. I'm also using Traefik v3 to access MinIO, there is only one tiny issue...
i can't upload large files, what do i mean by large? on 400mb test file, upload always fails ~75%
from traefik backend:
traefik | 2024-05-27T17:09:10Z DBG github.com/traefik/traefik/v3/pkg/server/service/proxy.go:100 > 502 Bad Gateway error="readfrom tcp 10.0.7.50:47894->10.0.7.29:9001: i/o timeout"
to test MinIO, i eliminate Traefik and used MinIO directly, and was able to upload 400MB file/object, which tells me to issue lays somewhere with Traefik and not with MinIO
labels I use with minio container:
labels:
- "traefik.enable=true"
- "traefik.http.routers.minio.rule=Host(`minio.X.X`)"
- "traefik.http.routers.minio.entrypoints=websecure"
- "traefik.http.routers.minio.tls.certresolver=myresolver"
- "traefik.http.middlewares.minio.ipwhitelist.sourcerange=X.X.X.X"
- "traefik.http.routers.minio.middlewares=minio@docker"
- "traefik.http.services.minio.loadbalancer.server.port=9001"
Please advise)
oh, i even tried different software (sftpgo ) and i've got exactly same issue (unable to upload 400mb file)
Share your full Traefik static and dynamic config, and docker-compose.yml.
I'm running Traefik inside of docker' container using docker-compose only
# cat docker-compose.yml
version: "3"
services:
traefik:
image: "${image}"
container_name: "traefik"
# cat docker-compose.override.yml
version: "3"
services:
traefik:
command:
- "--log.level=DEBUG"
- "--api.insecure=true"
- "--providers.docker=true"
- "--providers.docker.exposedbydefault=false"
- "--providers.docker.network=traefik_default"
- "--providers.swarm.endpoint=unix:///var/run/docker.sock"
- "--entrypoints.ssh.address=:2224"
- "--entrypoints.web.address=:80"
- "--entrypoints.websecure.address=:443"
- "--certificatesresolvers.myresolver.acme.httpchallenge=true"
- "--certificatesresolvers.myresolver.acme.httpchallenge.entrypoint=web"
# - "--certificatesresolvers.myresolver.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
- "--certificatesresolvers.myresolver.acme.email=XYZ@gmail.com"
- "--certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json"
- "--entrypoints.web.http.redirections.entrypoint.to=websecure"
- "--entrypoints.web.http.redirections.entrypoint.scheme=https"
- "--ping=true"
healthcheck:
test: ["CMD-SHELL","traefik healthcheck --ping"]
interval: 15s
timeout: 5s
retries: 2
extra_hosts:
- "host.docker.internal:host-gateway"
labels:
- "co.elastic.logs/module=traefik"
- "co.elastic.metrics/module=traefik"
- "co.elastic.metrics/hosts=traefik:8080"
networks:
- default
- elk_default
ports:
- "80:80"
- "443:443"
- "8080:8080"
restart: always
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:ro"
- "./letsencrypt:/letsencrypt"
whoami:
image: "traefik/whoami"
container_name: "whoami"
networks:
- default
labels:
- "traefik.enable=true"
- "traefik.http.routers.whoami.rule=Host(`whoami.dos.kalei.app`)"
- "traefik.http.routers.whoami.entrypoints=websecure"
- "traefik.http.routers.whoami.tls.certresolver=myresolver"
- "traefik.http.middlewares.whoami.ipwhitelist.sourcerange=34.152.26.97, 34.152.33.177"
- "traefik.http.routers.whoami.middlewares=whoami@docker"
networks:
default:
driver: overlay
attachable: true
elk_default:
name: elk_default
external: true
#
That does not match with your Docker networks.
how so? both container inside of same network
# docker network inspect traefik_default | egrep 'Name.*(traefik|minio)\"'
"Name": "traefik",
"Name": "minio",
#
my full minio docker-compose and docker-compose.override
# cat docker-compose.yaml
services:
minio:
command: server /data --console-address ":9001"
expose:
- 9000
- 9001
healthcheck:
test: ["CMD", "mc", "ready", "local"]
interval: 5s
timeout: 5s
retries: 5
image: minio/minio:latest
# cat docker-compose.override.yaml
services:
minio:
container_name: minio
env_file:
- ./.env
image: minio/minio:latest
labels:
- "traefik.enable=true"
- "traefik.http.routers.minio.rule=Host(`FQDN`)"
- "traefik.http.routers.minio.entrypoints=websecure"
- "traefik.http.routers.minio.tls.certresolver=myresolver"
- "traefik.http.middlewares.minio.ipwhitelist.sourcerange=A.B.C.D, E,F,G,H"
- "traefik.http.routers.minio.middlewares=minio@docker"
- "traefik.http.services.minio.loadbalancer.server.port=9001"
networks:
- default
- traefik_default
restart: always
volumes:
- ./data:/data
networks:
default:
name: minio
driver: overlay
attachable: true
traefik_default:
name: traefik_default
external: true
#
traefik_default != default in Traefik compose.
i understand how this may be confusing, however that's default behavior for docker-compose and all of docker' networks has _default in the network' name (there is no network that's called default):
# docker network ls | grep default | grep -v _default
#
using docker network inspect, one can see that's 100% is the case, if bring minio service down, it disappears in traefik_default network, and as soon as i bring it up it's back there) if network' fundamental wouldn't be set right, i wouldn't be able to open UI nor would copy smaller files...
No default needed for simple Traefik example
Are you sure it’s about size, not about time? I think because of some CVEs, both Traefik v2.11 and v3.0 use some new timeouts or have set more stricter values.
alexus
May 28, 2024, 7:06pm
11
right, of course one can always override this behavior with specified network' name
it could be very well be some timeout and not size for sure, do you know which though?)
