Traefik Error 500, but container has 200

Hi,

a wget inside the apache container works, but is responded with 500 Internal Server error if accessed through traefik. I would like to solve this with traefik 1.x. Any hints/ideas/pointers?

With traefik as Version v1.7.16 and started through the following docker-compose.yaml:

version: '2'

services:
  traefik:
    restart: always
    image: traefik:alpine
    command: --api --docker --docker.exposedbydefault=false --logLevel=DEBUG
    ports:
      - "90:80"
      - "9001:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - traefik-net

and the apache container as:

version: '2'

services:
    apache:
        restart: always
        build: apache
        container_name: apache
        healthcheck:
          test: "wget -qO- http://localhost >& /dev/null || exit 1"
          interval: 5m
          timeout: 5s
          retries: 1
          start_period: 30s
        cap_add:
          - SYS_ADMIN
        ports:
          - "9080:80"
        networks:
          - traefik_traefik-net
        labels:
          - "traefik.enable=true"
          - "traefik.frontend.rule=PathPrefix:/web;ReplacePathRegex:^/web/(.*) /$$1"
          - "traefik.docker.network=traefik_traefik-net"
          - "traefik.http.services.my-service.loadbalancer.server.port=80"

The apache container is running and healthy.

Inside the apache container a wget http://localhost/00003336.mp3 is successful. file get's received and saved.
traefik response is 500. logfile snippet is:

traefik_1  | time="2022-10-07T19:47:38Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/00003336.mp3\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"identity\"],\"Connection\":[\"Keep-Alive\"],\"User-Agent\":[\"Wget/1.20.3 (linux-gnu)\"],\"X-Replaced-Path\":[\"/web/00003336.mp3\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"docker.fritz.box:90\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"172.20.0.1:40166\",\"RequestURI\":\"/00003336.mp3\",\"TLS\":null}"
traefik_1  | time="2022-10-07T19:47:38Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" ForwardURL="http://172.20.0.8:80" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/00003336.mp3\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"identity\"],\"Connection\":[\"Keep-Alive\"],\"User-Agent\":[\"Wget/1.20.3 (linux-gnu)\"],\"X-Replaced-Path\":[\"/web/00003336.mp3\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"docker.fritz.box:90\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"172.20.0.1:40166\",\"RequestURI\":\"/00003336.mp3\",\"TLS\":null}"
traefik_1  | time="2022-10-07T19:47:38Z" level=debug msg="vulcand/oxy/forward: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"172.20.0.8:80\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"identity\"],\"Connection\":[\"Keep-Alive\"],\"User-Agent\":[\"Wget/1.20.3 (linux-gnu)\"],\"X-Replaced-Path\":[\"/web/00003336.mp3\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"docker.fritz.box:90\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"172.20.0.1:40166\",\"RequestURI\":\"/00003336.mp3\",\"TLS\":null}"
traefik_1  | time="2022-10-07T19:47:38Z" level=debug msg="vulcand/oxy/forward/http: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"172.20.0.8:80\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"identity\"],\"Connection\":[\"Keep-Alive\"],\"User-Agent\":[\"Wget/1.20.3 (linux-gnu)\"],\"X-Replaced-Path\":[\"/web/00003336.mp3\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"docker.fritz.box:90\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"172.20.0.1:40166\",\"RequestURI\":\"/00003336.mp3\",\"TLS\":null}"
traefik_1  | time="2022-10-07T19:47:38Z" level=debug msg="Upstream ResponseWriter of type *pipelining.writerWithoutCloseNotify does not implement http.CloseNotifier. Returning dummy channel."
traefik_1  | time="2022-10-07T19:47:39Z" level=debug msg="'500 Internal Server Error' caused by: net/http: HTTP/1.x transport connection broken: malformed HTTP status code \"SAMEORIGIN\""
traefik_1  | time="2022-10-07T19:47:39Z" level=debug msg="vulcand/oxy/forward/http: Round trip: http://172.20.0.8:80, code: 500, Length: 21, duration: 80.210793ms"
traefik_1  | time="2022-10-07T19:47:39Z" level=debug msg="vulcand/oxy/forward/http: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"172.20.0.8:80\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"identity\"],\"Connection\":[\"Keep-Alive\"],\"User-Agent\":[\"Wget/1.20.3 (linux-gnu)\"],\"X-Replaced-Path\":[\"/web/00003336.mp3\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"docker.fritz.box:90\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"172.20.0.1:40166\",\"RequestURI\":\"/00003336.mp3\",\"TLS\":null}"
traefik_1  | time="2022-10-07T19:47:39Z" level=debug msg="vulcand/oxy/forward: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"172.20.0.8:80\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"identity\"],\"Connection\":[\"Keep-Alive\"],\"User-Agent\":[\"Wget/1.20.3 (linux-gnu)\"],\"X-Replaced-Path\":[\"/web/00003336.mp3\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"docker.fritz.box:90\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"172.20.0.1:40166\",\"RequestURI\":\"/00003336.mp3\",\"TLS\":null}"
traefik_1  | time="2022-10-07T19:47:39Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/00003336.mp3\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"identity\"],\"Connection\":[\"Keep-Alive\"],\"User-Agent\":[\"Wget/1.20.3 (linux-gnu)\"],\"X-Replaced-Path\":[\"/web/00003336.mp3\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"docker.fritz.box:90\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"172.20.0.1:40166\",\"RequestURI\":\"/00003336.mp3\",\"TLS\":null}"

There is a http status code of SAMEORIGIN mentioned, which i do not understand. Should be a number...

Heders snooped are as follows:

GET /00003336.mp3 HTTP/1.1
Host: docker.fritz.box:90
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: de-DE,de;q=0.9
Cache-Control: no-cache
Pragma: no-cache
Sec-Gpc: 1
Upgrade-Insecure-Requests: 1
X-Forwarded-For: 172.25.0.1
X-Forwarded-Host: docker.fritz.box:90
X-Forwarded-Port: 90
X-Forwarded-Prefix: /web
X-Forwarded-Proto: http
X-Forwarded-Server: d15f6d9edb27
X-Real-Ip: 172.25.0.1

s: SAMEORIGIN
X-Content-Type-Options: nosniff
Strict-Transport-Security: max-age=31536000; includeSubDomains
Referrer-Policy: strict-origin
Feature-Policy: vibrate 'none'; geolocation 'none'; autoplay 'none'; notifications 'none'; push 'none'; microphone 'none'; speaker 'none'; fullscreen 'none'; midi 'none'; 
Last-Modified: Wed, 31 Mar 2004 18:17:58 GMT
ETag: "592487-3d6f15db90180"
Accept-Ranges: bytes
Content-Length: 5842055
Cache-Control: max-age=86400
Expires: Sun, 09 Oct 2022 13:04:57 GMT
Content-Type: audio/mpeg

here are the binary bytes of the file ...

A bit weird, if i slow down the answer from apache like shown below, everything works. If it is not slowed down, some headers of the answers are lost on their way up ...

In the meantime i tried to convert the setup to traefik2, seems like it has the same oddity ...

Would like to hear any hints on avoiding the slowdown ...

added snippet to apache configuration ...

ExtFilterDefine slowdown mode=output cmd=/bin/cat preservescontentlength
SetOutputFilter slowdown;slowdown;slowdown

When using image: traefik:alpine I would expect your Docker is using latest, which should be v2.

So I don’t think your config in your first post is using Treafik v1.

Your services seem to use different networks. Do you start them differently (compose vs. stack)?

What would be the image name, using a specific tag instead of using the floating latest?
I'm using docker compose for both, seems i have picked the wrong network.

You can use latest if you want v2. Otherwise check Traefik tags on Docker Hub.