Traefik upload error

hi, i have an instance of gitlab on prem behind traefik,
if i try to push to gitlab excluding traefik everything works correctly, however if i try to push with traefik in between, i get the following error:

Pushing to https://gitlab.company.it/developer/apache-superset.git
Enumerating objects: 6367, done.
Counting objects: 100% (6367/6367), done.
Delta compression using up to 8 threads
Compressing objects: 100% (6240/6240), done.
Writing objects: 100% (6365/6365), 148.38 MiB | 19.95 MiB/s, done.
Total 6365 (delta 2014), reused 0 (delta 0), pack-reused 0 (from 0)
POST git-receive-pack (155585598 bytes)
error: RPC failed; HTTP 499 curl 22 The requested URL returned error: 499
send-pack: unexpected disconnect while reading sideband packet
fatal: the remote end hung up unexpectedly
Everything up-to-date

traefik label:

    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.gitlab-mgmt-01.rule=Host(`${GITLAB_HOSTNAME}`)"
      - "traefik.http.routers.gitlab-mgmt-01.service=gitlab-mgmt-01"
      - "traefik.http.routers.gitlab-mgmt-01.entrypoints=websecure"
      - "traefik.http.services.gitlab-mgmt-01.loadbalancer.server.port=80"
      - "traefik.http.routers.gitlab-mgmt-01.tls=true"
      - "traefik.http.services.gitlab-mgmt-01.loadbalancer.passhostheader=true"
      - "traefik.http.routers.gitlab-mgmt-01.middlewares=compresstraefik"
      - "traefik.http.middlewares.compresstraefik.compress=true"

In the traefik logs I see no errors, but I see the error in the access log.

are there any default upload/timeout limits?
how can i increase them via docker labels?

i use version 3.0.0

Thanks

You can enable Traefik access log in JSON format to see if the errors come from target service (OriginStatus) or from Traefik (DownstreamStatus).

Also try to reduce complexity for debug, remove compress middleware.

Hi @bluepuma77 ,
I have removed the compression and tried again, but nothing changes.

i see the status of both as 499, these are the log access for push request.

if i remove traefik and try directly with gitlab, it works.

{
    "ClientAddr": "10.XXX.XXX.0:63491",
    "ClientHost": "10.XXX.XXX.0",
    "ClientPort": "63491",
    "ClientUsername": "-",
    "DownstreamContentSize": 276,
    "DownstreamStatus": 401,
    "Duration": 41063365,
    "OriginContentSize": 276,
    "OriginDuration": 40131267,
    "OriginStatus": 401,
    "Overhead": 932098,
    "RequestAddr": "gitlab.company.it",
    "RequestContentSize": 0,
    "RequestCount": 23328,
    "RequestHost": "gitlab.company.it",
    "RequestMethod": "GET",
    "RequestPath": "/developer/apache-superset.git/info/refs?service=git-receive-pack",
    "RequestPort": "-",
    "RequestProtocol": "HTTP/1.1",
    "RequestScheme": "https",
    "RetryAttempts": 0,
    "RouterName": "gitlab-mgmt-01@docker",
    "ServiceAddr": "172.XXX.XXX.0:80",
    "ServiceName": "gitlab-mgmt-01@docker",
    "ServiceURL": "http://172.XXX.XXX.0:80",
    "StartLocal": "2024-06-03T15:23:21.540295406Z",
    "StartUTC": "2024-06-03T15:23:21.540295406Z",
    "TLSCipher": "TLS_AES_128_GCM_SHA256",
    "TLSVersion": "1.3",
    "entryPointName": "websecure",
    "level": "info",
    "msg": "",
    "time": "2024-06-03T15:23:21Z"
}{
    "ClientAddr": "10.XXX.XXX.0:63491",
    "ClientHost": "10.XXX.XXX.0",
    "ClientPort": "63491",
    "ClientUsername": "-",
    "DownstreamContentSize": 196,
    "DownstreamStatus": 200,
    "Duration": 255961915,
    "OriginContentSize": 196,
    "OriginDuration": 254846106,
    "OriginStatus": 200,
    "Overhead": 1115809,
    "RequestAddr": "gitlab.company.it",
    "RequestContentSize": 0,
    "RequestCount": 23329,
    "RequestHost": "gitlab.company.it",
    "RequestMethod": "GET",
    "RequestPath": "/developer/apache-superset.git/info/refs?service=git-receive-pack",
    "RequestPort": "-",
    "RequestProtocol": "HTTP/1.1",
    "RequestScheme": "https",
    "RetryAttempts": 0,
    "RouterName": "gitlab-mgmt-01@docker",
    "ServiceAddr": "172.XXX.XXX.0:80",
    "ServiceName": "gitlab-mgmt-01@docker",
    "ServiceURL": "http://172.XXX.XXX.0:80",
    "StartLocal": "2024-06-03T15:23:22.104044266Z",
    "StartUTC": "2024-06-03T15:23:22.104044266Z",
    "TLSCipher": "TLS_AES_128_GCM_SHA256",
    "TLSVersion": "1.3",
    "entryPointName": "websecure",
    "level": "info",
    "msg": "",
    "time": "2024-06-03T15:23:22Z"
}{
    "ClientAddr": "10.XXX.XXX.0:63491",
    "ClientHost": "10.XXX.XXX.0",
    "ClientPort": "63491",
    "ClientUsername": "-",
    "DownstreamContentSize": 21,
    "DownstreamStatus": 499,
    "Duration": 60001939189,
    "OriginContentSize": 21,
    "OriginDuration": 60000885898,
    "OriginStatus": 499,
    "Overhead": 1053291,
    "RequestAddr": "gitlab.company.it",
    "RequestContentSize": 119078567,
    "RequestCount": 23330,
    "RequestHost": "gitlab.company.it",
    "RequestMethod": "POST",
    "RequestPath": "/developer/apache-superset.git/git-receive-pack",
    "RequestPort": "-",
    "RequestProtocol": "HTTP/1.1",
    "RequestScheme": "https",
    "RetryAttempts": 0,
    "RouterName": "gitlab-mgmt-01@docker",
    "ServiceAddr": "172.XXX.XXX.0:80",
    "ServiceName": "gitlab-mgmt-01@docker",
    "ServiceURL": "http://172.XXX.XXX.0:80",
    "StartLocal": "2024-06-03T15:23:33.600335009Z",
    "StartUTC": "2024-06-03T15:23:33.600335009Z",
    "TLSCipher": "TLS_AES_128_GCM_SHA256",
    "TLSVersion": "1.3",
    "entryPointName": "websecure",
    "level": "info",
    "msg": "",
    "time": "2024-06-03T15:24:33Z"
}

OriginStatus means it’s coming from your target service, check those logs.

HTTP status code 499
A Nginx HTTP server extension. This code is introduced to log the case when the connection is closed by client while HTTP server is processing its request, making server unable to send the HTTP header back

Is target port 80 correct?

yes, port 80 is correct on gitlab, it doesn't make sense to me that mapping port 80 to 8080 in docker and removing the labels to reach it via docker outside of traefik works.

is it possible cmq to set an upload limit or timeout via traefik labels?

It may be an issue with a security fix that came recently. You can try to extend some timeouts at the entrypoint (static configuration):

entryPoints:
# [...]
  web-secure:
    http:
    transport:
      respondingTimeouts:
        readTimeout: 300s
        idleTimeout: 300s
# [...]

see Traefik EntryPoints Documentation - Traefik
and Problems after Traefik update from 2.11.0 to 2.11.1: Solution inside

hi @wollomatic,
i tried different values but nothing changes, right now it returns me the error message “server returned empty response”

it is also giving me problems with outline wiki, where it closes the connection when uploading, although the server side limits are high.

is this a bug in version 3.0.0?

Thx!

Hi,

problem solved with

transport:
  respondingTimeouts:
    readTimeout: 0
    writeTimeout: 0
    idleTimeout: 0

Thx!

2 Likes