Traefik upload error

iali · June 3, 2024, 10:21am

hi, i have an instance of gitlab on prem behind traefik,
if i try to push to gitlab excluding traefik everything works correctly, however if i try to push with traefik in between, i get the following error:

Pushing to https://gitlab.company.it/developer/apache-superset.git
Enumerating objects: 6367, done.
Counting objects: 100% (6367/6367), done.
Delta compression using up to 8 threads
Compressing objects: 100% (6240/6240), done.
Writing objects: 100% (6365/6365), 148.38 MiB | 19.95 MiB/s, done.
Total 6365 (delta 2014), reused 0 (delta 0), pack-reused 0 (from 0)
POST git-receive-pack (155585598 bytes)
error: RPC failed; HTTP 499 curl 22 The requested URL returned error: 499
send-pack: unexpected disconnect while reading sideband packet
fatal: the remote end hung up unexpectedly
Everything up-to-date

traefik label:

    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.gitlab-mgmt-01.rule=Host(`${GITLAB_HOSTNAME}`)"
      - "traefik.http.routers.gitlab-mgmt-01.service=gitlab-mgmt-01"
      - "traefik.http.routers.gitlab-mgmt-01.entrypoints=websecure"
      - "traefik.http.services.gitlab-mgmt-01.loadbalancer.server.port=80"
      - "traefik.http.routers.gitlab-mgmt-01.tls=true"
      - "traefik.http.services.gitlab-mgmt-01.loadbalancer.passhostheader=true"
      - "traefik.http.routers.gitlab-mgmt-01.middlewares=compresstraefik"
      - "traefik.http.middlewares.compresstraefik.compress=true"

In the traefik logs I see no errors, but I see the error in the access log.

are there any default upload/timeout limits?
how can i increase them via docker labels?

i use version 3.0.0

Thanks

bluepuma77 · June 3, 2024, 2:50pm

You can enable Traefik access log in JSON format to see if the errors come from target service (OriginStatus) or from Traefik (DownstreamStatus).

Also try to reduce complexity for debug, remove compress middleware.

iali · June 3, 2024, 3:31pm

Hi @bluepuma77 ,
I have removed the compression and tried again, but nothing changes.

i see the status of both as 499, these are the log access for push request.

if i remove traefik and try directly with gitlab, it works.

{
    "ClientAddr": "10.XXX.XXX.0:63491",
    "ClientHost": "10.XXX.XXX.0",
    "ClientPort": "63491",
    "ClientUsername": "-",
    "DownstreamContentSize": 276,
    "DownstreamStatus": 401,
    "Duration": 41063365,
    "OriginContentSize": 276,
    "OriginDuration": 40131267,
    "OriginStatus": 401,
    "Overhead": 932098,
    "RequestAddr": "gitlab.company.it",
    "RequestContentSize": 0,
    "RequestCount": 23328,
    "RequestHost": "gitlab.company.it",
    "RequestMethod": "GET",
    "RequestPath": "/developer/apache-superset.git/info/refs?service=git-receive-pack",
    "RequestPort": "-",
    "RequestProtocol": "HTTP/1.1",
    "RequestScheme": "https",
    "RetryAttempts": 0,
    "RouterName": "gitlab-mgmt-01@docker",
    "ServiceAddr": "172.XXX.XXX.0:80",
    "ServiceName": "gitlab-mgmt-01@docker",
    "ServiceURL": "http://172.XXX.XXX.0:80",
    "StartLocal": "2024-06-03T15:23:21.540295406Z",
    "StartUTC": "2024-06-03T15:23:21.540295406Z",
    "TLSCipher": "TLS_AES_128_GCM_SHA256",
    "TLSVersion": "1.3",
    "entryPointName": "websecure",
    "level": "info",
    "msg": "",
    "time": "2024-06-03T15:23:21Z"
}{
    "ClientAddr": "10.XXX.XXX.0:63491",
    "ClientHost": "10.XXX.XXX.0",
    "ClientPort": "63491",
    "ClientUsername": "-",
    "DownstreamContentSize": 196,
    "DownstreamStatus": 200,
    "Duration": 255961915,
    "OriginContentSize": 196,
    "OriginDuration": 254846106,
    "OriginStatus": 200,
    "Overhead": 1115809,
    "RequestAddr": "gitlab.company.it",
    "RequestContentSize": 0,
    "RequestCount": 23329,
    "RequestHost": "gitlab.company.it",
    "RequestMethod": "GET",
    "RequestPath": "/developer/apache-superset.git/info/refs?service=git-receive-pack",
    "RequestPort": "-",
    "RequestProtocol": "HTTP/1.1",
    "RequestScheme": "https",
    "RetryAttempts": 0,
    "RouterName": "gitlab-mgmt-01@docker",
    "ServiceAddr": "172.XXX.XXX.0:80",
    "ServiceName": "gitlab-mgmt-01@docker",
    "ServiceURL": "http://172.XXX.XXX.0:80",
    "StartLocal": "2024-06-03T15:23:22.104044266Z",
    "StartUTC": "2024-06-03T15:23:22.104044266Z",
    "TLSCipher": "TLS_AES_128_GCM_SHA256",
    "TLSVersion": "1.3",
    "entryPointName": "websecure",
    "level": "info",
    "msg": "",
    "time": "2024-06-03T15:23:22Z"
}{
    "ClientAddr": "10.XXX.XXX.0:63491",
    "ClientHost": "10.XXX.XXX.0",
    "ClientPort": "63491",
    "ClientUsername": "-",
    "DownstreamContentSize": 21,
    "DownstreamStatus": 499,
    "Duration": 60001939189,
    "OriginContentSize": 21,
    "OriginDuration": 60000885898,
    "OriginStatus": 499,
    "Overhead": 1053291,
    "RequestAddr": "gitlab.company.it",
    "RequestContentSize": 119078567,
    "RequestCount": 23330,
    "RequestHost": "gitlab.company.it",
    "RequestMethod": "POST",
    "RequestPath": "/developer/apache-superset.git/git-receive-pack",
    "RequestPort": "-",
    "RequestProtocol": "HTTP/1.1",
    "RequestScheme": "https",
    "RetryAttempts": 0,
    "RouterName": "gitlab-mgmt-01@docker",
    "ServiceAddr": "172.XXX.XXX.0:80",
    "ServiceName": "gitlab-mgmt-01@docker",
    "ServiceURL": "http://172.XXX.XXX.0:80",
    "StartLocal": "2024-06-03T15:23:33.600335009Z",
    "StartUTC": "2024-06-03T15:23:33.600335009Z",
    "TLSCipher": "TLS_AES_128_GCM_SHA256",
    "TLSVersion": "1.3",
    "entryPointName": "websecure",
    "level": "info",
    "msg": "",
    "time": "2024-06-03T15:24:33Z"
}

bluepuma77 · June 3, 2024, 4:38pm

OriginStatus means it’s coming from your target service, check those logs.

HTTP status code 499
A Nginx HTTP server extension. This code is introduced to log the case when the connection is closed by client while HTTP server is processing its request, making server unable to send the HTTP header back

Is target port 80 correct?

iali · June 3, 2024, 8:43pm

yes, port 80 is correct on gitlab, it doesn't make sense to me that mapping port 80 to 8080 in docker and removing the labels to reach it via docker outside of traefik works.

is it possible cmq to set an upload limit or timeout via traefik labels?

wollomatic · June 4, 2024, 12:05pm

It may be an issue with a security fix that came recently. You can try to extend some timeouts at the entrypoint (static configuration):

entryPoints:
# [...]
  web-secure:
    http:
    transport:
      respondingTimeouts:
        readTimeout: 300s
        idleTimeout: 300s
# [...]

see Traefik EntryPoints Documentation - Traefik
and Problems after Traefik update from 2.11.0 to 2.11.1: Solution inside

iali · June 10, 2024, 1:34pm

hi @wollomatic,
i tried different values but nothing changes, right now it returns me the error message “server returned empty response”

it is also giving me problems with outline wiki, where it closes the connection when uploading, although the server side limits are high.

is this a bug in version 3.0.0?

Thx!

iali · June 13, 2024, 1:29pm

Hi,

problem solved with

transport:
  respondingTimeouts:
    readTimeout: 0
    writeTimeout: 0
    idleTimeout: 0

Thx!

system · June 16, 2024, 1:30pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
An error with gitlab registry behind traefik Traefik v2 file	2	1084	July 5, 2023
Traefik cannot see Gitlab, all networks tags set. Cannot understand issue Traefik v1 docker	1	624	July 21, 2019
Traefik + GitLab Docker = Bad Gateway due to ports issue Traefik v2 docker	0	161	April 28, 2024
Fallback error response while container is restarting Traefik v2 docker	11	1354	November 16, 2023
Http --> https grpc with insecure error Traefik v2 tcp , middleware	1	1043	April 1, 2020

Traefik upload error

Related topics