Intercepting 409 error and converting it to 500

Traefik Traefik v2 (latest)
1

Issue 10014

When running Traefik in a docker swarm, using labels as configuration - Traefik is intercepting a 409 error and returning a 500 error to the user.

The issue is caused by Inertiajs returning a 409 when it wants the session to redirect. I don't know why it does this, I'm sure they had a great reason - but I'm not going to be able to change it.

If I curl to the nginx server in the container (running Mixpost) I see the 409 and the X-Inertia-Location: header.

If I make the same request to the Traefik server, it passes the request to the Mixpost container, get's the 409 - but returns a 500 error with a plain body of "Internal Server Error"

I'm not sure how to get Traefik to pass the error through without munging it, leaving the extra headers intact also.

The compose for for Traefik is Compose for Traefik on Swarm (forum post) · GitHub

Labels for Mixpost are:

      labels:
        - traefik.enable=true
        - traefik.docker.network=proxy
        - traefik.constraint-label=traefik-public
        #----------------------------------------------- routers for: odoo --------------------------------------------------
        # http
        - traefik.http.routers.${APP_SITE}-http.rule=Host(`${APP_DOMAIN}`, `www.${APP_DOMAIN}`)
        - traefik.http.routers.${APP_SITE}-http.entrypoints=http
        - traefik.http.routers.${APP_SITE}-http.middlewares=servicests
        - traefik.http.routers.${APP_SITE}-http.middlewares=https-redirect
        - traefik.http.routers.${APP_SITE}-http.service=${APP_SITE}
        # https
        - traefik.http.routers.${APP_SITE}-https.rule=Host(`${APP_DOMAIN}`, `www.${APP_DOMAIN}`) && PathPrefix(`/`)
        - traefik.http.routers.${APP_SITE}-https.entrypoints=http3
        - traefik.http.routers.${APP_SITE}-https.service=${APP_SITE}
        - traefik.http.routers.${APP_SITE}-https.tls.certresolver=${APP_SITE_RESOLVER}
        - traefik.http.routers.${APP_SITE}-https.middlewares=gzip,limit,strip-prefix
        #====================================================== services ===========================================================
        - traefik.http.services.${APP_SITE}.loadbalancer.server.port=80
1 Like
2

What does Traefik debug log show?

3

@bluepuma77 Nothing helpful

---
ClientAddr: "REDACTED:51933"
ClientHost: REDACTED
ClientPort: '51933'
ClientUsername: "-"
DownstreamContentSize: 21
DownstreamStatus: 500
Duration: 36063247
OriginContentSize: 21
OriginDuration: 35969876
OriginStatus: 500
Overhead: 93371
RequestAddr: REDACTED
RequestContentSize: 2
RequestCount: 38
RequestHost: REDACTED
RequestMethod: POST
RequestPath: "/mixpost/REDACTED/accounts/add/facebook_page"
RequestPort: "-"
RequestProtocol: HTTP/3.0
RequestScheme: https
RetryAttempts: 0
RouterName: REDACTED-https@docker
ServiceAddr: 172.16.56.21:80
ServiceName: REDACTED@docker
ServiceURL:
  Scheme: http
  Opaque: ''
  User:
  Host: 172.16.56.21:80
  Path: ''
  RawPath: ''
  OmitHost: false
  ForceQuery: false
  RawQuery: ''
  Fragment: ''
  RawFragment: ''
StartLocal: '2023-07-07T09:52:24.REDACTEDZ'
StartUTC: '2023-07-07T09:52:24.REDACTEDZ'
TLSCipher: TLS_AES_128_GCM_SHA256
TLSVersion: '1.3'
downstream_Vary: Accept-Encoding
entryPointName: http3
level: info
msg: ''
origin_Vary: Accept-Encoding
request_Accept: text/html, application/xhtml+xml
request_Accept-Encoding: gzip, deflate, br
request_Accept-Language: en-AU,en;q=0.9
request_Content-Type: application/json
request_Cookie: XSRF-TOKEN=REDACTED;
  mixpost_session=REDACTED;
  remember_web_REDACTED=REDACTED
request_Origin: https://REDACTED
request_Priority: u=3, i
request_Referer: https://REDACTED/mixpost/REDACTED/accounts
request_Sec-Fetch-Dest: empty
request_Sec-Fetch-Mode: cors
request_Sec-Fetch-Site: same-origin
request_User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15
  (KHTML, like Gecko) Version/16.5.1 Safari/605.1.15
request_X-Forwarded-Host: REDACTED
request_X-Forwarded-Port: '443'
request_X-Forwarded-Proto: https
request_X-Forwarded-Server: f63ef805b488
request_X-Inertia: 'true'
request_X-Inertia-Version: 62e9b9b04c86c449d676142e1ffa8f45
request_X-Real-Ip: REDACTED
request_X-Requested-With: XMLHttpRequest
request_X-Xsrf-Token: REDACTED
time: '2023-07-07T09:52:24Z'
4

According to this your target service is responding with 500:

Out of curiosity: do you run your glusterfs inside swarm?

5

@bluepuma77 That's why I've logged the issue - the target is responding with 409, but traefik is transforming to 500.

172.16.1.141 - - [07/Jul/2023:08:50:37 +0000] "POST /mixpost/REDACTED/accounts/add/linkedin_page HTTP/1.1" 409 5 "https://REDACTED/mixpost/REDACTED/accounts" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36 OPR/99.0.0.0"

Re glusterfs - Assuming you mean the servers? No, depending on the environment they are bare metal or dedicated VM's. If you are talking about gluster clients, yes - we make heavy use of them inside swarms.

6

I just tested this, when the target service responds with 409, then Traefik returns a 409.

// server.js
const http = require('http');

const port = process.env.PORT || 3000;
const status = parseInt(process.env.STATUS || '200');

const server = http.createServer((req, res) => {
    res.statusCode = status;
    res.end('Return status code ' + status);
});

server.listen(port, () => {
    console.log(`Server running on port ${port}`);
});

# Dockerfile
FROM node:lts-alpine
WORKDIR /app
COPY server.js .
EXPOSE 3000
CMD [ "node", "server.js" ]
1 Like
7

But I admit with a very simple docker-compose.yml and docker stack deploy:

version: '3'

networks:
  proxy:
    name: proxy
    driver: overlay

volumes:
  traefik-certificates-new:

services:
  traefik:
    image: traefik:v2.10
    hostname: '{{.Node.Hostname}}'
    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
    networks:
      - proxy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - traefik-certificates-new:/certificates
    command:
      - --providers.docker=true
      - --providers.docker.swarmMode=true
      - --providers.docker.exposedByDefault=false
      - --providers.docker.network=proxy
      - --entryPoints.web.address=:80
      - --entryPoints.web.http.redirections.entryPoint.to=websecure
      - --entryPoints.web.http.redirections.entryPoint.scheme=https
      - --entryPoints.websecure.address=:443
      - --entryPoints.websecure.http.tls=true
      - --entryPoints.websecure.http.tls.certResolver=myresolver
      - --api.debug=true
      - --api.dashboard=true
      - --log.level=DEBUG
      - --accesslog=true
      - --certificatesResolvers.myresolver.acme.email=mail@example.com
      - --certificatesResolvers.myresolver.acme.storage=/certificates/acme.json
      - --certificatesresolvers.myresolver.acme.tlschallenge=true
    deploy:
      mode: global
      placement:
        constraints:
          - node.role==manager
      labels:
        - traefik.enable=true
        - traefik.http.routers.api.rule=Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))
        - traefik.http.routers.api.service=api@internal
        - traefik.http.routers.api.middlewares=auth
        - 'traefik.http.middlewares.auth.basicauth.users=test:$$apr1$$H6uskkkW$$IgXLP6ewTrSuBkTrqE8wj/'
        - traefik.http.services.dummy-svc.loadbalancer.server.port=9999

  whoami:
    image: errorweb:latest
    environment:
      - PORT=80
      - STATUS=409
    networks:
      - proxy
    deploy:
      mode: global
      placement:
        constraints:
          - node.role==manager
      labels:
        - traefik.enable=true
        - traefik.http.routers.whoami.entrypoints=websecure
        - traefik.http.routers.whoami.rule=Host(`error.example.com`)
        - traefik.http.services.whoami.loadbalancer.server.port=80
8

Hey @troy,

Could you please provide your full logs in DEBUG mode?

9

The log entry was for traefik in debug mode. I'm not sure what else I can provide?
Log of startup and first request is Traefik Log · GitHub

This is a packet capture of the nginx service to traefik (also updated the issue in Github)

03:58:39.628189 IP (tos 0x0, ttl 64, id 30808, offset 0, flags [DF], proto TCP (6), length 1701)
    48423c2a588f.8080 > sy3-swarm-def-traefik_traefik.jopc1yerv82m3hd7uyl88a6de.bq2qp5t4z94xtcmqyvo7chpsv.proxy.52656: Flags [P.], cksum 0x60f2 (incorrect -> 0x6261), seq 1:1650, ack 2731, win 501, options [nop,nop,TS val 3209530898 ecr 4187686443], length 1649: HTTP, length: 1649
        HTTP/1.1 409 Conflict
        Server: nginx
        Content-Type: text/html; charset=UTF-8
        Transfer-Encoding: chunked
        Connection: keep-alive
        X-Powered-By: PHP/8.1.21
        X-Inertia-Location: https://www.facebook.com/v16.0/dialog/oauth?REDACTED
        Cache-Control: no-cache, private
        Date: Tue, 11 Jul 2023 03:58:XX GMT
        Vary: X-Inertia
        Set-Cookie: REDACTED; expires=Tue, 11 Jul 2023 05:58:XX GMT; Max-Age=7200; path=/; samesite=lax
        Set-Cookie: laravel_session=REDACTED; expires=Tue, 11 Jul 2023 05:58:39 GMT; Max-Age=7200; path=/; httponly; samesite=lax
10

Hi,

To make Mixpost work correctly in Docker with a reverse proxy and handle HTTPS to HTTP requests properly, you can modify the nginx configuration. Here's an example of how you can do it:

First, make sure you have the necessary volumes defined in your Docker Compose file:

volumes:
  - mixpost-storage:/var/www/html/storage/app
  - mixpost-logs:/var/www/html/storage/logs
  - ./conf/nginx.conf:/etc/nginx/sites-enabled/default

Next, create the nginx.conf file in the ./conf directory with the following content:

map $http_x_forwarded_proto $fcgi_https {
    default off;
    https on;
}

server {
    listen 80;
    root /var/www/html/public;
    index index.php index.html;
    client_max_body_size 200M;

    location / {
        try_files $uri /index.php?$args;
    }

    location ~ \.php$ {
        try_files $uri =404;
        fastcgi_split_path_info ^(.+\.php)(/.+)$;
        fastcgi_pass unix:/var/run/php/php8.2-fpm.sock;

        fastcgi_index index.php;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_param PATH_INFO $fastcgi_path_info;
        fastcgi_param HTTPS $fcgi_https;
        fastcgi_read_timeout 1000;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_buffer_size 128k;
        proxy_buffers 4 256k;
        proxy_busy_buffers_size 256k;
    }
}

Once you've made these changes, restart your Docker containers for the new configuration to take effect. This configuration listens on port 80, handles PHP files with FastCGI, and sets up the necessary proxy headers for your reverse proxy to handle HTTPS to HTTP requests correctly.

Make sure to adapt the configuration to your specific setup and requirements.

11

Thanks @aitorroma
I already have the nginx conf defined as https://github.com/aperim/mixpost/blob/main/rootfs/etc/nginx/http.d/default.conf

I will compare what you have to see where I am going wrong, but nothing is jumping out at the moment

12

I've tried adding the fcgi_https reference as you suggested @aitorroma but still having the same problem. A 409 is being transformed by Traefik into a 500

Reference container file changes

13

This seems to be an issue specific to nginx --> Traefik

I know that nginx is sending the 409 because of the packet capture, but there's something that Traefik doesn't like about it, causing it to change it to a 409.

14

Hello @troy,

In the DEBUG logs you shared, there is this access log line:


{"ClientAddr":"[REDACTED","ClientHost":"REDACTED","ClientPort":"53448","ClientUsername":"-","DownstreamContentSize":21,"DownstreamStatus":500,"Duration":240145263,"OriginContentSize":21,"OriginDuration":239980563,"OriginStatus":500,"Overhead":164700,"RequestAddr":"example.com","RequestContentSize":2,"RequestCount":1,"RequestHost":"example.com","RequestMethod":"POST","RequestPath":"/mixpost/REDACTED/accounts/add/facebook_group","RequestPort":"-","RequestProtocol":"HTTP/2.0","RequestScheme":"https","RetryAttempts":0,"RouterName":"example-com-https@docker","ServiceAddr":"172.16.1.46:8080","ServiceName":"example-com@docker","ServiceURL":{"Scheme":"http","Opaque":"","User":null,"Host":"172.16.1.46:8080","Path":"","RawPath":"","OmitHost":false,"ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""},"StartLocal":"2023-07-11T04:07:12.96863466Z","StartUTC":"2023-07-11T04:07:12.96863466Z","TLSCipher":"TLS_CHACHA20_POLY1305_SHA256","TLSVersion":"1.3","downstream_Alt-Svc":"h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000","downstream_Vary":"Accept-Encoding","entryPointName":"http3","level":"info","msg":"","origin_Alt-Svc":"h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000","origin_Vary":"Accept-Encoding","request_Accept":"text/html, application/xhtml+xml","request_Accept-Encoding":"gzip, deflate, br","request_Accept-Language":"en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7","request_Content-Length":"2","request_Content-Type":"application/json","request_Cookie":"remember_web_REDACTED=REDACTED; XSRF-TOKEN=REDACTED; laravel_session=REDACTED","request_Origin":"https://example.com","request_Referer":"https://example.com/mixpost/REDACTED/accounts","request_Sec-Ch-Ua":"\"Not.A/Brand\";v=\"8\", \"Chromium\";v=\"114\", \"Opera GX\";v=\"100\"","request_Sec-Ch-Ua-Mobile":"?0","request_Sec-Ch-Ua-Platform":"\"macOS\"","request_Sec-Fetch-Dest":"empty","request_Sec-Fetch-Mode":"cors","request_Sec-Fetch-Site":"same-origin","request_User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 OPR/100.0.0.0","request_X-Forwarded-Host":"example.com","request_X-Forwarded-Port":"443","request_X-Forwarded-Proto":"https","request_X-Forwarded-Server":"74f5f0cb1034","request_X-Inertia":"true","request_X-Inertia-Version":"62e9b9b04c86c449d676142e1ffa8f45","request_X-Real-Ip":"REDACTED","request_X-Requested-With":"XMLHttpRequest","request_X-Xsrf-Token":"REDACTED","time":"2023-07-11T04:07:13Z"}

It indicates that the downstream status is a 500 error code.
This is usually the service response status code, but in some cases, it could also be produced by middleware in the chain, or by Traefik itself when an error did happen.

Nonetheless, in your case, there are no error logs in the DEBUG logs you shared, that could explain a 500 status code if Traefik produces it.

Does the issue happen on a route using the LDAP plugin?