X-Forwarded-Proto middleware is not applied for all requests

Traefik Traefik v3 (latest)
,
1

I have traefik forwarding to trino, but it is only working for some clients.

networks:
  proxy:
    name: proxy
    
services:
  traefik:
    image: traefik:v3.1.6
    ports:
      - 443:443
      - 8080:8080 # dashboard port 
    networks:
      - proxy
    platform: linux/amd64
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    command:
      - --api.dashboard=true
      - --api.insecure=true
      - --log.level=DEBUG
      - --accesslog=true
      - --accesslog.format=json
      - --accesslog.fields.names.ClientUsername=keep
      - --accesslog.fields.headers.defaultMode=keep
      - --providers.docker=true
      - --providers.docker.exposedByDefault=false 
      - --providers.docker.network=proxy
      - --entrypoints.websecure.address=:443
      - --entrypoints.websecure.asDefault=true

  trino:
    container_name: trino
    image: vastdataorg/trino-vast:429
    ports:
      - 18080:8080
    networks:
      - proxy
    platform: linux/amd64
    volumes:
      - ./vast.properties:/etc/trino/catalog/vast.properties:ro
      - ./config.properties:/etc/trino/config.properties:ro
      - ./log.properties:/etc/trino/log.properties
    labels:
      - traefik.enable=true
      - traefik.http.routers.mytrino.rule=PathPrefix(`/v1/`)
      - traefik.http.routers.mytrino.entrypoints=websecure
      - traefik.http.routers.mytrino.tls=true
      - traefik.http.routers.mytrino.middlewares=trino-headers@docker
      - traefik.http.middlewares.trino-headers.headers.customRequestHeaders.X-Forwarded-Proto=https

Client 1: Apache Superset, X-Forwarded-Proto is NOT getting set

traefik-1  | {
  "ClientAddr": "172.22.0.1:38230",
  "ClientHost": "172.22.0.1",
  "ClientPort": "38230",
  "ClientUsername": "-",
  "DownstreamContentSize": 19,
  "DownstreamStatus": 404,
  "Duration": 1721738,
  "GzipRatio": 0,
  "OriginContentSize": 0,
  "OriginDuration": 0,
  "OriginStatus": 0,
  "Overhead": 1721738,
  "RequestAddr": "192.168.75.89:443",
  "RequestContentSize": 0,
  "RequestCount": 1,
  "RequestHost": "192.168.75.89",
  "RequestMethod": "POST",
  "RequestPath": "/v1/statement",
  "RequestPort": "443",
  "RequestProtocol": "HTTP/1.1",
  "RequestScheme": "http",
  "RetryAttempts": 0,
  "StartLocal": "2024-10-21T14:19:04.401049853Z",
  "StartUTC": "2024-10-21T14:19:04.401049853Z",
  "downstream_Content-Type": "text/plain; charset=utf-8",
  "downstream_X-Content-Type-Options": "nosniff",
  "entryPointName": "websecure",
  "level": "info",
  "msg": "",
  "request_Accept": "*/*",
  "request_Accept-Encoding": "gzip, deflate, br",
  "request_Content-Length": "8",
  "request_User-Agent": "Trino Python Client/0.330.0",
  "request_X-Forwarded-Host": "192.168.75.89:443",
  "request_X-Forwarded-Port": "443",
  "request_X-Forwarded-Proto": "http",
  "request_X-Forwarded-Server": "2e4e7b9b44e2",
  "request_X-Real-Ip": "172.22.0.1",
  "request_X-Trino-Catalog": "vastdb",
  "request_X-Trino-Client-Capabilities": "PARAMETRIC_DATETIME",
  "request_X-Trino-Schema": "nyc_import",
  "request_X-Trino-Session": "",
  "request_X-Trino-Source": "Apache Superset",
  "request_X-Trino-Time-Zone": "Etc/UTC",
  "request_X-Trino-Transaction-Id": "NONE",
  "request_X-Trino-User": "trino",
  "time": "2024-10-21T14:19:04Z"
}

Note the following AREN'T output like they are with client 2.

traefik-1  | 2024-10-21T14:19:12Z DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:228 > Serving default certificate for request: ""
traefik-1  | 2024-10-21T14:19:12Z DBG github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196 > Service selected by WRR: 24

Client 2: Python trino library client, X-Forwarded-Proto IS getting set

traefik-1  | 2024-10-21T14:19:12Z DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:228 > Serving default certificate for request: ""
traefik-1  | 2024-10-21T14:19:12Z DBG github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196 > Service selected by WRR: 24928be724304dc8
traefik-1  | {
  "ClientAddr": "172.22.0.1:53428",
  "ClientHost": "172.22.0.1",
  "ClientPort": "53428",
  "ClientUsername": "-",
  "DownstreamContentSize": 319,
  "DownstreamStatus": 200,
  "Duration": 223749137,
  "OriginContentSize": 319,
  "OriginDuration": 222660408,
  "OriginStatus": 200,
  "Overhead": 1088729,
  "RequestAddr": "192.168.75.89",
  "RequestContentSize": 28,
  "RequestCount": 2,
  "RequestHost": "192.168.75.89",
  "RequestMethod": "POST",
  "RequestPath": "/v1/statement",
  "RequestPort": "-",
  "RequestProtocol": "HTTP/1.1",
  "RequestScheme": "https",
  "RetryAttempts": 0,
  "RouterName": "mytrino@docker",
  "ServiceAddr": "172.22.0.3:8080",
  "ServiceName": "trino-trino@docker",
  "ServiceURL": "http://172.22.0.3:8080",
  "StartLocal": "2024-10-21T14:19:12.940784484Z",
  "StartUTC": "2024-10-21T14:19:12.940784484Z",
  "TLSCipher": "TLS_AES_128_GCM_SHA256",
  "TLSVersion": "1.3",
  "downstream_Content-Encoding": "gzip",
  "downstream_Content-Length": "319",
  "downstream_Content-Type": "application/json",
  "downstream_Date": "Mon, 21 Oct 2024 14:19:12 GMT",
  "downstream_Vary": "Accept-Encoding",
  "downstream_X-Content-Type-Options": "nosniff",
  "entryPointName": "websecure",
  "level": "info",
  "msg": "",
  "origin_Content-Encoding": "gzip",
  "origin_Content-Length": "319",
  "origin_Content-Type": "application/json",
  "origin_Date": "Mon, 21 Oct 2024 14:19:12 GMT",
  "origin_Vary": "Accept-Encoding",
  "origin_X-Content-Type-Options": "nosniff",
  "request_Accept": "*/*",
  "request_Accept-Encoding": "gzip, deflate",
  "request_Content-Length": "28",
  "request_User-Agent": "Trino Python Client/0.330.0",
  "request_X-Forwarded-Host": "192.168.75.89",
  "request_X-Forwarded-Port": "443",
  "request_X-Forwarded-Proto": "https",
  "request_X-Forwarded-Server": "2e4e7b9b44e2",
  "request_X-Real-Ip": "172.22.0.1",
  "request_X-Trino-Catalog": "vastdb",
  "request_X-Trino-Client-Capabilities": "PARAMETRIC_DATETIME",
  "request_X-Trino-Session": "",
  "request_X-Trino-Source": "trino-sqlalchemy",
  "request_X-Trino-Time-Zone": "Europe/London",
  "request_X-Trino-Transaction-Id": "NONE",
  "request_X-Trino-User": "admin",
  "time": "2024-10-21T14:19:13Z"
}

With Client 1, I can only specify a URL, i.e. trino://trino@192.168.75.89:443/vastdb/nyc_import?verify=false&schema=https

With Client 2, I have a bit more control:

from sqlalchemy import create_engine
from sqlalchemy.sql import text

engine = create_engine(
    'trino://admin@192.168.75.89:443/vastdb',
    connect_args={
        "http_scheme": "https",
        "verify": False
    }
)
connection = engine.connect()
connection.execute(text('use vast."vastdb|nyc_import"'))
rows = connection.execute(text('select count(*) from vastdb.nyc_import.nyc_copy')).fetchall()
for row in rows:
    print(row)

Any ideas why middleware is inconsistently getting applied?

2

I fixed this on Superset's end, it required a parameter:

  • Engine Parameters: {"connect_args":{"http_scheme":"https"}}

I guess because it was sending http by default, traefik couldn't find a router.

3

Both have request_X-Forwarded-Proto.