How to forward request to external service with same wildcard cert?

OCram85 · July 28, 2023, 10:05am

Hey folks,
having some trouble getting this scenario to work:

I'm running a traefik as reverse proxy in a docker swarm cluster.
Additionally there is a hproxy pair with an virtual service IP, that forwards all traffic on Layer 4 (tcp).
So the SSL termination is done via traefik.
traefik uses a valid wildcard cert like *.sub1.mydomain.com
Im also using an DNS fallback entry for the subdomain that points to the haproxy service IP.

Now I need to run the same stack again on a different location. Unfortunately the migration phase to this new location takes longer than planned and I tried to migrate single services.

Is there a way to filter the requests and forward it to the new location?

I already tried to use a tcp filter on the HaProxy, but this didn't work because both sites use the same certificat:

tcp-request inspect-delay 5s
tcp-request content accept if { req.ssl_hello_type 1 }
use_backend  NewProd-LB if { req.ssl_sni -i service1.sub1.mydomain.com }

So I also tried to add a new tcp router and service in traefik with the fileprovider based config:

tcp:
  routers:
    code:
      entryPoints:
        - "http"
        - "https"
      rule: HostSNI(`service1.sub1.mydomain.com`)
      service: "code-srv"
      tls:
        passthrough: true
  services:
    code-srv:
      loadBalancer:
        servers:
          - address: "192.168.50.50"

The first request for the complete domain will either land on site a or site b. Then all other following request will be forwarding to the same target, like there is some caching happening.

Did I miss anything?

bluepuma77 · July 28, 2023, 7:18pm

Can you explain again what is going where? Your old site has 1x sub-domain, 1x IP, 2x haproxy, 1x Traefik in Docker Swarm.

Same for new site? Different domain, IP, Swarm?

How do you access new Swarm? H

ow do you see it’s keeping the same target? Are you sure it’s not the browser with request and second request to icon, so next request is on first target again?

OCram85 · July 31, 2023, 7:38am

Let my try to explain

This is the layout of my single site setup which is working fine:

Then I created another stack on a new site:

Now I tried to find any way to forward the request based on http/https queries. So I could migrate my docker services individually.

I hope this was a bit more detailed, but let me answer to your questions:

Both sites running their own docker swarm cluster. traefik runs as a single instance as docker service in each cluster.
There is an active/passive haproxy cluster with keepalived and a bound service IP per site.

New swarm should be accessed through the old site until the migration phase is done. The I would change the DNS Wildcard entry to point at the haproxy service IP on the new site.

I tried multiple private browser sessions with flushed dns anc arp cache on multiple clients while inspecting the haproxy stats (open tcp sessions). Then I wondered if I ran into the same haproxy issue: ACL filters when using mode tcp? - #2 by lukastribus - Help! - HAProxy community

bluepuma77 · July 31, 2023, 9:35am

Share your full Traefik static and dynamic config, and docker-compose.yml if used.

The TLS cert is a custom purchased wildcard or created via LE?

HostSNI() should only work on TLS connections, so probably not on http entrypoint.

loadbalancer.server.address needs a port (doc).

OCram85 · July 31, 2023, 1:00pm

So let me share my common config:

HaProxy Config for master on site A:

global
  log /dev/log daemon
  maxconn 32768
  chroot /var/lib/haproxy
  user haproxy
  group haproxy
  daemon
  stats socket /var/lib/haproxy/stats user haproxy group haproxy mode 0640 level operator
  tune.bufsize 32768
  tune.ssl.default-dh-param 2048
  ssl-default-bind-ciphers ALL:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK:!RC4:!ADH:!LOW@STRENGTH

defaults
  log     global
  mode    http
  option  log-health-checks
  option  log-separate-errors
  option  dontlog-normal
  option  dontlognull
  option  httplog
  option  socket-stats
  retries 3
  option  redispatch
  maxconn 10000
  timeout connect     5s
  timeout client     50s
  timeout server    450s

frontend metrics
  bind *:8404
  option http-use-htx
  http-request use-service prometheus-exporter if { path /metrics }

frontend http_front
  bind 192.168.50.3:80
  stats uri /haproxy?stats
  stats refresh 5s
  stats show-desc MASTER
  stats show-legends
  stats show-node site-a-master
  default_backend http_back

frontend https_front
  bind 192.168.50.3:443
  option tcplog
  mode tcp
  default_backend https_back

backend http_back
  balance roundrobin
  server node1 192.168.100.1:80 check
  server node2 192.168.100.2:80 check
  server node3 192.168.100.3:80 check
  # local nginx with static maintenance page if no backend is available
  server maintenance 127.0.0.1:8081 check backup

backend https_back
  mode tcp
  option ssl-hello-chk
  option tcplog
  balance roundrobin
  server node1 192.168.100.1:443 check
  server node2 192.168.100.2:443 check
  server node3 192.168.100.3:443 check
  server maintenance 127.0.0.1:4443 check backup

traefik config as docker stack / service:

version: "3.8"
services:
  traefik:
    image: traefik:v2.9.8
    ports:
      - "80:80"
      - "443:443"
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.role == manager
      labels:
        - "traefik.enable=true"
        - "traefik.docker.network=traefik-public"
        - "traefik.http.routers.dashboard.rule=Host(`dashboard.mysub.domain.com`)"
        - "traefik.http.routers.dashboard.entrypoints=https"
        - "traefik.http.routers.dashboard.tls=true"
        - "traefik.http.routers.dashboard.service=api@internal"
        - "traefik.http.middlewares.admin-auth.basicauth.usersfile=/traefik/users"
        - "traefik.http.routers.dashboard.middlewares=admin-auth"
        - "traefik.http.services.api.loadbalancer.server.port=8080"

    command:
      - "--providers.docker"
      - "--providers.docker.exposedbydefault=false"
      - "--providers.docker.swarmMode"
      - "--providers.docker.swarmModeRefreshSeconds=8"
      - "--entrypoints.http.address=:80"
      - "--entrypoints.https.address=:443"
      - "--entrypoints.http.http.redirections.entryPoint.to=https"
      - "--entrypoints.http.http.redirections.entryPoint.scheme=https"
      #- "--accesslog"
      - "--log"
      #- "--log.level=DEBUG"
      - "--api"
      - "--providers.file.directory=/traefik/fileprovider"
      - "--providers.file.watch=true"
      - "--pilot.dashboard=false"
      - "--metrics.prometheus=true"
      - "--metrics.prometheus.addrouterslabels=true"

    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - traefik:/traefik
      - cert:/cert

    networks:
      - traefik-public

networks:
  traefik-public:
    external: true

volumes:
  traefik:
  cert:

static ssl config to bind purchased wildcard cert:

tls:
  certificates:
    - certFile: /cert/fullchain.pem
      keyFile: /cert/private.key
      stores:
        - default
  stores:
    default:
      defaultCertificate:
        certFile: /cert/fullchain.pem
        keyFile: /cert/private.key

service example for site A or site B:

version: "3.8"

networks:
  traefik-public:
    external: true

services:
  whoami-siteA:
    image: containous/whoami
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
        delay: 10s
        max_attempts: 3
        window: 120s
      labels:
        - "traefik.enable=true"
        - "traefik.docker.lbswarm=true"
        - "traefik.docker.network=traefik-public"
        - "traefik.http.routers.whoami.rule=Host(`whoami.mysub.domain.com`)"
        - "traefik.http.routers.whoami.entrypoints=https"
        - "traefik.http.routers.whoami.tls=true"
        - "traefik.http.routers.service=whoami-srv"
        - "traefik.http.services.whoami-srv.loadbalancer.server.port=80"
    networks:
      - traefik-public
    
networks:
  traefik-public:
    external: true

example for file based config to forward a service from site A to site B.

This service needs to be running in site b with an config like shown above. There can't be a service withe the same name on site A.

tcp:
  routers:
    whoami-siteB:
      entryPoints:
        - "http"
        - "https"
      rule: HostSNI(`whoami.mysub.domain.com`)
      service: "whoami-siteB-srv"
      tls:
        passthrough: true
  services:
    whoami-site2-srv:
      loadBalancer:
        servers:
          - address: "192.168.55.3"

bluepuma77 · July 31, 2023, 2:29pm

OCram85:

tcp:
  routers:
    whoami-siteB:
      entryPoints:
        - "http"
        - "https"
      rule: HostSNI(`whoami.mysub.domain.com`)
      service: "whoami-siteB-srv"
      tls:
        passthrough: true
  services:
    whoami-site2-srv:
      loadBalancer:
        servers:
          - address: "192.168.55.3"

As stated before:

HostSNI() should only work on TLS connections, so probably not on http entrypoint.
loadbalancer.server.address is missing a port (doc).

OCram85 · August 2, 2023, 9:11am

You're right, that was an copy & paste error. This should be:

tcp:
  routers:
    whoami-site2:
      entryPoints:
        #- "http"
        - "https"
      rule: HostSNI(`whoami-site2.mysub.domain.com`)
      service: "whoami-site2-srv"
      tls:
        passthrough: true
  services:
    whoami-site2-srv:
      loadBalancer:
        servers:
          - address: "192.168.55.3:443"

Sorry I don't get it. Do you mean I should remove the http entrypoint in this tcp route+ service configuration to ensure and test if the filtering/forwarding works?

Update:

Forwarding seems to work if I use curl because there is no open session left after the request:

If I try the same in any browser session, first query wins and will decide on which site the request will be forwarded. This includes all other services defined in traefik:

Chrome session:
- First query to: whoami-site1.***
- Second request to: whoami-site2.***
Another browser or chrome private tab shows the opposite result:

image896×525 24.9 KB

bluepuma77 · August 2, 2023, 12:28pm

Sorry, nothing matches:

In code you use

whoami.mysub.domain.com

service: "whoami-siteB-srv"

services:
    whoami-site2-srv:

In curl it’s site1 and site2.

Topic		Replies	Views
TCP SNI passthrough with "wildcard" routing? Traefik v2 (latest) tcp	7	2662	December 20, 2021
FileProvider: httpService and httprouter: How to specify url route for external file based routes with stripprefix Traefik v2 (latest) docker , file	0	436	May 21, 2020
Configure traefik to use authentication server and forward request to Elasticsearch Traefik v2 (latest) docker-swarm	0	780	July 20, 2021
Reverse-proxying to non-local target Traefik v2 (latest) file	10	1596	October 30, 2020
Question : forwardAuth middleware Traefik v2 (latest) docker-swarm	6	634	December 10, 2019

How to forward request to external service with same wildcard cert?

Update:

Related Topics