Forward to other Traefik instance

my setup

I have 2 traefik instances (server1 and server2). One is on the internet and the other is in the LAN. The LAN is connected to server1 with a VPN tunnel. The router in the LAN redirects all traffic for example.com to server2

goal

The goal is to access services on both servers with subdomains of example.com without going through the internet when in the LAN.

what works

  • both server can ping each other with the following IP:
    • server1: 10.0.0.1
    • server2: 192.168.0.1
  • access to s2.example.com from the LAN
  • access to s1.example.com and auth.example.com from the Internet
  • authentication of s2.example.com with auth.example.com through 10.0.0.1:9091
  • there are no errors shown in neither of the 2 traefik dashboards

what doesn't work

When I the following I get a 404 page not found error:

  • access s2.example.com from the Internet
  • access s1.example.com from the LAN

what I tried

This is my current setup:

server1

traefik.yml
entryPoints:
  web:
    address: :80
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
  websecure:
    address: :443
    http:
      tls: {}
      middlewares:
        - security_headers@file

providers:
  docker:
    exposedByDefault: false
    network: proxy

  file:
    directory: /etc/traefik
    watch: true

api:
  dashboard: true
  insecure: false

log:
  level: INFO
traefik_dynamic.yml
http:
  services:
    lan:
      loadBalancer:
        servers:
          - url: "https://192.168.0.1:443"

  routers:
    lan:
      rule: "HostRegexp(`^{subhost:[a-zA-Z0-9-]+}\\.example\\.com$`) && !Header(`X-Forwarded-From`, `lan.example.com`)"
      entrypoints: "websecure"
      service: "lan"
      middlewares: "lan"
      priority: 1

  middlewares:
    lan:
      headers:
        customRequestHeaders:
          X-Forwarded-From: "cloud.example.com"

    security_headers:
      headers:
        customResponseHeaders:
          # X-Robots-Tag prevents webcrawlers like google to do stuff
          # https://www.link-assistant.com/news/google-tags-overview.html#Robots-meta-tags0
          X-Robots-Tag: "none,noarchive,nosnippet,notranslate,noimageindex"
          server: ""
          X-Forwarded-Proto: "https"
        sslProxyHeaders:
          X-Forwarded-Proto: https
        referrerPolicy: "strict-origin-when-cross-origin"
        hostsProxyHeaders:
          - "X-Forwarded-Host"
        customRequestHeaders:
          X-Forwarded-Proto: "https"
        contentTypeNosniff: true
        browserXssFilter: true
        forceSTSHeader: true
        stsIncludeSubdomains: true
        stsSeconds: 63072000
        stsPreload: true

tls:
  certificates:
    - certFile: /etc/traefik/fullchain.pem
      keyFile: /etc/traefik/privkey.pem

server2

traefik.yml
entryPoints:
  web:
    address: :80
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
  websecure:
    address: :443
    http:
      tls: {}
      middlewares:
        - security_headers@file

providers:
  docker:
    exposedByDefault: false
    network: proxy
  file:
    directory: /etc/traefik
    watch: true

api:
  dashboard: true
  insecure: false

log:
  level: INFO
traefik_dynamic.yml
http:
  services:
    cloud:
      loadBalancer:
        servers:
          - url: "https://10.0.0.1:443"
  
  routers:
    cloud:
      rule: "HostRegexp(`^{subhost:[a-zA-Z0-9-]+}//.example//.com$`) && !Header(`X-Forwarded-From`, `cloud.example.com`)"
      entrypoints: "websecure"
      service: "cloud"
      middlewares: "cloud"
      priority: 1

  middlewares:
    cloud:
      headers:
        customRequestHeaders:
          X-Forwarded-From: "lan.example.com"
    authelia:
      forwardauth:
        address: http://10.0.0.1:9091/api/verify?rd=https://auth.example.com
        trustForwardHeader: true
        authResponseHeaders: "Remote-User,Remote-Groups,Remote-Name,Remote-Email"
    security_headers:
      headers:
        customResponseHeaders:
          X-Robots-Tag: "none,noarchive,nosnippet,notranslate,noimageindex"
          server: ""
          X-Forwarded-Proto: "https"
        sslProxyHeaders:
          X-Forwarded-Proto: https
        referrerPolicy: "strict-origin-when-cross-origin"
        hostsProxyHeaders:
          - "X-Forwarded-Host"
        customRequestHeaders:
          X-Forwarded-Proto: "https"
        contentTypeNosniff: true
        browserXssFilter: true
        forceSTSHeader: true
        stsIncludeSubdomains: true
        stsSeconds: 63072000
        stsPreload: true
  
tls:
  certificates:
    - certFile: /etc/traefik/fullchain.pem
      keyFile: /etc/traefik/privkey.pem

Question

How can I make this setup work? Is there might a different easier approach for this?

Enable and check Traefik debug log and Traefik access log in JSON format.

I created simple Traefik examples. When I look at your config, I suggest you get rid of all the headers and middlewares for testing and then add it back, bit by bit.

Server one proxies to 192…, is that just the wrong config?