Traefik, Cloudflare Proxy and multiple domains

I think my post might be closely related to Traefik Setup w/ 1 Service and multiple Domains (different TLDs) + SSL / TLS - #5 by clovisd and is also posted on the cloudflare community board at https://community.cloudflare.com/t/multiple-domains-to-same-public-ipv4-managed-by-traefik/647650.


I have a traefik setup behind my public IPv4 that redirects to a number of services. This works perfectly fine for one domain, whether i redirect to a service on the same host or to another service within my LAN. The DNS setup has one proxied A record with the name domain1_com pointing to my public IPv4 and then a number of subdomains as CNAME records pointing to domain1_com.

The problem arises when I add a second domain to my traefik stack. This domain2_com also has a proxied A record pointing to my public IPv4 and currently un-proxied CNAME records for the subdomains pointing to domain2_com. SSL certs are valid and automatically generated by traefik, even the CNAMEs are added automatically and work fine. I want to also enable the proxy on the CNAME records, but as soon as I do this, I get redirected to domain1_com instead of my specific subdomain. As soon as I enable the proxy on the CNAME records, nothing ends up in my traefik logs (probably due to caching of the original domain1_com?).

Pardon my weird formatting, much of the stuff in this text was considered a link… This is my traefik docker-compose service definition:

traefik:
    <<: *common-keys-core  # this just adds a restart policy and secure_opt no-new-privileges:true
    image: traefik:v2.11
    container_name: traefik
    command:
      - --log.level=DEBUG
      - --global.sendAnonymousUsage=false
      - --api.dashboard=true
      - --api=true
      - --api.insecure=false
      - --providers.docker=true
      - --providers.docker.exposedbydefault=false
      - --providers.docker.network=websrv
      - --serversTransport.insecureSkipVerify=true
      - --entrypoints.websecure.address=:80
      - --entrypoints.websecure.address=:443
      # attempts to route pihole dns
      # - --entrypoints.dns.address=:53/tcp
      # - --entrypoints.dns-udp.address=:53/udp
      - --entrypoints.https.http.tls.options=tls-opts@file
      # Allow these IPs to set the X-Forwarded-* headers - Cloudflare IPs: https://www.cloudflare.com/ips/
      - --entrypoints.https.forwardedHeaders.trustedIPs=$CLOUDFLARE_IPS,$LOCAL_IPS
      - --accessLog=true
      - --accessLog.filePath=/logs/access.log
      - --accessLog.bufferingSize=100 # Configuring a buffer of 100 lines
      - --accessLog.filters.statusCodes=204-299,400-499,500-599
      - --providers.file.directory=/rules
      - --providers.file.watch=true
      # - --certificatesResolvers.dns-cloudflare.acme.caServer=https://acme-staging-v02.api.letsencrypt.org/directory 
      - --certificatesresolvers.dns-cloudflare.acme.dnschallenge=true
      - --certificatesresolvers.dns-cloudflare.acme.dnschallenge.provider=cloudflare
      - --certificatesresolvers.dns-cloudflare.acme.dnschallenge.resolvers=1.1.1.1:53,1.0.0.1:53
      - --certificatesresolvers.dns-cloudflare.acme.dnschallenge.delayBeforeCheck=90 # To delay DNS check and reduce LE hitrate
      - --certificatesresolvers.dns-cloudflare.acme.email=$ACME_EMAIL
      - --certificatesresolvers.dns-cloudflare.acme.storage=/letsencrypt/acme.json
    ports:
      - "1423:80"
      - "8443:443"
      - "8080:8080"
      # attempts to route pihole dns
      # - 53:53/udp
      # - 53:53/tcp
    dns:
      - $CLOUDFLAREDNS
      - $PRIMARYDNS
      - $SECONDARYDNS
    volumes:
      - $DATADIR/letsencrypt:/letsencrypt
      - $DATADIR/traefik_rules:/rules
      - $DATADIR/traefik_certs:/certs:ro
      - $LOGDIR:/logs
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      - TZ=$TZ
      - CF_API_EMAIL=$CLOUDFLARE_EMAIL
      - CF_API_KEY=$CLOUDFLARE_API_KEY
      - DOMAINNAME_CLOUD_SERVER # Passing the domain name to the traefik container to be able to use the variable in rules. 
    networks:
      websrv:
        ipv4_address: 10.10.10.254
      isolated:
        ipv4_address: 10.20.30.254
      proxies:
        ipv4_address: 99.99.99.254
    depends_on:
      - cf-ddns
      # - cloudflare-ddns
      - cf-companion
    labels:
      - "traefik.enable=true"
      # HTTP-to-HTTPS Redirect
      - "traefik.http.routers.http-catchall.entrypoints=web"
      - "traefik.http.routers.http-catchall.rule=HostRegexp(`{any:.+}`)"
      - "traefik.http.routers.http-catchall.middlewares=redirect-to-https"
      - "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"
      # HTTP Routers
      - "traefik.http.routers.traefik-rtr.rule=Host(`traefik.$DOMAIN1_COM`)"
      - "traefik.http.routers.traefik-rtr.entrypoints=websecure"
      - "traefik.http.routers.traefik-rtr.tls=true" # Some people had 404s without this
      - "traefik.http.routers.traefik-rtr.tls.certresolver=dns-cloudflare" # Comment out this line after first run of traefik to force the use of wildcard certs
      - "traefik.http.routers.traefik-rtr.service=api@internal"
      ## Services - API
      - "traefik.http.services.traefik-rtr.loadbalancer.server.port=8080"
      ## Middlewares
      - "traefik.http.routers.traefik-rtr.middlewares=middlewares-basic-auth@file,middlewares-rate-limit@file,middlewares-https-redirectscheme@file,middlewares-secure-headers@file,middlewares-compress@file"
      # catchall
      - "traefik.http.routers.http-catchall.entrypoints=websecure"
      - "traefik.http.routers.http-catchall.rule=hostregexp(`{host:.+}`)"
      - "traefik.http.routers.http-catchall.middlewares=middlewares-https-redirectscheme@file"

Enable Traefik access log in JSON format, to be able to distinguish between OriginStatus and DownstreamStatus - to see what is creating the redirect.

Thanks for the suggestion. For the working domain test.example.com it looks like this:

{"ClientAddr":"10.20.30.1:45100","ClientHost":"10.20.30.1","ClientPort":"45100","ClientUsername":"-","DownstreamContentSize":153,"DownstreamStatus":404,"Duration":1123351,"OriginContentSize":153,"OriginDuration":919097,"OriginStatus":404,"Overhead":204254,"RequestAddr":"test.example.com","RequestContentSize":0,"RequestCount":1650,"RequestHost":"test.example.com","RequestMethod":"GET","RequestPath":"/favicon.ico","RequestPort":"-","RequestProtocol":"HTTP/2.0","RequestScheme":"https","RetryAttempts":0,"RouterName":"proxy-test-1-rtr@docker","ServiceAddr":"99.99.99.13:80","ServiceName":"proxy-test-1-svc@docker","ServiceURL":"http://99.99.99.13:80","StartLocal":"2024-05-03T18:58:06.913469604+02:00","StartUTC":"2024-05-03T16:58:06.913469604Z","TLSCipher":"TLS_AES_128_GCM_SHA256","TLSVersion":"1.3","entryPointName":"websecure","level":"info","msg":"","time":"2024-05-03T18:58:06+02:00"}

for the not working domain test.example2.com like this:

{"ClientAddr":"10.20.30.1:39720","ClientHost":"10.20.30.1","ClientPort":"39720","ClientUsername":"-","DownstreamContentSize":153,"DownstreamStatus":404,"Duration":1261373,"OriginContentSize":153,"OriginDuration":1081950,"OriginStatus":404,"Overhead":179423,"RequestAddr":"test.mle.sh","RequestContentSize":0,"RequestCount":1194,"RequestHost":"test.mle.sh","RequestMethod":"GET","RequestPath":"/favicon.ico","RequestPort":"-","RequestProtocol":"HTTP/2.0","RequestScheme":"https","RetryAttempts":0,"RouterName":"proxy-test-2-rtr@docker","ServiceAddr":"99.99.99.14:80","ServiceName":"proxy-test-2-svc@docker","ServiceURL":"http://99.99.99.14:80","StartLocal":"2024-05-03T18:21:36.003116159+02:00","StartUTC":"2024-05-03T16:21:36.003116159Z","TLSCipher":"TLS_AES_128_GCM_SHA256","TLSVersion":"1.3","entryPointName":"websecure","level":"info","msg":"","time":"2024-05-03T18:21:36+02:00"}

There is no redirect, only "not found"

from target service. Traefik seems to be working fine.

Share your full static and dynamic config and all docker-compose.yml.

these are my reverse proxy services, running in the same stack as traefik, and in a shared network:

  proxy-test-1:
    container_name: proxy-test-1
    build:
      context: $DATADIR/proxies/
      dockerfile: Dockerfile
      args:
        PROXY_SERVICE_NAME: proxy-test-1
        PROXY_DOMAIN_NAME: test.${DOMAIN1_COM}
        PROXY_HOST: ${SERVER_RPI5}:1110
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=proxies"
      - "traefik.http.routers.proxy-test-1-rtr.rule=Host(`test.$DOMAIN1_COM`)"
      - "traefik.http.routers.proxy-test-1-rtr.entrypoints=websecure"
      - "traefik.http.routers.proxy-test-1-rtr.tls.certresolver=dns-cloudflare"
      - "traefik.http.services.proxy-test-1-svc.loadbalancer.server.port=80"
    restart: unless-stopped
    networks:
      proxies:
        ipv4_address: 99.99.99.13

  proxy-test-2:
    container_name: proxy-test-2
    build:
      context: $DATADIR/proxies/
      dockerfile: Dockerfile
      args:
        PROXY_SERVICE_NAME: proxy-test-2
        PROXY_DOMAIN_NAME: test.${DOMAIN2_COM}
        PROXY_HOST: ${SERVER_RPI5}:1110
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=proxies"
      - "traefik.http.routers.proxy-test-2-rtr.rule=Host(`test.$DOMAIN2_COM`)"
      - "traefik.http.routers.proxy-test-2-rtr.entrypoints=websecure"
      - "traefik.http.routers.proxy-test-2-rtr.tls.certresolver=dns-cloudflare"
      - "traefik.http.services.proxy-test-2-svc.loadbalancer.server.port=80"
    restart: unless-stopped
    networks:
      proxies:
        ipv4_address: 99.99.99.14

they are built using this Dockerfile:

FROM nginx:alpine
ARG PROXY_SERVICE_NAME
ENV PROXY_SERVICE_NAME $PROXY_SERVICE_NAME
ARG PROXY_DOMAIN_NAME
ENV PROXY_DOMAIN_NAME $PROXY_DOMAIN_NAME
ARG PROXY_HOST
ENV PROXY_HOST $PROXY_HOST

RUN echo "upstream ${PROXY_SERVICE_NAME} {\
    server ${PROXY_HOST};\
}\
server {\
    listen 80;\
    server_name ${PROXY_DOMAIN_NAME};\
    location / {\
        proxy_set_header Host \$host;\
        proxy_set_header X-Real-IP \$remote_addr;\
        proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;\
        proxy_set_header X-Forwarded-Proto \$scheme;\
        proxy_buffering off;\
        proxy_request_buffering off;\
        proxy_http_version 1.1;\
        proxy_intercept_errors on;\
        proxy_pass http://${PROXY_SERVICE_NAME};\
    }\
    access_log /var/log/nginx/access.log;\
    error_log  /var/log/nginx/error.log error;\
}" > /etc/nginx/conf.d/default.conf

CMD ["nginx", "-g", "daemon off;"]

The service running under ${SERVER_RPI5}:1110 is a simple nginx server with default config.

In general the config looks okay, but …

This sounds like a fun IP, bits it’s actually a public and not a private address:

The designated private IP address ranges as specified by the Internet Assigned Numbers Authority (IANA) are:

  1. 10.0.0.0 - 10.255.255.255 (10/8 prefix)
  2. 172.16.0.0 - 172.31.255.255 (172.16/12 prefix)
  3. 192.168.0.0 - 192.168.255.255 (192.168/16 prefix)

Further I recommend to clean up your config, place http-to-https redirect and TLS/certresolver globally on entrypoint, not on every router. Compare to simple Traefik example.

What are you trying to achieve with this complicated setup? You can use a Traefik dynamic config file with loadbalancer.server.url and let Traefik connect to the external services directly. Compare to simple Traefik external example.

Thanks for taking your time trying to solve this, the background info and the suggestions - this cleaned up my config quite a bit, all of the tips are great. I also didn't think about 99.99.99.X being reserved for public IPs - thanks a lot!

However, even with traefik taking care of the redirects via dynamic config, actually private IP addresses, centrally managed redirects / cert resolving, my problem persists.. :confused:

Share your full Traefik static and dynamic config and docker-compose.yml :laughing:

proxies.yml:

http:
  routers:
    rtr-test-1:
      rule: "Host(`test.domain1.com`)"
      middlewares:
        - chain-no-auth
      service: svc-rpi5-test
    rtr-test-2:
      rule: "Host(`test.domain2.com`)"
      middlewares:
        - chain-no-auth
      service: svc-rpi5-test

  services:
    svc-rpi5-test:
      loadBalancer:
        servers:
          - url: "http://192.168.0.60:1110"

.env file:

LOCAL_IPS=127.0.0.1/32,10.0.0.0/8,192.168.0.0/16,172.16.0.0/12
CLOUDFLARE_IPS=173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/13,104.24.0.0/14,172.64.0.0/13,131.0.72.0/22
DOMAINNAME_CLOUD_SERVER=domain1.com
PRIMARYDNS=9.9.9.9
SECONDARYDNS=149.112.112.112
CLOUDFLAREDNS=1.1.1.1

middlewares.yml:

http:
  middlewares:
    middlewares-basic-auth:
      basicAuth:
        usersFile: "/rules/credentials.txt"
        realm: "Traefik 2 Basic Auth"

    middlewares-rate-limit:
      rateLimit:
        average: 100
        burst: 50

    middlewares-https-redirectscheme:
      redirectScheme:
        scheme: https
        permanent: true

    middlewares-secure-headers:
      headers:
        accessControlAllowMethods:
          - GET
          - OPTIONS
          - PUT
        accessControlMaxAge: 100
        hostsProxyHeaders:
          - "X-Forwarded-Host"
        stsSeconds: 63072000
        stsIncludeSubdomains: true
        stsPreload: true
        forceSTSHeader: true
        customFrameOptionsValue: "allow-from https:{{env "DOMAINNAME_CLOUD_SERVER"}}"
        contentTypeNosniff: true
        browserXssFilter: true
        referrerPolicy: "same-origin"
        permissionsPolicy: "camera=(), microphone=(), geolocation=(), payment=(), usb=(), vr=()"
        customResponseHeaders:
          X-Robots-Tag: "none,noarchive,nosnippet,notranslate,noimageindex"
          server: ""

    middlewares-compress:
      compress: {}
    
    nextcloud-middlewares-secure-headers:
      headers:
        accessControlMaxAge: 100
        sslRedirect: true
        stsSeconds: 63072000
        stsIncludeSubdomains: true
        stsPreload: true
        forceSTSHeader: true
        customFrameOptionsValue: "sameorigin" 
        frameDeny: false
        contentTypeNosniff: true
        browserXssFilter: true
        referrerPolicy: "no-referrer"
        featurePolicy: "camera 'none'; geolocation 'none'; microphone 'none'; payment 'none'; usb 'none'; vr 'none';"
        customResponseHeaders:
          X-Robots-Tag: "noindex, nofollow"
          server: ""
 
    nextcloud-redirect:
      redirectRegex:
        permanent: true
        regex: "https://(.*)/.well-known/(?:card|cal)dav"
        replacement: "https://${1}/remote.php/dav/"

middleware-chains.yml:

http:
  middlewares:
    chain-nextcloud:
      chain:
        middlewares:
          - middlewares-rate-limit
          - nextcloud-middlewares-secure-headers
          - nextcloud-redirect

    chain-no-auth:
      chain:
        middlewares:
          - middlewares-rate-limit
          - middlewares-https-redirectscheme
          - middlewares-secure-headers
          # - middlewares-compress

tls-opts.yml:

tls:
  options:
    tls-opts:
      minVersion: VersionTLS12
      cipherSuites:
        - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
        - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
        - TLS_AES_128_GCM_SHA256
        - TLS_AES_256_GCM_SHA384
        - TLS_CHACHA20_POLY1305_SHA256
        - TLS_FALLBACK_SCSV # Client is doing version fallback. See RFC 7507
      curvePreferences:
        - CurveP521
        - CurveP384
      sniStrict: true
  certificates:
    - certFile: /certs/certificate.pem
      keyFile: /certs/priv_key.pem

docker-compose.yml:

version: "3.9"

networks:
  default:
    driver: bridge
  websrv:
    name: websrv
    driver: bridge
    ipam:
      config:
        - subnet: 10.10.10.0/24
  isolated:
    name: isolated
    driver: bridge
    ipam:
      config:
        - subnet: 10.20.30.0/24
  proxies:
    name: proxies
    driver: bridge
    ipam:
      config:
        - subnet: 10.99.99.0/24
  dhcp-tier:
    external: true

x-common-keys-core: &common-keys-core
  networks:
    - websrv
  security_opt:
    - no-new-privileges:true
  restart: always

services:
  traefik:
    <<: *common-keys-core
    image: traefik:v3.0
    container_name: traefik
    command:
      - --log.level=DEBUG
      - --global.sendAnonymousUsage=false
      - --api.dashboard=true
      - --api=true
      - --api.insecure=false
      - --providers.docker=true
      - --providers.docker.exposedbydefault=false
      - --providers.docker.network=websrv
      - --serversTransport.insecureSkipVerify=true
      - --entrypoints.web.address=:80
      - --entrypoints.web.http.redirections.entrypoint.to=websecure
      - --entrypoints.web.http.redirections.entrypoint.scheme=https
      - --entrypoints.websecure.address=:443
      - --entrypoints.websecure.asDefault=true
      - --entrypoints.websecure.http.tls.certresolver=dns-cloudflare
      - --entrypoints.https.http.tls.options=tls-opts@file
      - --entrypoints.https.forwardedHeaders.trustedIPs=$CLOUDFLARE_IPS,$LOCAL_IPS
      - --accessLog=true
      - --accessLog.filePath=/logs/access.json
      - --accessLog.bufferingSize=100 # Configuring a buffer of 100 lines
      - --accessLog.filters.statusCodes=204-299,400-499,500-599
      - --accessLog.format=json
      - --providers.file.directory=/rules
      - --providers.file.watch=true
      - --certificatesresolvers.dns-cloudflare.acme.dnschallenge=true
      - --certificatesresolvers.dns-cloudflare.acme.dnschallenge.provider=cloudflare
      - --certificatesresolvers.dns-cloudflare.acme.dnschallenge.resolvers=1.1.1.1:53,1.0.0.1:53
      - --certificatesresolvers.dns-cloudflare.acme.dnschallenge.delayBeforeCheck=90 # To delay DNS check and reduce LE hitrate
      - --certificatesresolvers.dns-cloudflare.acme.email=$ACME_EMAIL
      - --certificatesresolvers.dns-cloudflare.acme.storage=/letsencrypt/acme.json
    ports:
      - "1423:80"
      - "8443:443"
      - "8080:8080"
    dns:
      - $CLOUDFLAREDNS
      - $PRIMARYDNS
      - $SECONDARYDNS
    volumes:
      - $DATADIR/letsencrypt:/letsencrypt
      - $DATADIR/traefik_rules:/rules
      - $DATADIR/traefik_certs:/certs:ro
      - $LOGDIR:/logs
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      - TZ=$TZ
      - CF_API_EMAIL=$CLOUDFLARE_EMAIL
      - CF_API_KEY=$CLOUDFLARE_API_KEY
      - DOMAINNAME_CLOUD_SERVER # Passing the domain name to the traefik container to be able to use the variable in rules. 
    networks:
      websrv:
        ipv4_address: 10.10.10.254
      isolated:
        ipv4_address: 10.20.30.254
      proxies:
        ipv4_address: 10.99.99.254
    depends_on:
      - cf-ddns
      - cf-companion
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik-rtr.rule=Host(`traefik.domain1.com`)"
      - "traefik.http.routers.traefik-rtr.entrypoints=websecure"
      - "traefik.http.routers.traefik-rtr.service=api@internal"
      - "traefik.http.routers.traefik-rtr.middlewares=middlewares-basic-auth@file,middlewares-rate-limit@file,middlewares-https-redirectscheme@file,middlewares-secure-headers@file"

any news on this? unfortunately my setup is still partially unproxied :confused: