Handling Multiple Customers, Cloudflare, Lets Encrypt, Docker

Traefik Traefik v2
,
1

Hi,
I have set up a few instances of traefik but am looking for some guidance at scale.

For each customer we deploy a docker compose stack with various services. Influx, MQTT, Nodered, Grafana ect. I don't want to get a TLS cert for each service, just a wildcard cert per customer.
For example:
Customer1 - *.customer1.mydomain.com
Customer2 - *.customer2.mydomain.com
Then each sub service such as grafana.customer1.mydomain.com or nodered.customer1.mydomain.com will use the same cert *.customer1.mydomain.com.
This should mean I only need one cert per customer and reduce the likely hood of a let's encrypt rate limit.

I have made an example stack using multiple whoami containers as the services.
I am not sure if each customers tls.domains[0].main=customer.mydomain.com" & tls.domains[0].sans=*.customer.mydomain.com should be done in the docker labels or the dynamic config. The customer base will continue to grow.

I'm also not sure I can get certs for each *.customer.mydomain.com or if I can only get *.mydomain.com
I have seen posts about not getting sub-subdomains but I think that is for ..mydomain.com. Please correct me if I am wrong. If I could do that then I would just have one cert for all services of all customers. I am using

certificatesResolvers:
  cloudflare:
    acme:
      email: ###@###.com
      storage: acme.json
      dnsChallenge:
        provider: cloudflare
        resolvers:
          - "1.1.1.1:53"
          - "1.0.0.1:53"

I am also not sure how the domains array works. For example

.domains[0].main
.domains[0].sans
.domains[1].main
.domains[1].sans

Should I be making a .domains[0,1,2,3].main & sans per customer?

version: '3.9'

networks:
  customer_net

services:

  whoami1:
    image: "traefik/whoami"
    container_name: "whoami1"
    restart: unless-stopped
    expose:
     - 80 # for access inside docker network without traefik
    networks:
       - customer_net
    labels:
    # expose
      - "traefik.enable=true" # Expose container with Traefik
      - "traefik.http.routers.whoami1.rule=Host(`whoami1.customer.mydomain.com`)" # how to access the service
      - "traefik.http.routers.whoami1.entrypoints=https" # access with https, there is a redirect from http to https in traefik.yml
      - "traefik.http.services.whoami1.loadbalancer.server.port=80" # expose container port 80
    # TLS configuration
      - "traefik.http.routers.whoami1.tls=true" # use TLS
      - "traefik.http.routers.whoami1.tls.certresolver=cloudflare" # use cloudflare cert resolver. Defined in traefik.yml
      - "traefik.http.routers.whoami1.tls.domains[0].main=customer.mydomain.com" # get cert for main
      - "traefik.http.routers.whoami1.tls.domains[0].sans=*.customer.mydomain.com" # get cert for sans
    # basic auth
      - "traefik.http.middlewares.whoami1-auth.basicauth.users=USER:HASHEDPW" # 
      - "traefik.http.routers.whoami1.middlewares=whoami1-auth" # apply basic auth login
      
  whoami2:
    image: "traefik/whoami"
    container_name: "whoami2"
    restart: unless-stopped
    expose:
     - 80 # for access inside docker network without traefik
    networks:
       - customer_net
    labels:
    # expose
      - "traefik.enable=true" # Expose container with Traefik
      - "traefik.http.routers.whoami2.rule=Host(`whoami2.customer.mydomain.com`)" # how to access the service
      - "traefik.http.routers.whoami2.entrypoints=https" # access with https, there is a redirect from http to https in traefik.yml
      - "traefik.http.services.whoami2.loadbalancer.server.port=80" # expose container port 80
    # TLS configuration
      - "traefik.http.routers.whoami2.tls=true" # use TLS
      - "traefik.http.routers.whoami2.tls.certresolver=cloudflare" # use cloudflare cert resolver. Defined in traefik.yml
      - "traefik.http.routers.whoami2.tls.domains[0].main=customer.mydomain.com" # get cert for main
      - "traefik.http.routers.whoami2.tls.domains[0].sans=*.customer.mydomain.com" # get cert for sans
    # basic auth
      - "traefik.http.middlewares.whoami2-auth.basicauth.users=USER:HASHEDPW" # 
      - "traefik.http.routers.whoami2.middlewares=whoami2-auth" # apply basic auth login

Thanks for any help

2

In general this is correct:

tls.domains[0].main=customer.mydomain.com
tls.domains[0].sans=*.customer.mydomain.com

You usually also need to have a DNS entry for customer.mydomain.com for LetsEncrypt dnsChallenge to be successful with your DNS provider.

How many customers do you have? If I remember correctly, LetsEncrypt has a limit of 50 different (sub-)domains (or wildcards) per week.

The main limit is Certificates per Registered Domain (50 per week). A registered domain is, generally speaking, the part of the domain you purchased from your domain name registrar. For instance, in the name www.example.com, the registered domain is example.com. In new.blog.example.co.uk, the registered domain is example.co.uk. We use the Public Suffix List to calculate the registered domain. Exceeding the Certificates Per Registered Domain limit is reported with the error message too many certificates already issued, possibly with additional details. (Source)

So if you have more, it could be risky if you need to re-create them all at once. It looks pretty professional with app.tenant.example.com, but maybe app-tenant.example.com is a better solution.

3

Thanks,
I'd seen it was 50 per week too.
I've figured out how to do paths.
So I get a cert for

tls.domains[0].main=customer.mydomain.com
tls.domains[0].sans=*.customer.mydomain.com

then that covers all of my customer.mydomain.com/app

Now I only have 1 cert that covers all my customers and services.
I created a wildcard cert in Cloudflare DNS for *.mydomain.com to my cloud server.

I think I'm going to have to use PathPrefix as some of my services with have customer.mydomain.com/app///_

4

Be aware that most web apps don’t like PathPrefix, as they mostly respond with absolute paths for links, redirects, images and scripts.

This usually only works for web apps which can be configured with some kind of "base path".

5

Sorry slow reply. I later stumbled into this exact issue.
Lucky the services I'm using, grafana, chronograf, nodered have their own config options for adjusting the basepath.