How to replicate a dynamic config for many domains?

[shot]

I have several routers and middlewares configured for a single domain in my dynamic file provider. Now that everything is working perfectly, what I need to do is replicate that configuration for many (potentially hundreds) of other domains. Not only that, but I should be able to add or remove domains dynamically.

As I understand, there is no way to access a "host" variable (similar to nginx) within the dynamic config. Instead, the working config must be replicated for each domain using Go templating. However, I'm struggling to understand the following:

• Where should the list of domains reside? I've seen posts about using environment variables, but then how would one make those dynamically updatable? Is there another option?

• Would it make any sense for each domain's configuration to be a separate file in the watch directory? Is there a limit to the size of a configuration file or the number of files in the watch directory?

Thanks for any info or insights.

[bluepuma77]

Many options available. How do you want to handle your 100 services?

Are they all Docker services/containers and you want to use Traefik Docker Configuration Discovery?

You want them in one or multiple compose files?

Inside a compose file you can declare repetitive parts. But you can also overload compose with another compose file. Or use ENV vars.

Or rather use 1 or multiple Traefik dynamic configurations loaded from file(s), without discovery?

[shot]

Thanks for the reply, @bluepuma77.

@bluepuma77

Are they all Docker services/containers

No, none of them are. That’s why I asked specifically about the dynamic configuration file provider.

@bluepuma77

Or rather use 1 or multiple Traefik dynamic configurations loaded from file(s)

If I use a separate file for each service, then if I need to tweak the configuration, I’d have to update hundreds of files, right?

Templating seems like the best option from a maintenance standpoint, but I still don’t understand how to add/remove services (just different domains really) in that scenario in an automated/dynamic way.

I guess my situation is similar to this one, except that I would need to be able to update many different domains dynamically. As I understand, environment variables are manually/statically configured, right?

[shot]

Ok, here's a snippet in an attempt to clarify...

http:

   services:

      myService1_svc:
         loadBalancer:
            servers:
               - url: https://xxx.xxx.xxx.xxx

   routers:

      myRouter1_rtr:
         entrypoints: websecure
         rule: Host(`mydomain1.dev`)
         middlewares:
            - myMiddleware1_mw
         service: myService1_svc
         tls: {}

   middlewares:

      myMiddleware1_mw:
         addPrefix:
            prefix: /special1

Basically, I need to replicate the service, router, and middleware for each of many domains. I understand the concept of templating, but I'm just not sure where the template gets its data. I mean, where do the unique domains and server URLs reside? How does the template find them?

[bluepuma77]

Check the Traefik Go Templating section with example.

It is possible to use loops, so maybe you can just declare your domains and target services in arrays.

[shot]

Thanks. Yes, I read the docs (as well as most template-related forum posts) before posting my message here. My main question was how to supply the data used by the template, and how to do so in a dynamic (eventually automated) way.

So I did eventually work out one approach using Go templating dict types, as shown below.

The following template...

{{ $proxies := list

	(dict "Domain" "hostdomain1.com"
  		  "Server" "backendserver1.com")

	(dict "Domain" "hostdomain2.com" 
  		  "Server" "backendserver2.com")
}}

http:
	services:
	{{range $proxies}}
		{{.Domain | replace "." "_"}}_svc:
			loadBalancer:
				servers:
					- url: https://{{.Server}}
	{{end}}

...will result in the following output...

http:
	services:
	
		hostdomain1_com_svc:
			loadBalancer:
				servers:
					- url: https://backendserver1.com
	
		hostdomain2_com_svc:
			loadBalancer:
				servers:
					- url: https://backendserver2.com

Being entirely unfamiliar with Go, it actually took much more time than anticipated to get the syntax right. The following template playground tool was absolutely invaluable in working out the kinks...

• repeatit.io

 
There's nothing like instantaneous real-time feedback when you're experimenting - especially in unfamiliar terrain. Kudos to the devs who put that together!

Anyway, for now, I can simply update the YAML template with relevant data in order to add new services. Yippee!

[bluepuma77]

We built an internal tool to build the compose with the required parameters, then execute docker compose.

Alternatively you could build a service that provides the full config via http to providers.http.

[shot]

Nice. I'll keep that in mind.

As a relevant aside, I discovered that Traefik doesn't properly handle files in the watch directory that aren't of an expected file type - i.e. have a known extension.

I had to create a special .goyaml file type to appease my IDE in order to get proper syntax coloring and maintain my sanity. Well, Traefik would ingest it but not properly assimilate it. It would tell me a middleware was missing. When I simply changed the file extension back to .yml (or .yaml), it would work fine.

Anyway, there was a simple work-around, but it took me a while to figure out what was happening, so maybe someone else can be spared the head banging!

EDIT

Upon closer inspection of the logs, I guess Traefik wasn't attempting to load the .goyaml file after all. I think it threw me at the time, because I was expecting to see an error in the logs to that effect. Anyway, I guess the moral of the story is, use a recognized file extension!