Hi,
I am developing my custom middleware for a while. The middleware functionality needs some workers and reporters which should be implemented in a go routine.
But there's a problem. Every time a configuration changes and traefik loads the new configuration, it creates middlewares again, and this causes the problem that go routines in the previous middlewares are orphaned!
For example if I create 10 goroutines in my middleware, when I reload dynamic configs 5 times, there would be 50 goroutines!! which should no longer be active and should be terminated.
But as far as I saw, there's nothing in middleware-level that can check if the configs are reloaded or not.
I saw stopChan from Server struct, but we don't have access to that in a middleware.
I also tried to work with the context that a middleware receives when created, but that didn't help too.
So I want a way to terminate go routines of destroyed middlewares.
I'd appreciate if someone could help me with that.
Could you give more information about the purpose of your middleware?
What kind of middleware are you trying to do ?
Maybe you can provide the repo with the code your middleware.
Unfortunately my repo is in a private registry so I can't provide you with exact code. But I can show what I did in the place of changes.
So I created a brand new package called producer. In it, there's a single exported function:
func InitializeWorkerPool(buffer *ProducerBuffer, config *dynamic.Servicelog) {
wd := sync.WaitGroup{}
for i := 0; i < config.NumOfWorkers; i++ {
wd.Add(1)
go producerWorker(&wd, buffer, config)
}
wd.Wait()
wd.Add(1)
go reportDeliveryHandler(&wd, buffer)
wd.Wait()
}
This function is called in the New function of my custom middleware. parameters are a buffer and config, which this middleware has access to.
Inside buffer, there's a channel. The middleware tries to write some data to this channel. The producerWorker go routine is created for example 10 times and these workers will listen constantly to this channel. For every message that is written to this channel by that middleware, one of this workers will receive that and then does its work and then afterwards, they will listen to this channel again. (a simple for loop, for what it's worth :D)
There's also another go routine called reportDeliveryHandler that only reports the status of the buffer at a certain period of time.
So when the dynamic configs are reloaded, that New function of that middleware is called again with new worker poll with new go routines and those previous go routines are orphaned. And there doesn't seem to be any way to realize this change inside that middleware!
@koalado Hi
Honestly I looked into the solutions that we had at that moment and there were two options:
to modify all functions in the server and router so that they pass their routinePool to the middlewares and then use them
add another kind of routinePool (which I called global pool) and then use them where ever we want, but make it empty after each config reload and clean it up on graceful shutdown.
Well If it were up to me, I'd choose method 1, as it's cleaner and better practice. But I also had this in mind that I want to keep my version updated with original traefik. So in this method, we would have a great inconsistency between original version of traefik and our own version, and on each upgrade with traefik, we should have gone through a long list of conflicts to resolve them. Personally I tried it a couple of times and believe me, you don't want to do this! : D
So I proceed to method 2, though it's a bit anti-pattern. But I defined a global pool, in the safe package, and defined it in a way that it's a map[string]*Pool. So:
wherever I need a Pool of goroutines, I create one with a new name and save them there.
whenever the configs are restarted, I simply call its Cleanup method to clean all existing routes up, so if they're worker or something, they're stopped by stop signal and then closed. No more orphaned.
on shutdown, I simply clean up all pools in there, beside the s.routinePool.Cleanup() in server.go/Server.Close(), so no memory leak occurs in this stage nor the others.
As for middlewares, depends on your usage, but for ours for example, I needed a buffer and hundreds of workers. So I gave them all a random generated name, and put all those goroutines there. This way if any other middlewares were going to use this global routine, there would be no conflict, and they're distinguishable.
I hope I elaborated my solutions and reasons correctly : )