Interesting! You show a lot of examples where the server is created as a child of your EnvKey process. That kind of suggests you might encounter EnvKey being the PID 1 in a containerized application. Does EnvKey have a good "PID 1 discipline" -- in particular, reaping zombie processes?
Good question. envkey-source catches kill signals and then sends a SIGTERM on to the process it's running. The process then has a few seconds to handle the SIGTERM and clean up before it's forcefully killed with SIGKILL (the amount of time is configurable via an argument).
Does that answer your question?
If your child processes spawn grandchild processes and then exit or otherwise die, and envkey-source is running as PID 1, then envkey-source will become the parent of those orphaned grandchild processes. When those orphans exit, envkey-source must check those process's exit status. Until then, those process IDs can't be reused, and the processes stick around as zombie processes.
In other words, PID 1 is special, because it may have child processes that it never created, and needs to be aware and handle them properly. Otherwise, you can end up leaking zombie processes.
It sounds like envkey-source isn't aware that it may adopt orphaned child processes. Killing them isn't the main issue. Checking their exit statuses is the main issue.
Thanks, I'll have to look into this more deeply. Currently cleanup is being left to the watched process, but it sounds like more rigorous monitoring of grandchild processes is needed.