I do something similar, albeit a bit more bare-bones. I keep my dotfiles in a dotfiles folder that is a git repo and then just have a simple script to symlink everything.

I have a couple of submodules in there (zplug, pyenv) and a bash script to keep them up to date, and then a few self managed binaries, like fnm. That's about it.

I don't bother trying to manage anything outside of home or deal with secret values by using something like git-secret.

Overall, it works great and has allowed me to keep track of how my core tools are configured without too much hassle.