Have you had any experience building on top of it directly outside of blaze/bazel?

I have - https://github.com/cruise-automation/isopod and more generally with https://github.com/stripe/skycfg