The index as a data structure is really starting to show its age, especially as developers adapt Git to monorepo scale. It's really fast for repositories up to a certain size, but big tech organizations grow exponentially, and start to suffer performance issues. At some point, you can't afford to use a data structure that scales with the size of the repo, and have to switch to one that scales with the size of the user's change.

I spent a good chunk of time working around the lack of sparse indexes in libgit2, which produced speedups on the order of 500x for certain operations, because reading and writing the entire index is unnecessary for most users of a monorepo: https://github.com/libgit2/libgit2/issues/6036. I'm excited to see sparse indexes make their way into Git proper.

Shameless plug: I'm working on improving monorepo-scale Git tooling at https://github.com/arxanas/git-branchless, such as with in-memory rebases: https://blog.waleedkhan.name/in-memory-rebases/. Try it out if you work in a Git monorepo.

I'm intrigued by this but the readme could maybe use some work to describe how you envision it being used day-to-day? All the examples seem to be about using it to fix things but I'm not at all clear how it helps enable a new workflow.

Even if it was just a link to a similar tool?