I think writing their own physics engine might be wrong way to go here. I understand that Bullet leaves a lot to be desired, but my instinct is that the complexity from their own engine will leave a lot of edge case bugs that need to be ironed out over time, and that games using their own physics engine will suffer from a lot more quirks in the meantime.

A few years ago, as someone who has worked on physics engines for robotics simulations, I would have agreed with you. But now a lot of people are doing greenfield physics engines for games and having it work out pretty well. There's a ton of established academic and conference literature in the area now and it's not nearly as scary as it used to be.

For example, Horizon: Forbidden West uses a custom physics engine that started out as one of the core dev's fun side projects: https://github.com/jrouwe/JoltPhysics

Physics engines (at least game quality physics engines) are starting to drift in to "solved problem" territory and there's enough literature now that you can get something reasonable going yourself after doing some weekend reading.

Edit to add: Godot has had its own engine available for a long time, so it's not a totally new effort. It's a heavy refactor and a large improvement but the bones for this were laid years ago so some of that technical debt you're describing has already been paid down.