The problem with "alternative" Wayland compositors is that so much more functionality is pushed into the compositor under the Wayland architecture compared to window managers on X11. So it's a ton more work to write a Wayland compositor than an X11 window manager, and most of it is tedious boilerplate that almost certainly has nothing to do with your motivation for writing a new compositor in the first place. This in turn affords many more opportunities to introduce bugs, missing functionality etc. I trust "weird" compositors much less than I trust "weird" X11 window managers - it's more like using an alternative Xorg than an alternative WM.

"...there is still a lot of boilerplate involved in writing a compositor, much more than for an X11 window manager, namely setting up your own rendering code, registering and storing input devices and screens in your own data structures, passing input events to windows, calculating bounds for bars and other overlays (courtesy of layer-shell) and others. X11 handles all of this for you..."

https://tudorr.ro/blog/technical/2021/01/26/the-wayland-expe...

Several people mentioning wlroots. I can't say I've ever attempted it myself, but if you take a look at the blog post above, the author is very clear on the fact that even with the benefit of wlroots (which they highly praise) writing a Wayland compositor is still a lot more work than writing an X11 window manager (which they have also done, so they are something of an authority). wlroots doesn't abstract everything away, it just gives you a bunch of useful tools.

"With Wayland, you handle everything, even with wlroots."

On the other hand, there is nothing in Wayland that prevents anyone from writing a compositor that splits off some window management tasks to a separate process. Afaik nobody has bothered to do so yet, but if there is demand then it could be done. Or maybe there isn't that much demand for such thing.