Sadly the remote ssh and a few other MS extensions don't work with VSCodium

That one is closed source. It's kind of oddly designed - it ssh:es in to the remote host to download some binary from MS servers, executes it and then communicates with it using SSH TCP forwarding.

"Some binary" = the same version of VS Code (maybe without Electron and some other parts).

Makes sense as you need to - edit files, run tasks and terminals, and run extensions which access the remote filesystem (eg Jump To Definition taking you to header files installed in system folders) - basically everything you need VS Code to do when you run it locally.

The alternative would be running VS Code as a webserver remotely and the local VS Code Electron just acting like a browser window pointing at your server. It probably wouldn't feel as responsive.