Keyloggers are trivial to do in userspace Linux via LD_PRELOAD attacks[0], and typically your user account has permission to read ~/.ssh/id_rsa.
People mention $HOME access. This is something that we're trying to solve with Flatpack: filesystem access should be sandboxed by default. But that requires coordination with desktop environments like Gnome, otherwise everyone just grants programs anything they want because the UX is bad.
And then on top of that we have X11, which is its own mess, and we're trying to address that with Wayland. But Wayland isn't perfect yet for desktop recording, and there's not a ton of effort from software like Emacs to get off of X and onto Wayland because of "what's the point?" arguments. So Flatpack becomes a lot less valuable because X11 keylogging is so easy.
Then we have just flat-out bad user security, where people are setting up sudo without a password. So process isolation becomes a lot less valuable because programs can just manipulate the raw filesystem.
And then we have Spectre/Meltdown leaking passwords, but who cares because "people don't set passwords anyway."?
And whenever a group of people get together and propose any fixes in isolation, there is inevitably someone in the Linux community who will stand up and say, "Look, Wayland is pointless because someone wrote a keylogger[0]. Why are we spending any time fixing this stuff?"
Imagine you are on a boat with 10 holes in the bottom, all of them leaking water. If you want to fix that problem, there is inevitably going to be a period where 5 of the holes are patched and 5 of them aren't. And if you get to that point and start re-opening the holes that did get patched, it's going to be very hard to make any more progress.
https://github.com/Aishou/wayland-keylogger
At present Linux desktops aren't very secure against user installed malicious software. It is however fortunate that most software is installed from curated repos.
It's not clear that just switching to wayland is worth much at this point in time.