Although the tech behind Figma is nothing but impressive I find that the biggest downside to current collaborative tools is that they're expected to be used collaboratively at all times.
I would say that no more than 10% of the time I spend on Figma is for collaborating with other people in short brainstorming sessions, team workshops, etc. The other 90% is spent by myself working and polishing said prototypes, but still having to deal with the loading times, server hiccups, and so on.
An "offline mode" of sorts would also be a silly feature to expect since their entire stack is built around collaboration. Seems like a difficult balance.
Take a look at https://automerge.org/ and the stack those folks are building. You're exactly right that it's a difficult balance (specifically the trick is proving commutativity for the domain-specific data of your application). But automerge (and then https://github.com/inkandswitch/peritext) show it's at least possible. Good stuff.