I've always been afraid to invest in something like a yubikey or even to use the TPM on any devices I own. I don't ever want to depend on "something I have" that can't be backed up or recovered in any way.
As an example, when I started using a password manager last year, I also made sure to start hosting the (encrypted) passwords database publicly (on a web server) so that if I ever lose it for any reason (SSD fails, etc) I'll be able to download it back onto a computer and unlock it with my master password.
If I ever lose my passwords database I'll also lose access to every internet account I've ever made. It would be far too risky to make it rely on any physical possession of mine.
Some people (most people??) would feel safe knowing that it's impossible for anyone to get into their accounts without their yubikey, but I'd just always be afraid of losing the yubikey.
If you're wanting to protect things further you can also also split your backups via a secret sharing scheme (like http://point-at-infinity.org/ssss/) and distribute the fragments to people or places your at least partially trust.