My Yubikey requires physical touch to authorize an SSH connection so it doesn't matter if I forward my agent to malicious servers as their attempts to hijack my socket will result in timeouts from my not authorizing their use with my physical key

If you have a Mac with Touch ID, you can also use secretive, which requires fingerprint authentication:

https://github.com/maxgoedjen/secretive

Extra nice with the new Apple Magic Keyboard with Touch ID.