do you mind sharing which KVM worked well for you?
I also keep switching frequently and looking for something similar
KVM is a bit of a lie since I'm not switching monitor (both plugged in & switching via OSD).
But for toggling over usb stuff this works fine and is cheap
I do the same thing (plug both computers directly into different inputs on the monitor but use a USB switch to switch the mouse & keyboard). If you install this software on both computers, you can eliminate the "switching via OSD" step, because it will watch for the USB connect / disconnect events and initiate the input switch automatically via the DDC connection to the monitors, so it gives you true one-touch KVM switching at the cost of a USB switch.
This works especially well in cases where an actual KVM switch would be expensive or unavailable (in my case, I'm switching three 4K HDR DisplayPort monitors, which would require a very expensive KVM switch).