Have you heard of VFIO for Graphics Cards in Linux host with Windows Guest? Haven't tried it yet, but it's my winter project and I'm excited:
I haven't personally done much with VFIO but I have looked at it. From my initial look it seemed as if it would require two gpus, one for the host and one for the guest but it looks like some people have single gpu setups working