I've tried this and it took ~5 minutes to generate 10 seconds of studio on my 3080. That just doesn't work if you're trying to generate an hour long podcast.
Also there's this which offers a speedup to normal tortoise: https://github.com/152334H/tortoise-tts-fast