Why exactly do I still need a middleman in 2023 to talk to someone else's computer? Is NAT the only reason?

Also, why exactly did we introduce IPv6 again? Everything today is NAT-within-NAT-within-NAT (much of it using IPv4), and almost nobody has a publicly routable IP address. Was the whole transition just a massive waste of effort?

You don't.

https://github.com/miroslavpejic85/mirotalk

It's even faster for 1 on 1 conversations, but as others said, if there are too much participants it will be slower.