https://github.com/alangpierce/sucrase
The real issue isn't JS, but is the lack of profiling (as shown here) combined with wanting to be everything to everyone leading to layers of bloated abstraction.
The dirty secret is, as Sucrase (https://github.com/alangpierce/sucrase) reveals, just that Babel is slow and the codebase really pays homage to the project name. Simply computing the config for transforms takes dozens of milliseconds on my machine as it does a bunch of filesystem reads.
So esbuild/swc etc. aren't gaining some magical compiled language speedup, they're just... not doing the dumb things that Babel is. Not ragging on Babel here, it's a 7 year old project originally written by a guy fresh out of high school. I bet a ground up rewrite of Babel could accomplish the same.