SWC is written in rust, and Babel is written in javascript, so you've proven the OP's point. Vite can be configured to use SWC.
The recent conversation was about Vercel benchmarking a tuned Turbopack+SWC vs. a default Vite+Babel, not really an apples to apples comparison. When Vite is configured to use SWC as a compiler, Vite's HMR gets faster; but it's not the default for compatibility reasons.
Sucrase is faster or really close to SWC (see rhe benchmarks https://github.com/alangpierce/sucrase). Everyone still uses Babel because of the transforms.
And yes, Babel can also be made faster if enough effort is dedicated into it. it's not an impossible feat.