I just serve my ESM modules directly from the filesystem. It works well, builds in literally 0 seconds.
You also don't need a transpiler like Babel if you drop support for IE11. Almost all ES6 features– such as classes, proxies, arrow functions, and async are supported by modern browsers and their older versions. As long as you avoid features like static, private, and ??=, you get instant compile times.
It's not 2015. You don't need a transpiler to write modern JavaScript. As of writing, the latest version of Safari is 14 and the latest version of Chrome is 88. These features fully work in Safari 13 and Chrome 70, which have less than 1% of marketshare.
I would say the primary motivators at this point have shifted from browser compatibility to syntaxes like TypeScript and JSX that will (likely) never be supported natively
I suggest using a single-pass transpiler like Surcase[0] that doesn't translate to IE11-compatible syntax, and just loops through the string once, avoiding generating an AST– making it much faster– removing TypeScript annotations and desugaring JSX.
If you additionally need to support IE11, you can use a development build with Surcase and a production build with a bundler like Snowpack. C/C++ developers have been doing things like this for ages: compiling files as objects during development, and compiling them into one big binary for production.