Alas, many of us still need to support IE11. We alas get a lot of value out of TypeScript, which needs to be transpiled. <1 second with esbuild is pretty good!
Here's my suggestion if you want faster builds:
In development, use an index.html that loads it directly from the filesystem, using a single-pass annotation remover such as https://github.com/alangpierce/sucrase A single-pass transpiler doesn't build an AST, instead just looping over everything once, which will surely make it faster than esbuild.
In production, use a bundler like snowpack.