How Vanta's engineering team improved productivity with esbuild

May 19, 2022

As of early 2022, Vanta’s backend consists of Typescript services and lambdas, and our frontend is a React and Typescript-based single-page application. We host all this code in a monorepo. Over time, we experienced build slowdowns due to our organically growing codebase.

Ultimately, we decided to use esbuild to ensure developer edit/refresh cycles stay smooth. In this article, we’ll explore why we made this decision, how we implemented esbuild, and the results that followed.



The challenge

Our codebase has always been structured as a tree of packages. The leaf directories are services that include a common directory and are bundled up by a Dockerfile. For development, we have a docker compose setup that spins up all services, and essentially runs tsc --watch for each service.

Back in the early days, we leveraged symlinks to manage the dependency tree, which we replaced with Typescript Project References to simplify our setup and accelerate builds.

We experienced reasonable performance with this setup for a few years until tsc --watch incremental builds were slowing down. It was taking tens of seconds to re-compile our backend for even the most minimal changes. Our first recourse was to understand the problem using Typescript’s great performance debugging guide. We discovered a few type-heavy packages, like our very own ts-json-validator, and certain usage patterns of mongoose were leading to severe slowdowns. Through a series of casts and manually provided types, we relieved some burden from Typescript's automatic inference and managed to cut build times by >50%. Even then, with a growing engineering team and codebase, we knew that this wouldn’t be a long-term solution, and re-compilation times steadily crept back up.

After another round of Typescript profiling, we noticed an interesting comment on HackerNews that talked about replacing the Typescript compiler with esbuild. After a quick manual prototype, we found that esbuild could re-compile one of our smaller services in under a second, so we decided to investigate whether we could replace all our transpilation with esbuild.

What is esbuild?

esbuild is a fast Javascript bundler written in Go that focuses on correctness and performance. Bundlers traditionally take existing source code and “bundle” it up into a smaller output file so that it can be served faster over the internet, but over the years, bundlers have taken on additional responsibilities, like transpiling from Typescript to Javascript. All we really needed for backend code was that transpilation step, along with a few niceties like source maps for better stack traces in our observability tools.

The strategy

EsModuleInterop

Our biggest roadblock to enabling esbuild globally in our codebase was that we didn’t have the Typescript option esModuleInterop enabled.

Understanding the module ecosystem

Modern programming languages have a module system so developers can separate code with reasonable boundaries into a self-contained package or module and import that code somewhere else. Javascript, famously written in ten days, did not start off with a module system. Over time, one working group came up with commonjs for modules in Node. Sometime after, the community came up with a different module system—ES Modules, or ESM, with dedicated Javascript syntax for imports. The Javascript community is standardizing ES Modules since they’re the in-built system, and have slightly better and stricter semantics. Here’s a good blog on the intricacies and nuances of the two systems.

Difference in syntax

How Typescript fits in

Typescript added import statements before Javascript natively supported import. And Typescript transpiled import * as statements into CommonJS require statements:

This breaks compatibility with the new ESM spec, as CommonJS require has looser semantics than ES6 import * as. For example, require is allowed to return a non-object, whereas import * as is required to return a plain object. This behavior goes against one of their stated design goals to align with ECMAScript proposals.


Since Typescript’s generated code didn’t conform to the spec, they released an option for interoperability called esModuleInterop. They’ve enabled it by default for new projects, and highly recommend that it’s enabled for all projects going forward.  esbuild does not need to break the spec, so its generated code has different semantics than Typescript generated code with esModuleInterop: false. For example, esbuild generates code that assumes imported modules are not callable and are read-only. This is such a common issue that there’s documentation about it, and the recommended path forward is to enable esModuleInterop for codebases to use esbuild.

We needed to take on two kinds of tasks to correctly enable this option: re-do our imports and clean up mocking code.

Re-doing imports

esModuleInterop automatically enables allowSyntheticDefaultImports, so we migrated many of our imports. This wasn’t strictly required, but it was the cleaner end state with this option enabled.

Example:

to

Unfortunately, this led to a few breakages that weren’t caught at compile-time, since this only retained the same behavior for packages that didn’t have pre-existing default exports. We discovered most bugs here through testing and deployment in our staging environment.

Re-doing mocks

We use sinon for our mocks in tests. An example:

This would no-longer work with esModuleInterop because ES modules are not directly assignable. We were essentially modifying code that was imported, which is a code-smell. We decided to remove mocks where it was easy, and export objects that sinon could mutate when it wasn’t, rather than trying to redo the import itself. This was another common issue with a lot of discussion in the community.

The changes here were pretty mechanical as soon as we got a handle on the problem, and it took one PR for each of the handful of modules affected.

These migrations were the bulk of the work required before we could use esbuild. 

Build script

With our migration to esModuleInterop complete, we wrote a bash script that essentially set up nodemon to watch our Typescript files, and re-ran esbuild on changes.

A large issue worth noting here is that esbuild is explicitly not a Typescript compiler replacement, so it doesn’t grok project references. Fortunately, since it’s so fast, we rebuild all running services in parallel when there are any code changes to our common directories, and that’s worked out fine in practice. Another issue is that esbuild doesn’t recognize JSON entry-points, so it doesn’t copy over JSON files into the output directory which can be imported with resolveJsonModule: true in Typescript. So we just copy those over manually through a find and cp.

The rollout

First, since esbuild is not a Typescript compiler, we made it clear to developers that their code might rebuild successfully through esbuild, even if compilation really should have failed due to type errors. Instead, they could use editors to find most compilation errors and run the Typescript compiler on watch mode manually while doing large refactors to catch compilation errors. We also run the Typescript compiler in CI to make sure that no compilation errors get through. Developers seemed to prefer this trade-off.

Second, our team strongly believes that our systems are as consistent as possible in development and production. Therefore, if developers were interacting with esbuild-built code locally, we should ship esbuild-built code to production. To roll this out, we first enabled esbuild for local development behind an opt-in flag. Next, we pushed esbuild-built code to our staging environment, and finally to production. After almost a year in production, we haven’t noticed any issues with the exception of missing source maps which we fixed by tweaking a few flags.

The results

Overall, we’ve been satisfied with our esbuild experience. Looking back, we should have looked into swc a little more, especially since we used Parcel for our front-end code and Parcel 2 uses swc behind the scenes. But it seems easy to switch from one to the other since the bulk of the work was not strictly related to esbuild.

Most developers aren’t aware that we use esbuild behind the scenes, and we count that as a success. On the other hand, we’ve run into more issues around the Typescript server in code editors, mainly around OOMs due to pathological third-party dependencies and organic codebase and dependency growth, so we’ll continue investing on that front.

Credits go to Evan Wallace, the author behind esbuild, for helping push the Javascript ecosystem towards better, faster tooling. If you’re interested in solving tricky problems to make the internet a more secure place, we’re hiring! Check out our open roles here

Learn more about engineering at Vanta 

How Vanta supports a distributed engineering team
3 GraphQL pitfalls and steps to avoid them
9 security tips for startups (and how coding plays a part)

“Vanta's expert team helped analyze our compliance requirements and shared what was needed to complete a SAQ-D. Because of this, we accelerated our timelines, saved hundreds of hours and thousands of dollars in costs.”

Klas Hesselman
Co-founder  |  Flow Networks
Vanta automates security compliance.
Please enter your first name
Please enter your last name
Please enter a valid email address
Please enter a job title
Please enter your company name
Please enter your company website
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.