Share this article
How Vanta's engineering team improved productivity with esbuild
As of early 2022, Vanta’s backend consists of Typescript services and lambdas, and our frontend is a React and Typescript-based single-page application. We host all this code in a monorepo. Over time, we experienced build slowdowns due to our organically growing codebase.
Ultimately, we decided to use esbuild to ensure developer edit/refresh cycles stay smooth. In this article, we’ll explore why we made this decision, how we implemented esbuild, and the results that followed.
The challenge
Our codebase has always been structured as a tree of packages. The leaf directories are services that include a common directory and are bundled up by a Dockerfile. For development, we have a docker compose setup that spins up all services, and essentially runs tsc --watch for each service.
Back in the early days, we leveraged symlinks to manage the dependency tree, which we replaced with Typescript Project References to simplify our setup and accelerate builds.
We experienced reasonable performance with this setup for a few years until tsc --watch incremental builds were slowing down. It was taking tens of seconds to re-compile our backend for even the most minimal changes. Our first recourse was to understand the problem using Typescript’s great performance debugging guide. We discovered a few type-heavy packages, like our very own ts-json-validator, and certain usage patterns of mongoose were leading to severe slowdowns. Through a series of casts and manually provided types, we relieved some burden from Typescript's automatic inference and managed to cut build times by >50%. Even then, with a growing engineering team and codebase, we knew that this wouldn’t be a long-term solution, and re-compilation times steadily crept back up.
After another round of Typescript profiling, we noticed an interesting comment on HackerNews that talked about replacing the Typescript compiler with esbuild. After a quick manual prototype, we found that esbuild could re-compile one of our smaller services in under a second, so we decided to investigate whether we could replace all our transpilation with esbuild.
What is esbuild?
esbuild is a fast Javascript bundler written in Go that focuses on correctness and performance. Bundlers traditionally take existing source code and “bundle” it up into a smaller output file so that it can be served faster over the internet, but over the years, bundlers have taken on additional responsibilities, like transpiling from Typescript to Javascript. All we really needed for backend code was that transpilation step, along with a few niceties like source maps for better stack traces in our observability tools.
The strategy
EsModuleInterop
Our biggest roadblock to enabling esbuild globally in our codebase was that we didn’t have the Typescript option esModuleInterop enabled.
Understanding the module ecosystem
Modern programming languages have a module system so developers can separate code with reasonable boundaries into a self-contained package or module and import that code somewhere else. Javascript, famously written in ten days, did not start off with a module system. Over time, one working group came up with commonjs for modules in Node. Sometime after, the community came up with a different module system—ES Modules, or ESM, with dedicated Javascript syntax for imports. The Javascript community is standardizing ES Modules since they’re the in-built system, and have slightly better and stricter semantics. Here’s a good blog on the intricacies and nuances of the two systems.
Difference in syntax
How Typescript fits in
Typescript added import statements before Javascript natively supported import. And Typescript transpiled import * as statements into CommonJS require statements:
This breaks compatibility with the new ESM spec, as CommonJS require has looser semantics than ES6 import * as. For example, require is allowed to return a non-object, whereas import * as is required to return a plain object. This behavior goes against one of their stated design goals to align with ECMAScript proposals.
Since Typescript’s generated code didn’t conform to the spec, they released an option for interoperability called esModuleInterop. They’ve enabled it by default for new projects, and highly recommend that it’s enabled for all projects going forward. esbuild does not need to break the spec, so its generated code has different semantics than Typescript generated code with esModuleInterop: false. For example, esbuild generates code that assumes imported modules are not callable and are read-only. This is such a common issue that there’s documentation about it, and the recommended path forward is to enable esModuleInterop for codebases to use esbuild.
We needed to take on two kinds of tasks to correctly enable this option: re-do our imports and clean up mocking code.
Re-doing imports
esModuleInterop automatically enables allowSyntheticDefaultImports, so we migrated many of our imports. This wasn’t strictly required, but it was the cleaner end state with this option enabled.
Example:
to
Unfortunately, this led to a few breakages that weren’t caught at compile-time, since this only retained the same behavior for packages that didn’t have pre-existing default exports. We discovered most bugs here through testing and deployment in our staging environment.
Re-doing mocks
We use sinon for our mocks in tests. An example:
This would no-longer work with esModuleInterop because ES modules are not directly assignable. We were essentially modifying code that was imported, which is a code-smell. We decided to remove mocks where it was easy, and export objects that sinon could mutate when it wasn’t, rather than trying to redo the import itself. This was another common issue with a lot of discussion in the community.
The changes here were pretty mechanical as soon as we got a handle on the problem, and it took one PR for each of the handful of modules affected.
These migrations were the bulk of the work required before we could use esbuild.
Build script
With our migration to esModuleInterop complete, we wrote a bash script that essentially set up nodemon to watch our Typescript files, and re-ran esbuild on changes.
A large issue worth noting here is that esbuild is explicitly not a Typescript compiler replacement, so it doesn’t grok project references. Fortunately, since it’s so fast, we rebuild all running services in parallel when there are any code changes to our common directories, and that’s worked out fine in practice. Another issue is that esbuild doesn’t recognize JSON entry-points, so it doesn’t copy over JSON files into the output directory which can be imported with resolveJsonModule: true in Typescript. So we just copy those over manually through a find and cp.
The rollout
First, since esbuild is not a Typescript compiler, we made it clear to developers that their code might rebuild successfully through esbuild, even if compilation really should have failed due to type errors. Instead, they could use editors to find most compilation errors and run the Typescript compiler on watch mode manually while doing large refactors to catch compilation errors. We also run the Typescript compiler in CI to make sure that no compilation errors get through. Developers seemed to prefer this trade-off.
Second, our team strongly believes that our systems are as consistent as possible in development and production. Therefore, if developers were interacting with esbuild-built code locally, we should ship esbuild-built code to production. To roll this out, we first enabled esbuild for local development behind an opt-in flag. Next, we pushed esbuild-built code to our staging environment, and finally to production. After almost a year in production, we haven’t noticed any issues with the exception of missing source maps which we fixed by tweaking a few flags.
The results
Overall, we’ve been satisfied with our esbuild experience. Looking back, we should have looked into swc a little more, especially since we used Parcel for our front-end code and Parcel 2 uses swc behind the scenes. But it seems easy to switch from one to the other since the bulk of the work was not strictly related to esbuild.
Most developers aren’t aware that we use esbuild behind the scenes, and we count that as a success. On the other hand, we’ve run into more issues around the Typescript server in code editors, mainly around OOMs due to pathological third-party dependencies and organic codebase and dependency growth, so we’ll continue investing on that front.
Credits go to Evan Wallace, the author behind esbuild, for helping push the Javascript ecosystem towards better, faster tooling. If you’re interested in solving tricky problems to make the internet a more secure place, we’re hiring! Check out our open roles here.
Learn more about engineering at Vanta
How Vanta supports a distributed engineering team
3 GraphQL pitfalls and steps to avoid them
9 security tips for startups (and how coding plays a part)
No items found.
FEATURED VANTA RESOURCE
The ultimate guide to scaling your compliance program
Learn how to scale, manage, and optimize alongside your business goals.
PCI Compliance Selection Guide
Determine Your PCI Compliance Level
If your organization processes, stores, or transmits cardholder data, you must comply with the Payment Card Industry Data Security Standard (PCI DSS), a global mandate created by major credit card companies. Compliance is mandatory for any business that accepts credit card payments.
When establishing strategies for implementing and maintaining PCI compliance, your organization needs to understand what constitutes a Merchant or Service Provider, and whether a Self Assessment Questionnaire (SAQ) or Report on Compliance (ROC) is most applicable to your business.
Answer a few short questions and we’ll help identify your compliance level.
1
2
3
4
!
👍
Does your business offer services to customers who are interested in your level of PCI compliance?
Identify your PCI SAQ or ROC level
The PCI Security Standards Council has established the below criteria for Merchant and Service Provider validation. Use these descriptions to help determine the SAQ or ROC that best applies to your organization.
Good news! Vanta supports all of the following compliance levels:
SAQ A
A SAQ A is required for Merchants that do not require the physical presence of a credit card (like an eCommerce, mail, or telephone purchase). This means that the Merchant’s business has fully outsourced all cardholder data processing to PCI DSS compliant third party Service Providers, with no electronic storage, processing, or transmission of any cardholder data on the Merchant’s system or premises.
Get PCI DSS certified
SAQ A-EP
A SAQ A-EP is similar to a SAQ A, but is a requirement for Merchants that don't receive cardholder data, but control how cardholder data is redirected to a PCI DSS validated third-party payment processor.
Learn more about eCommerce PCI
SAQ D
for service providers
A SAQ D includes over 200 requirements and covers the entirety of PCI DSS compliance. If you are a Service Provider, a SAQ D is the only SAQ you’re eligible to complete.
Use our PCI checklist
ROC
Level 1 for service providers
A Report on Compliance (ROC) is an annual assessment that determines your organization’s ability to protect cardholder data. If you’re a Merchant that processes over six million transactions annually or a Service Provider that processes more than 300,000 transactions annually, your organization is responsible for both a ROC and an Attestation of Compliance (AOC).
Automate your ROC and AOC
Download this checklist for easy reference
Questions?
Learn more about how Vanta can help. You can also find information on PCI compliance levels at the PCI Security Standards Council website or by contacting your payment processing partner.
Related resources
The compliance news you need. Delivered securely to your inbox.
Subject to Vanta's Privacy Policy, you agree to allow Vanta to contact you via the email provided for marketing and other purposes
