Share this article
How Vanta's engineering team improved productivity with esbuild
As of early 2022, Vanta’s backend consists of Typescript services and lambdas, and our frontend is a React and Typescript-based single-page application. We host all this code in a monorepo. Over time, we experienced build slowdowns due to our organically growing codebase.
Ultimately, we decided to use esbuild to ensure developer edit/refresh cycles stay smooth. In this article, we’ll explore why we made this decision, how we implemented esbuild, and the results that followed.
Our codebase has always been structured as a tree of packages. The leaf directories are services that include a common directory and are bundled up by a Dockerfile. For development, we have a docker compose setup that spins up all services, and essentially runs tsc --watch for each service.
Back in the early days, we leveraged symlinks to manage the dependency tree, which we replaced with Typescript Project References to simplify our setup and accelerate builds.
We experienced reasonable performance with this setup for a few years until tsc --watch incremental builds were slowing down. It was taking tens of seconds to re-compile our backend for even the most minimal changes. Our first recourse was to understand the problem using Typescript’s great performance debugging guide. We discovered a few type-heavy packages, like our very own ts-json-validator, and certain usage patterns of mongoose were leading to severe slowdowns. Through a series of casts and manually provided types, we relieved some burden from Typescript's automatic inference and managed to cut build times by >50%. Even then, with a growing engineering team and codebase, we knew that this wouldn’t be a long-term solution, and re-compilation times steadily crept back up.
After another round of Typescript profiling, we noticed an interesting comment on HackerNews that talked about replacing the Typescript compiler with esbuild. After a quick manual prototype, we found that esbuild could re-compile one of our smaller services in under a second, so we decided to investigate whether we could replace all our transpilation with esbuild.
What is esbuild?
Our biggest roadblock to enabling esbuild globally in our codebase was that we didn’t have the Typescript option esModuleInterop enabled.
Understanding the module ecosystem
Difference in syntax
How Typescript fits in
This breaks compatibility with the new ESM spec, as CommonJS require has looser semantics than ES6 import * as. For example, require is allowed to return a non-object, whereas import * as is required to return a plain object. This behavior goes against one of their stated design goals to align with ECMAScript proposals.
Since Typescript’s generated code didn’t conform to the spec, they released an option for interoperability called esModuleInterop. They’ve enabled it by default for new projects, and highly recommend that it’s enabled for all projects going forward. esbuild does not need to break the spec, so its generated code has different semantics than Typescript generated code with esModuleInterop: false. For example, esbuild generates code that assumes imported modules are not callable and are read-only. This is such a common issue that there’s documentation about it, and the recommended path forward is to enable esModuleInterop for codebases to use esbuild.
We needed to take on two kinds of tasks to correctly enable this option: re-do our imports and clean up mocking code.
esModuleInterop automatically enables allowSyntheticDefaultImports, so we migrated many of our imports. This wasn’t strictly required, but it was the cleaner end state with this option enabled.
Unfortunately, this led to a few breakages that weren’t caught at compile-time, since this only retained the same behavior for packages that didn’t have pre-existing default exports. We discovered most bugs here through testing and deployment in our staging environment.
We use sinon for our mocks in tests. An example:
This would no-longer work with esModuleInterop because ES modules are not directly assignable. We were essentially modifying code that was imported, which is a code-smell. We decided to remove mocks where it was easy, and export objects that sinon could mutate when it wasn’t, rather than trying to redo the import itself. This was another common issue with a lot of discussion in the community.
The changes here were pretty mechanical as soon as we got a handle on the problem, and it took one PR for each of the handful of modules affected.
These migrations were the bulk of the work required before we could use esbuild.
With our migration to esModuleInterop complete, we wrote a bash script that essentially set up nodemon to watch our Typescript files, and re-ran esbuild on changes.
A large issue worth noting here is that esbuild is explicitly not a Typescript compiler replacement, so it doesn’t grok project references. Fortunately, since it’s so fast, we rebuild all running services in parallel when there are any code changes to our common directories, and that’s worked out fine in practice. Another issue is that esbuild doesn’t recognize JSON entry-points, so it doesn’t copy over JSON files into the output directory which can be imported with resolveJsonModule: true in Typescript. So we just copy those over manually through a find and cp.
First, since esbuild is not a Typescript compiler, we made it clear to developers that their code might rebuild successfully through esbuild, even if compilation really should have failed due to type errors. Instead, they could use editors to find most compilation errors and run the Typescript compiler on watch mode manually while doing large refactors to catch compilation errors. We also run the Typescript compiler in CI to make sure that no compilation errors get through. Developers seemed to prefer this trade-off.
Second, our team strongly believes that our systems are as consistent as possible in development and production. Therefore, if developers were interacting with esbuild-built code locally, we should ship esbuild-built code to production. To roll this out, we first enabled esbuild for local development behind an opt-in flag. Next, we pushed esbuild-built code to our staging environment, and finally to production. After almost a year in production, we haven’t noticed any issues with the exception of missing source maps which we fixed by tweaking a few flags.
Overall, we’ve been satisfied with our esbuild experience. Looking back, we should have looked into swc a little more, especially since we used Parcel for our front-end code and Parcel 2 uses swc behind the scenes. But it seems easy to switch from one to the other since the bulk of the work was not strictly related to esbuild.
Most developers aren’t aware that we use esbuild behind the scenes, and we count that as a success. On the other hand, we’ve run into more issues around the Typescript server in code editors, mainly around OOMs due to pathological third-party dependencies and organic codebase and dependency growth, so we’ll continue investing on that front.
Learn more about engineering at Vanta
Determine whether the GDPR applies to you and if so, if you are a processor or controller (or both)
Do you sell goods or service in the EU or UK?
Do you sell goods or services to EU businesses, consumers, or both?
Do you have employees in the EU or UK?
Do persons from the EU or UK visit your website?
Do you monitor the behavior of persons within the EU?
Create a Data Map by taking the following actions
Identify and document every system (i.e. database, application, or vendor) which stores or processes EU or UK based personally identifiable information (PII)
Document the retention periods for PII in each system
Determine whether you collect, store, or process “special categories” of data
Determine whether your Data Map meets the requirements for Records of Processing Activities (Art. 30)
Determine whether your Data Map includes the following information about processing activities carried out by vendors on your behalf
Determine your grounds for processing data
For each category of data and system/application have you determined the lawful basis for processing based on one of the following conditions?
Take inventory of current customer and vendor contracts to confirm new GDPR-required flow-down provisions are included
Review all customer contracts to determine that they have appropriate contract language (i.e. Data Protection Addendums with Standard Contractual Clauses)
Review all in-scope vendor contracts to determine that they have appropriate contract language (i.e. Data Protection Addendums with Standard Contractual Clauses)
Have you performed a risk assessment on vendors who are processing your PII?
Determine if you need to do a Data Protection Impact Assessment
Is your data processing taking into account the nature, scope, context, and purposes of the processing, likely to result in a high risk to the rights and freedoms of natural persons?
Review product and service design (including your website or app) to ensure privacy notice links, marketing consents, and other requirements are integrated
Does the notice to the data subject include the following items?
Does the notice also include the following items?
Do you have a mechanism for persons to change or withdraw consent?
Update internal privacy policies to comply with notification obligations
Update internal privacy notices for EU employees
Determine if you need to appoint a Data Protection Officer, and appoint one if needed
Have you determined whether or not you must designate a Data Protection Officer (DPO) based on one of the following conditions (Art. 37)?
If you export data from the EU, consider if you need a compliance mechanism to cover the data transfer, such as model clauses
If you transfer, store, or process data outside the EU or UK, have you identified your legal basis for the data transfer (note: most likely covered by the Standard Contractual Clauses)
Have you performed and documented a Transfer Impact Assessment (TIA)?
Confirm you are complying with other data subject rights (i.e. aside from notification)
Do you have a defined process for timely response to Data Subject Access Requests (DSAR) (i.e. requests for information, modification or deletion of PII)?
Are you able to provide the subject information in a concise, transparent, intelligible and easily accessible form, using clear and plain language?
Do you have a process for correcting or deleting data when requested?
Do you have an internal policy regarding a Compelled Disclosure from Law Enforcement?
Determine if you need to appoint an EU-based representative, and appoint one if needed
Have you appointed an EU Representative or determined that an EU Representative is not needed based on one of the following conditions?
If operating in more than one EU state, identify a lead Data Protection Authority (DPA)
Do you operate in more than one EU state?
If so, have you designated the Supervisory Authority of the main establishment to act as your Lead Supervisory Authority?
Implement Employee Trainings to Demonstrate Compliance with GDPR Principles and Data Subject Rights
Have you provided appropriate Security Awareness and Privacy training to your staff?
Update internal procedures and policies to ensure you can comply with data breach response requirements
Have you created and implemented an Incident Response Plan which included procedures for reporting a breach to EU and UK Data Subjects as well as appropriate Data Authorities?
Do breach reporting policies comply with all prescribed timelines and include all recipients i.e. authorities, controllers, and data subjects?
Implement appropriate technical and organizational measures to ensure a level of security appropriate to the risk
Have you implemented encryption of PII at rest and in transit?
Have you implemented pseudonymization?
Have you implemented appropriate physical security controls?
Have you implemented information security policies and procedures?
Can you access EU or UK PII data in the clear?
Do your technical and organizational measure ensure that, by default, only personal data which are necessary for each specific purpose of the processing are processed?
Develop a roadmap for successful implementation of an ISMS and ISO 27001 certification
Implement Plan, Do, Check, Act (PDCA) process to recognize challenges and identify gaps for remediation
Consider ISO 27001 certification costs relative to org size and number of employees
Clearly define scope of work to plan certification time to completion
Select an ISO 27001 auditor
Set the scope of your organization’s ISMS
Decide which business areas are covered by the ISMS and which are out of scope
Consider additional security controls for business processes that are required to pass ISMS-protected information across the trust boundary
Inform stakeholders regarding scope of the ISMS
Establish an ISMS governing body
Build a governance team with management oversight
Incorporate key members of top management, e.g. senior leadership and executive management with responsibility for strategy and resource allocation
Conduct an inventory of information assets
Consider all assets where information is stored, processed, and accessible
- Record information assets: data and people
- Record physical assets: laptops, servers, and physical building locations
- Record intangible assets: intellectual property, brand, and reputation
Assign to each asset a classification and owner responsible for ensuring the asset is appropriately inventoried, classified, protected, and handled
Execute a risk assessment
Establish and document a risk-management framework to ensure consistency
Identify scenarios in which information, systems, or services could be compromised
Determine likelihood or frequency with which these scenarios could occur
Evaluate potential impact of each scenario on confidentiality, integrity, or availability of information, systems, and services
Rank risk scenarios based on overall risk to the organization’s objectives
Develop a risk register
Record and manage your organization’s risks
Summarize each identified risk
Indicate the impact and likelihood of each risk
Document a risk treatment plan
Design a response for each risk (Risk Treatment)
Assign an accountable owner to each identified risk
Assign risk mitigation activity owners
Establish target dates for completion of risk treatment activities
Complete the Statement of Applicability worksheet
Review 114 controls of Annex A of ISO 27001 standard
Select controls to address identified risks
Complete the Statement of Applicability listing all Annex A controls, justifying inclusion or exclusion of each control in the ISMS implementation
Continuously assess and manage risk
Build a framework for establishing, implementing, maintaining, and continually improving the ISMS
Include information or references to supporting documentation regarding:
- Information Security Objectives
- Leadership and Commitment
- Roles, Responsibilities, and Authorities
- Approach to Assessing and Treating Risk
- Control of Documented Information
- Internal Audit
- Management Review
- Corrective Action and Continual Improvement
- Policy Violations
Assemble required documents and records
Review ISO 27001 Required Documents and Records list
Customize policy templates with organization-specific policies, process, and language
Establish employee training and awareness programs
Conduct regular trainings to ensure awareness of new policies and procedures
Define expectations for personnel regarding their role in ISMS maintenance
Train personnel on common threats facing your organization and how to respond
Establish disciplinary or sanctions policies or processes for personnel found out of compliance with information security requirements
Perform an internal audit
Allocate internal resources with necessary competencies who are independent of ISMS development and maintenance, or engage an independent third party
Verify conformance with requirements from Annex A deemed applicable in your ISMS's Statement of Applicability
Share internal audit results, including nonconformities, with the ISMS governing body and senior management
Address identified issues before proceeding with the external audit
Undergo external audit of ISMS to obtain ISO 27001 certification
Engage an independent ISO 27001 auditor
Conduct Stage 1 Audit consisting of an extensive documentation review; obtain feedback regarding readiness to move to Stage 2 Audit
Conduct Stage 2 Audit consisting of tests performed on the ISMS to ensure proper design, implementation, and ongoing functionality; evaluate fairness, suitability, and effective implementation and operation of controls
Address any nonconformities
Ensure that all requirements of the ISO 27001 standard are being addressed
Ensure org is following processes that it has specified and documented
Ensure org is upholding contractual requirements with third parties
Address specific nonconformities identified by the ISO 27001 auditor
Receive auditor’s formal validation following resolution of nonconformities
Conduct regular management reviews
Plan reviews at least once per year; consider a quarterly review cycle
Ensure the ISMS and its objectives continue to remain appropriate and effective
Ensure that senior management remains informed
Ensure adjustments to address risks or deficiencies can be promptly implemented
Calendar ISO 27001 audit schedule and surveillance audit schedules
Perform a full ISO 27001 audit once every three years
Prepare to perform surveillance audits in the second and third years of the Certification Cycle
Consider streamlining ISO 27001 certification with automation
Transform manual data collection and observation processes into automated and continuous system monitoring
Identify and close any gaps in ISMS implementation in a timely manner
Download this checklist for easy referenceDownload Now
Determine which annual audits and assessments are required for your company
Perform a readiness assessment and evaluate your security against HIPAA requirements
Review the U.S. Dept of Health and Human Services Office for Civil Rights Audit Protocol
Conduct required HIPAA compliance audits and assessments
Perform and document ongoing technical and non-technical evaluations, internally or in partnership with a third-party security and compliance team like Vanta
Document your plans and put them into action
Document every step of building, implementing, and assessing your compliance program
Vanta’s automated compliance reporting can streamline planning and documentation
Appoint a security and compliance point person in your company
Designate an employee as your HIPAA Compliance Officer
Schedule annual HIPAA training for all employees
Distribute HIPAA policies and procedures and ensure staff read and attest to their review
Document employee trainings and other compliance activities
Thoroughly document employee training processes, activities, and attestations
Establish and communicate clear breach report processes
to all employees
Ensure that staff understand what constitutes a HIPAA breach, and how to report a breach
Implement systems to track security incidents, and to document and report all breaches
Institute an annual review process
Annually assess compliance activities against theHIPAA Rules and updates to HIPAA
Continuously assess and manage risk
Build a year-round risk management program and integrate continuous monitoring
Understand the ins and outs of HIPAA compliance— and the costs of noncompliance
Download this checklist for easy referenceDownload Now
FEATURED VANTA RESOURCE
The ultimate guide to scaling your compliance program
Learn how to scale, manage, and optimize alongside your business goals.