BlogProduct updates
June 13, 2025

Root Cause Analysis (RCA) of Product Bug (Inc-868)

Written by
Jeremy Epling
Chief Product Officer
Reviewed by
No items found.

Accelerating security solutions for small businesses 

Tagore offers strategic services to small businesses. 

A partnership that can scale 

Tagore prioritized finding a managed compliance partner with an established product, dedicated support team, and rapid release rate.

Standing out from competitors

Tagore's partnership with Vanta enhances its strategic focus and deepens client value, creating differentiation in a competitive market.

On May 26, we became aware of a product code change that resulted in a subset of data from less than 20% of our third-party integrations being exposed to other Vanta customers. This incident was not security related and did not involve API keys, credentials, or an intrusion.

In total, fewer than 4% of Vanta customers were impacted, and we communicated directly with all impacted customers upon identification of the issue, throughout the remediation process, and upon full remediation. We are also sharing this Root Cause Analysis (RCA) publicly to provide full clarity into the cause of the incident and steps we’re taking to prevent it in the future. 

Trust is not just a platitude—it’s fundamental to who we are and what we do at Vanta. While we never want incidents to happen, we believe that being transparent, proactive and accountable is how we continue to earn our customers’ trust. 

Overview

On May 22, 2025 at 5:11pm PDT, we deployed a code change that caused a subset of data synced via Vanta’s third-party integration APIs to be written into the wrong customers’ tenants. The incorrect data surfaced for impacted customers in several downstream systems including our compliance monitoring, personnel management, access review, and vulnerability management products. 

Vanta became aware of this bug on Monday, May 26 at 3:01pm PDT when a customer contacted our support team to state that they saw unexpected data from an integration in their Vanta tenant. That started our investigation into this issue and triggered our Incident Response process. The bug was confirmed on Tuesday, May 27 at 8:04am PDT and the code change was immediately reverted. The majority of impacted database tables were successfully restored to their pre-incident state by 5:05pm PDT that same day. 

Once we reverted the code change, data self-healed through subsequent syncs for over 98% of impacted customers. We spent the remainder of active remediation time closely examining all downstream systems including test and vulnerability history, personnel and device management, access reviews, exports, and ticketing integrations to correct any data that hadn’t self-healed. Full remediation was completed by June 3, 2025 at 3:21 PDT.

Root Cause Determination

The incident was caused by a code change intending to improve performance for our data ingestion APIs. These APIs allow customers to sync data from third party tools into Vanta through custom integrations. The code change removed a domain ID filter applied during these API calls, which the team believed wasn’t needed since Vanta was passing a globally unique resource ID. However, the resource ID was only unique within each customer’s specific installation of an integration with a third-party tool, and could be repeated/reused in other customer installations of the same tool.

As a result, in some cases, data synced from Customer A’s third party integrations was applied to Customer B when Customer B had resources in a soft-deleted state with identical resource IDs to those of Customer A’s resources. Active data already stored in Vanta, as well as data fetched by Vanta’s native integrations, was not impacted by the incident. Existing tests and code review did not detect this bug before it was deployed to production.

The data exposed to other customers was synced from a limited number (less than 20%) of total third-party integrations. The data varied based on which integrations the customer had installed and could have included metadata about employee security training records, employee access to tools, devices, and/or vulnerabilities. 

MFA credentials were not affected. Nor were passwords, API keys, financial information, or healthcare information. 

Timeline

  • May 22, 2025 5:11 PM PDT - Data exposure event started with a code change deployed to production
  • May 26, 2025 3:01 PM PDT - Customer notifies Vanta Support team about unexpected data from one of their installed integrations and Vanta’s investigation begins immediately
  • May 27, 2025 8:04 AM PDT - Vanta engineering confirms the bug leading to the reported behavior
  • May 27, 2025 9:55 AM PDT - Code change is identified and reverted
  • May 27, 2025 5:05 PM PDT - Impacted tables in source database are restored to their pre-incident state
  • May 29, 2025 7:28 AM PDT - Vanta completes analysis of impacted customers
  • May 29, 2025 11:40 AM PDT - Vanta begins notifying affected customers
  • May 30, 2025 7:07 PM PDT - Remediations in downstream systems are completed for more than 95% of impacted customers 
  • June 3, 2025 3:21 PM PDT - Remediations in downstream systems are completed for all impacted customers
  • June 13, 2025 9:00 AM PDT- RCA published to customers, Vanta’s blog, and Vanta’s Trust Center

Preventive Actions 

While we never want incidents to occur, we're taking the opportunity to strengthen our systems and fortify processes with several preventative measures including: 

Additional cross-customer query safeguards in third-party integrations

Vanta has numerous protections to prevent queries that blend data from multiple customers, but our ingestion system for third-party integrations lacked sufficient database-layer protections. We will add these protections to all third party integrations by July 18th, 2025.

Additional pre-deployment testing for third-party integrations

We are making changes to ensure that changes like this are caught before deployment through rigorous testing in the Build, Deploy, and Quality Assurance process. While we regularly add and improve testing, we have increased unit and static testing in response to this incident to fully validate code changes around data ingestion. We have added a suite of tests verifying domain isolation when our applications query, add, update, or delete data from third-party integrations.

Vanta Code Risk Assessment and Review 

We will engage a third party to conduct a stand-alone, thorough review of our code base to identify any additional high risk areas that need attention and remediation by Q4 2025. 

Guidance for Customers

No action is needed from our customers. We know this wasn’t ideal, and we’re sorry for the extra time and concern it may have caused. We are committed to full transparency and our team is here to answer any questions you have. 

Please reach out to support@vanta.com and we will be happy to discuss the RCA findings further.

Access Review Stage Content / Functionality
Across all stages
  • Easily create and save a new access review at a point in time
  • View detailed audit evidence of historical access reviews
Setup access review procedures
  • Define a global access review procedure that stakeholders can follow, ensuring consistency and mitigation of human error in reviews
  • Set your access review frequency (monthly, quarterly, etc.) and working period/deadlines
Consolidate account access data from systems
  • Integrate systems using dozens of pre-built integrations, or “connectors”. System account and HRIS data is pulled into Vanta.
  • Upcoming integrations include Zoom and Intercom (account access), and Personio (HRIS)
  • Upload access files from non-integrated systems
  • View and select systems in-scope for the review
Review, approve, and deny user access
  • Select the appropriate systems reviewer and due date
  • Get automatic notifications and reminders to systems reviewer of deadlines
  • Automatic flagging of “risky” employee accounts that have been terminated or switched departments
  • Intuitive interface to see all accounts with access, account accept/deny buttons, and notes section
  • Track progress of individual systems access reviews and see accounts that need to be removed or have access modified
  • Bulk sort, filter, and alter accounts based on account roles and employee title
Assign remediation tasks to system owners
  • Built-in remediation workflow for reviewers to request access changes and for admin to view and manage requests
  • Optional task tracker integration to create tickets for any access changes and provide visibility to the status of tickets and remediation
Verify changes to access
  • Focused view of accounts flagged for access changes for easy tracking and management
  • Automated evidence of remediation completion displayed for integrated systems
  • Manual evidence of remediation can be uploaded for non-integrated systems
Report and re-evaluate results
  • Auditor can log into Vanta to see history of all completed access reviews
  • Internals can see status of reviews in progress and also historical review detail
FEATURED VANTA RESOURCE

The ultimate guide to scaling your compliance program

Learn how to scale, manage, and optimize alongside your business goals.