Giving the Vanta Agent a computer

This blog is part of our Trustcraft series, in which we dig into Vanta’s approach to building with AI. Read the first blog in this series to learn more about how we define Trustcraft.

‍

The Vanta Agent acts as an automated, 24/7 GRC engineer. It helps customers streamline security and compliance tasks like collecting evidence, verifying documentation, generating security questionnaire responses, and aligning company policies with actual system settings.

‍

Earlier this year, we wanted to transform the class of problems the Agent could address. We added sandbox support to the Agent by giving it isolated Linux environments to do work. This post is about how the sandbox improved our Agent operations, the customer outcomes it has unlocked, and what we learned in the process.

‍

How sandboxes work in the Vanta agent

A lot of GRC work isn’t a simple question-answer problem. Customers ask things like: “Compare these new framework requirements against our current program,” “Import my existing policies and controls into Vanta,” or “Generate a report of everything blocking audit readiness.” These tasks require the Agent to reason across documents, spreadsheets, product data, and complex relationships. That reasoning has to satisfy our Trustcraft bar: good enough that a compliance team can stake their program on it, not just good enough to demo. That's why we don't ship “AI sprinkle.” AI has to carry real compliance work to a reliable outcome.

‍

One of the constraints here is the model context window. It’s a great place for the Agent to reason, but the window doesn’t usually hold a customer’s entire security program, the scratch work it produces along the way, and the final artifacts the model generates. When too much work has to happen inside the prompt, the Agent suffers from context rot, getting lost in the middle of the material, and is forced to make poor tradeoffs like sampling the data, summarizing aggressively, or asking the user to finish the job elsewhere.

‍

The sandbox gives the Agent a place to work. At Vanta, that means a temporary filesystem and command-line environment backed by E2B, a third-party sandboxing and code-execution platform. The Agent is given tools to inspect files, write scripts, run data transformations, and generate outputs without dragging every intermediate step through the conversation. Model capability is part of why this works: recent models have become really good at using bash and exploring filesystems on their own, and we expect that to keep improving.

‍

Sandboxes are not necessary for every conversation. They generally add extra latency and cost due to a few seconds of startup time and additional reasoning steps interacting with the sandbox. Most interactions are simple enough to handle through ordinary tools, so a sandbox is lazily instantiated only when a workflow calls for it: a large file upload, bulk data analysis, report generation, etc. 

‍

What this unlocked

Before sandbox support, the Agent hit a few recurring limitations. 

‍

When customers uploaded large documents for review, the content had to be passed directly into the model. This often overwhelmed the context window or forced the Agent to work from a lossy summary. With a sandbox, large documents are routed to the workspace. The sandbox gives the agent full control with command line utilities to prepare text and save the agent reasoning tokens for higher leverage actions. We see the Agent using sed for narrowing down chunks of text, head for finding existence of the header in a csv file, and general pagination utilities like less for manipulating the document text. The Agent can use these sandbox tools to inspect specific portions of the file, read the file in chunks, and search for keywords more efficiently, allowing it to still answer from the source material without pulling whole files into the conversation.

‍

The same limitation applied to structured product data. A compliance program is a large graph consisting of frameworks, policies, controls, evidence, risks, third-party vendors, personnel, audit requirements, and more. Analytic tasks often require comparing multiple records, finding gaps and inconsistencies, and aggregating across a complete view of a customer's program. Previously, the Agent reviewed only sampled portions of data and produced approximated answers. With sandboxes, it operates more comprehensively over full datasets and produces more complete, accurate reports.

‍

Sandbox support also enables us to build much more complex interactions like:

• Analyzing audit progress and producing a prioritized task list
• Performing accurate gap analysis across controls, evidence, and policies
• Comparing policy versions or making targeted edits
• Importing and analyzing several large files at once
• Generating formatted reports and downloadable exports from analyzed data

‍

How this impacts the customer experience

Sandboxes are especially valuable for customers who come to Vanta with an existing compliance program. They may already have policy libraries, control spreadsheets, evidence mappings, audit artifacts, and supporting documents to migrate. Rather than asking the customer to translate that material to Vanta’s format themselves, the Agent now receives and classifies those files, extracts useful details to map to Vanta concepts, and guides the user through review. 

‍

The sandbox has also enabled the Agent to produce real, downloadable artifacts. Before, if a customer asked, “Give me a report of all my failing controls by their owners in a prioritized list,” the Agent provided a sample set of items and instructions for constructing the report manually. Now the Agent completes the work on its own and responds: “I’ve exported all 145 controls that need attention into a CSV, sorted by owner. You can download the report below.” 

‍

The biggest shift is that sandboxes help the Vanta Agent move from answering questions within its context window to performing more complex workflows: aggregations, file analysis, and exports. Jobs that used to require the model to hold and format everything in one pass now can happen computationally instead.

‍

What we learned

Getting the Vanta Agent to use sandboxes effectively has been an interesting challenge and we’re still learning. Here are our biggest takeaways to date.

‍

1. The challenge is in routing work to the right place

At first glance, the rule seems simple: small datasets stay in context and larger data goes to the sandbox. In practice, the right choice depends on both the type of data and the task. A small dataset may still belong in the sandbox if the user is asking for aggregation, comparison, or export. A larger file may be better handled inline if the Agent needs broad semantic understanding more than structured computation.

‍

Our first approach was to let the Agent make this decision on its own based on the user query, with guidance to leverage the sandbox for tasks that involve aggregating or analyzing data, processing new files, or generating reports. To support this, the Agent needed clear levers to either pull necessary data into context or send it to the file system for review in parts. 

‍

In many cases, the Agent made the right call, routing data to the sandbox and writing Python scripts to parse it without needing explicit step-by-step prompting. In other cases, agent-led routing wasn't reliable enough because the Agent doesn't always know how much data will land in its context window before it calls its tools. It can make a routing decision that looks reasonable at plan time but causes problems at execution time. For those flows, it's been more robust to encode the routing decision deterministically in the harness, rather than leaving it to the Agent's judgment at runtime. In our experience, the right answer is probably a hybrid: Agent-led routing for tasks where context size is predictable, harness-enforced routing where it isn't.

‍

2. Document analysis made routing more nuanced

In the Vanta Agent, customers can upload their own files for review, and it quickly became clear that not all files should be handled the same way. When data is in the context window, the Agent can natively perform semantic search over it and capture the overall meaning. When data is in the sandbox, exact search and structured operations are much easier, but fuzzy semantic matching becomes more difficult. This would require the right retrieval tools or harness support for the agent to read data in chunks and decide what to keep and discard (i.e. compacting the file). For aggregations and report generation, the sandbox is clearly better. For “find the section of this document relevant to a vague question,” the right answer depends on the document type, the file size, and task.

‍

That led us to treat file types differently. Some heuristics were simple: images usually stay with the model so we can use the model’s multimodality, small text files can stay inline because the context cost is low, and large spreadsheets generally go to the sandbox, where the Agent can search, extract, and compute over them programmatically. PDFs are more nuanced. They often benefit from native document understanding, but large PDFs can overwhelm the context window, may need sandbox-based extraction, or may need to retain the exact content to be saved in a customer’s Vanta program.

‍

The broader lesson is that “put everything in the sandbox” is not a universal answer. The Agent needs the right routing behavior for every job and the right tools to support its retrieval needs.

‍

3. Find the failure modes

Sandbox support introduced needed capabilities as well as new ways for overall quality to degrade. A common scenario was the Agent reaching for the sandbox when it did not need to, which can add unnecessary latency, token usage, and operational complexity to a question that could have been answered directly. This gets more pronounced with higher-reasoning models with larger thinking budgets, which are more inclined to reach for a tool just because it’s available. 

‍

Tuning when to use the sandbox and when to keep things simple is an ongoing challenge. The goal is not to maximize sandbox usage, but to use it wisely when it can produce a better customer outcome.

‍

4. The sandbox is an ephemeral workspace, not a database

As sandboxes were adopted for more customer workflows, we had to define what they are and are not for. Other agent tools still handle the durable parts of a workflow like modifying the program, persisting generated artifacts in the product, and updating product state. Vanta’s system remains the source of truth for all customer data. This is especially important in avoiding divergence and multiple versions of program data, enforcing user authorization and permission scopes, and upholding trust. The sandbox serves as a place for the Agent to work and remains separate from the product state.

‍

Sandbox environments are scoped to individual conversations and users. We learned this when building multi-session and multi-user workflows: any work that needs to be retained, revisited, and extended later has to be written to durable storage explicitly. This is especially critical for flows that span multiple sessions or require collaboration across multiple users. The sandbox can help the Agent stage and transform artifacts, but any durable state needs to live in the product. Sharing sandboxes too broadly can also create risks around leaking privileged data across users, so it’s important to maintain a clear separation between the Agent workspace and product primitives.

‍

5. Evals have to expand with the Agent’s new capabilities

Sandbox support made the Agent more capable, but it also made its behavior more complex. The Agent is no longer only choosing which tool to call. It is deciding when to materialize data, how to inspect files, when to run code, how much of a dataset to analyze, and what to return to the user.

‍

That meant our quality signals had to expand. We needed to evaluate not only the final answer quality, but also the process that produced it: Did the Agent use the sandbox for the right tasks? Did it avoid unnecessary sandbox work? Did it analyze the full dataset or just whatever it happened to inspect first? Did it produce a useful artifact? Did the added latency pay for itself? Did the overall flow respect authorization boundaries and user permission scopes? These evals run on offline datasets curated by our GRC subject matter experts as observability and traceability remain core to how we keep the Agent’s outputs explainable and auditable as the capabilities expand.

‍

The practices that got us here are still important: crisp prompting, clear tool interfaces, direct inspection of traces, and evals that measure not just the answer, but the path the Agent took to get there.

‍

Want to work on problems like this?

The Vanta engineering team is growing. See open roles.

‍