Design tools around information, not APIs

This blog is part of our Trustcraft series, in which we dig into Vanta’s approach to building with AI. Read the first blog in this series to learn more about how we define Trustcraft.

‍

The Vanta Agent acts as an automated, 24/7 GRC engineer. It helps customers streamline security and compliance tasks like collecting evidence and verifying documentation, among other things.

‍

In an earlier blog post, we talked about why we gave the Vanta Agent a sandbox and the value it unlocked. However, a compute layer is only effective if the agent retrieval works reliably. This post explains how we redesigned our tool interfaces so the model can decide what data it needs while deterministic code handles fetching, enriching, and routing it.

‍

The scaling problem with web API-shaped tools

Most APIs are not built for agents. Teams often build agent tools around existing services designed for web and other backend consumers, and re-working services to match your agent tools is not always feasible or even desirable. These interfaces can map cleanly to simple agent queries. “Show me the critical vulnerabilities detected in the last seven days,” is quite simple, but some environments have millions of historical vulnerability records. When those same APIs are exposed directly in agent tools, larger tasks like “Summarize all open and closed vulnerability records,” can result in sifting through millions of records, pushing pagination, batching, joins, and follow-up calls into the agent loop.

‍

Relying on a non-deterministic LLM to orchestrate deterministic data retrieval patterns can create opportunities for incomplete data, malformed calls, unnecessary latency, and context spent on plumbing. For example, let’s assume you have 10 calls needed for a given workflow using your APIs. If you leave the model to select the correct tool, in the right order, with the right parameters 95% of the time, that leaves you with a workflow having a dismal theoretical success rate of only 60%.

‍
                                        0.9510≈ 60%

Your harness should handle the deterministic pieces of your workflow.

‍

Make retrieval declarative

‍

We refactored our tool interfaces so the agent could fetch data declaratively. Instead of forcing the agent to reason about pagination mechanics, we let it specify the amount of data it needs: the first N results or all matching results. The tool infrastructure is then responsible for walking through the pages. If the agent requests additional data from the tool, the same infrastructure fetches enrichment data and joins it before returning to the user.

‍

Consolidating retrieval and enrichment is most useful when one tool exists to hydrate records returned by another. Funneling IDs between tools is mechanical work that can be encapsulated in the tool itself. However, it should not turn every domain into a single general-purpose query tool.

‍

Broader retrieval still needs guardrails. “Fetch all” against a broad query might mean tens or hundreds of thousands of records, so filters and limits should be part of the contract. In our case, we cap a single tool response’s output to limit data going over the wire. If a query matches more than that limit, the tool returns the results, lets the agent know that the result is incomplete, and asks it to add filters to reduce the output size. The important part is to make truncation explicit and give the agent a recovery path, rather than presenting a partial result as complete.

‍

Completeness comes with a latency tradeoff. However, the optimization target for agents is often different from web—we’re willing to spend more time on retrieval when it results in a more complete and accurate answer.

‍

Pushing deterministic pagination and joins behind the tool code changes the model's job from figuring out how to stitch together data from tools to deciding what data would answer the user's question. That distinction matters when you're building over hundreds of internal APIs with different shapes, pagination schemes, and ownership boundaries. 

‍

This aligns with a broader lesson emerging in agent tool design: Rather than simply wrapping existing API endpoints, tools can consolidate lower-level operations and expose interfaces that better match how agents decompose tasks. Anthropic has described a similar principle in its guidance on writing effective tools for agents.

‍

The pattern is simple: Make intent declarative and execution deterministic.

‍

Route large outputs to the sandbox

‍

Letting a tool retrieve complete datasets can produce outputs far larger than should go into model context. Automatic routing makes that retrieval practical in our architecture (shout out to Jason Lam from the AI platform team for building this).

‍

We automatically redirect large tool outputs to a JSON file in the sandbox rather than returning them directly in the model's context. If the response surpasses the threshold, we return the file path, schema, preview, and basic metrics like record counts so that the agent can operate on the data. Small responses stay in context, preserving the fast loop you want for conversational queries. Larger responses become data the agent can filter, aggregate, and compute over with code in the sandbox.

‍

‍

We saw improvements in both efficiency and quality from our changes. Token usage and agent cost per task dropped by ~27%, largely because we kept intermediate datasets out of context and reduced the number of tool calls and decisions the agent needs to make. For quality evaluation, we looked at our task completion metric, which evaluates whether the agent successfully completed the user tasks. On more complex analytical queries, task completion rate increased by more than 45% relative to the previous implementation.

‍

Different query patterns, consistently successful outcomes

Our agent might need to answer a question about an audit’s completion date in one turn, then analyze the customer’s entire program for audit readiness in the next. Those tasks require very different amounts of data, but they don’t need different systems. Whether the answer takes ten records in context or ten thousand in the sandbox, our agent can focus its reasoning on what really matters—solving the user’s problem.

‍

Want to work on projects and experiments like this? The Vanta engineering team is growing. See open roles.