You can build verifiers for messy problems

This blog is part of our Trustcraft series, in which we dig into Vanta’s approach to building with AI. Read the first blog in this series to learn more about how we define Trustcraft.

‍

Models are exceptional at coding because they can check their work. From compilers onward, the software industry has spent more than 70 years developing tooling and techniques like static analysis, TDD, and observability. These deterministic checks allow both people and agents to improve their work iteratively.

‍

Other knowledge-work tasks are different. There often isn’t a compiler, correctness is ambiguous, and experts can disagree about the right answer. If AI is used to solve these problems reliably, it needs to have judgement people can trust.

‍

A verifier evaluates a piece of work and gives feedback to whoever produced it—human or agent. This is different from an evaluator, which measures the quality of the AI system, and is used by the team building it.

‍

At Vanta, we built an AI verifier for compliance called evidence evaluation. Given a requirement, it reviews evidence the way an auditor would and explains what’s missing. Verifiers have an especially high quality bar, since both humans and agents rely on their judgement to decide what to do next. The rest of this post is about how we built one.

‍

‍

The difference between messy problems and olympiad problems

AI models have long beaten humans at all kinds of hard problems, like PhD-level science and international math olympiads. What they often have trouble with is real-world tasks, because they may lack the right context to complete the task (e.g., imagine asking a brilliant, over-eager alien to book a flight for you with no information about where you want to go), or they don’t know what good looks like. These are what we consider “messy problems.” 

‍

Assessing compliance evidence is especially messy because it’s an ambiguous problem: different auditors may disagree about whether evidence meets certain requirements. Vanta has helped companies navigate audits for eight years. That experience shapes how we think about evaluation: measuring whether the AI reaches the same PASS or FLAG decision as the auditor is an extremely valuable signal.

‍

However, auditor agreement alone is insufficient to define a trustworthy verifier. Our goal isn’t simply to predict whether an auditor would pass evidence. We want the verifier to help customers produce stronger evidence, and that means it sometimes needs to hold a higher bar than historical audit outcomes.

‍

For example, at Vanta, we’re opinionated that evidence should be attributable to a date. A screenshot without a date doesn’t establish the state of a system at a particular point in time, and a report without a date can’t demonstrate when it was produced. Adding a date is a low-effort change for customers that meaningfully improves the quality of the evidence and aligns with the expectations of many auditors. Even if a particular auditor could accept the undated version, we’d rather encourage the stronger version.

‍

Stop building AI products like it’s 2025

We first started building our evidence evaluation AI verifier in early 2025 with Sonnet 3.7 and GPT-4o. At that time, social media was flooded with reports of prompting instructions like “take a deep breath” or “make no mistakes”, and they actually worked. Frontier models weren’t reliable enough to meet the quality bar without significant handholding. We needed to narrow problems and constrain models. We had to give them very long lists of what specifically not to do.

‍

That was the right architecture in 2025. Since then, models have become far more capable, and those have become irrelevant, as agents are now completing important and complex tasks in all kinds of domains.

‍

However, many AI products built earlier still have the same architecture from the time they were built. Imagine buying a fully autonomous humanoid robot and only allowing it to press the “Start” button on your dishwasher. You already have something capable of doing the dishes, but you’ve wrapped it in a workflow that prevents it from doing the job.

‍

Deleting most of the pipeline

‍

We deleted the vast majority of our pipeline and gave the model more control. As a result:

• Quality improved
• The feature is 17X faster (2 minutes to 7 seconds today)
• The feature is more flexible and can handle more types of inputs

‍

Defining what good looks like

We could confidently overhaul the architecture because of our robust evaluation system and tooling.

‍

We assess AI quality with:

• Offline evaluation
• Online evaluation
• Product metrics

‍

Offline evaluation

We have a curated dataset with a small number of inputs and labelled outputs. Through close pairing with subject matter experts, we separated concerns between admissibility and adequacy. The initial architecture had different pipelines to address these problems, and while they are now in one prompt, the labels have survived the changes in implementation details.

‍

‍

Online Evaluation

When using LLM-as-a-judge (LaaJ), the challenge is to create metrics that are both useful and trusted. Evaluators need to be aligned and need maintenance.  

‍

Badness metrics detect failure modes. They are easier to align, but incomplete as they won’t be able to demonstrate that an evaluation is good. These failure modes are based on a narrow end-to-end user pain. For example, the pedantry evaluator flags feedback that’s technically correct but immaterial to whether the evidence meets the requirements. Compliance can be generally pedantic based on a broader definition, but what matters here is immaterial feedback that’s irrelevant to the audit itself.

‍

Goodness metrics are broader. This is much less precise but more complete and harder to align, and we’ll never fully trust these. They’ll give us a directional idea of where we are, and the badness metrics serve as a validity check for the goodness metrics (i.e., if goodness is high and badness contradicts this, we know the goodness evaluator is wrong).

‍

Our tooling allows us to replay representative traces and assess them with the LaaJ evaluators, giving us greater observability and auditability into how the system performs.

‍

Product metrics

Products ultimately exist to deliver an outcome for the end user. No matter how good the offline and online evaluation is, it’s still important to capture signals closer to what the end-user is experiencing. When building AI features, inputs and user needs can change over time, so we need to ensure users are happy with the product.

‍

‍

Want to work on experiments like this?

The Vanta engineering team is growing. See open roles.