Unlocking AI innovation with quality hill climbing

At Vanta’s recent Company Kickoff in New Orleans, we took the opportunity to showcase an essential approach for rapidly improving AI products: Quality hill climbing. Quality hill climbing is the idea that, given a clear dataset of what "good" looks like, collaborative iteration can swiftly enhance AI features and effectiveness.

‍

Our engineering organization had two hours of dedicated time together, where we organized a hands-on AI activity centered on evaluating and iterating prompts. We wanted the challenge to feel as close to real work as possible, so we used Vanta’s own data to evaluate our prompts. Like our customers, Vanta uses Vanta for its annual SOC 2 audits. 

‍

We took our most recent SOC 2 audit and removed key pieces of data simulating imperfect audit evidence, then we challenged ourselves—can we, in 45 minutes, quickly iterate to a prompt that reliably detects missing audit evidence? We split up into groups of five to experiment with different prompt engineering techniques and various LLM models. At the end of the workshop, teams were graded based on the accuracy of their results and total latency. 

‍

The results were impressive—teams were able to go from a base score of 30% on our eval set to over 75%. After the workshop, we received lots of great feedback from our engineers: 

‍

"The AI workshop was excellent! Working with real data on a real problem illustrated how vast the problem space is for AI applications. It was a blast to iterate with the team and see our solution evolve." - Michael Vobejda

‍

The event made me realize that more context is not always better. Trying to get an LLM to handle many suggestions in context does not necessarily increase the accuracy of your results. Sometimes, simple language that is very clear with the responsibilities expected of a model is better than trying to get it to solve problems with a ton of explicit guidance. - Josue Guzman

‍

‍

Here are some general takeaways from running AI workshops: 

‍

#1:  Prompt engineering is iterative

As engineers progressed through the exercise, what seemed like daunting prompt engineering became manageable through iterative practice. They discovered that crafting effective prompts is a lot like debugging and testing code. Having a good evaluation dataset is also key to understanding if your iterations are improving or regressing.

‍

The workshop emphasized that even logically sound prompts might not always yield the desired outcomes, highlighting the importance of embracing an iterative approach to prompt engineering: Write. Run. Observe. Tweak. Through these quick feedback rounds, prompt engineering went from an intimidating concept to an accessible tool.

‍

#2 Dogfooding is fuel for real progress 

One powerful method we highlighted was dogfooding—using Vanta's own compliance artifacts as the foundation for our AI experimentation. This strategy not only grounded the exercise in real-world relevance but reinforced how iterative refinement using actual company data accelerates AI feature quality.

‍

AI can be complex—which makes using them the best way to learn. Grounding the exercise in real-world data gave the engineering team a jump start for understanding the parts of AI tooling they were less familiar with. Of course, there’s always a risk of overfitting your prompt to the test data, but given a limited experimentation window this worked well.

‍

#3: Prompting is already part of the work we do everyday

One of the best moments came after the challenge, when someone said: “This is just what I do with ChatGPT anyway.” That aha moment helped engineers create a mental model for AI tooling. 

‍

The realization that prompt engineering wasn't an entirely new skillset, but rather something many of our engineers were already utilizing, was empowering. Our team quickly realized that effective prompt development is less about esoteric techniques and more about disciplined iteration guided by clear specifications and thoughtful evaluation.

‍

‍

Making AI a team sport

When AI first landed in software development, it brought along terms like “prompt engineer,” “chain of thought,” and “few-shot examples”, making prompt engineering seem like an art few could do. We don’t want that kind of gatekeeping affecting how we develop software that helps our customers, and after this workshop, many of our engineers understand that prompt engineering has a process, just like software engineering. By teaching quality hill climbing, we empowered over 100 engineers to leverage their existing software development expertise to create awesome, accurate AI features. 

‍

These workshops are just the start, but they’ve already made a difference in lowering the barrier to entry. Thanks a lot to the team behind the activity: Ruyan, Tina, Walt, Noam, Dan, and Kevin! 

‍

If you love solving meaningful problems, and want to work with an innovative team, check out our open roles at Vanta.