The background agent that outgrew me

This blog is part of our Trustcraft series, in which we dig into Vanta’s approach to building with AI. Read the first blog in this series to learn more about how we define Trustcraft.

‍

I don't work on an AI team; I work on product engineering. A while ago, a colleague made an offhand remark, the kind of thing you say without quite meaning it: if only we could have an AI clone of you.

‍

I decided to take it almost literally.

‍

Cloning a code reviewer

My goals were modest: get better at using AI tools, and find ways to scale myself so I could help the people around me. So I went down my own list of responsibilities looking for the first thing worth handing off. My review load was high at the time, which made code review the obvious place to start. I built a Slack bot to do my code reviews for me, and I called it JanBot.

‍

‍

I didn't want a generic AI review tool. Those exist, and they tend to give generic feedback. I wanted something more opinionated, something that reviewed more like me. So I started with my own PR reviews from the past couple of years at Vanta, distilling the things I consistently cared about, then expanded it by mining reviews from a dozen or so other engineers whose judgment I trusted and knew well.

‍

Here is where it got interesting. Once you line up that many reviewers side by side, you notice how little they overlap. Some fixate on mechanical details. Others care mostly about naming and semantics. A few only ever zoom out to the shape of the whole feature. Put together, they made a sharper reviewer than any of us individually. I set out to clone myself, but ended up creating a tool that synthesized a far wider range of engineering perspectives than I could provide alone.

‍

Reviewing the reviewer

JanBot lived in Slack, and it was self-serve. You asked it for a review in a thread, and it did a few things at once. It posted its comments and a recommendation in Slack, along with a briefing for whoever would review the PR: the risk and blast radius of the change, and whether the author's description matched what the code actually did. It also staged the line-by-line comments as a pending review in GitHub, drafts attached to the code that nobody could see until I submitted them. Nothing reached the PR without my sign-off.

‍

One small rule did a surprising amount of work. PRs were never cluttered with machine reviews nobody had vetted, but people still got fast, early feedback in Slack while they waited on a human. It also kept me in the loop by design. Submitting the pending review meant opening GitHub and reading JanBot's comments right next to the code, deciding what was worth keeping; by the time I'd worked through its take, I usually understood the PR well enough to weigh in myself.

‍

Being the gate also meant I could tune JanBot more aggressively than the AI code-review tools we run internally. Those run across the entire eng org and post straight to the PR, so they have to keep false positives low. JanBot didn't. I read every comment first, so it could flag anything even mildly suspicious and I'd filter the noise myself. I'd rather skim past a few false positives than miss something real.

‍

Having learned from real review comments, it also inherited a recognizable tone: warm, curious, more likely to ask a question than hand down a verdict. People responded to that, and started requesting reviews earlier. From the start I had wired up observability, so every use pinged me and I could watch how people actually reached for it. Those notifications became a small joy of the workday. Not all of them were review requests.

‍

‍

Rewriting in Go

The first version was in JavaScript, mostly because it was the first thing I reached for. I didn't put much thought into it, and I'll admit a lot of JanBot was vibe-coded: I leaned heavily on AI to build it, which was half the point. It held up for a couple of weeks, until I was breaking it with every change and wanted a compiler catching my mistakes before my users did.

‍

The obvious typed language was TypeScript, but in my experience, it's hard to get right. The type system is powerful and easy to get wrong, and I still manage to trip over those details. I saw the same pattern in JanBot's reviews, which were noticeably better on Go than on TypeScript. My theory is that Go is smaller and more opinionated, so there's a stronger shared sense of what "good" looks like, for a human and for a model trying to aim at it.

‍

I could see other teams eventually wanting in, and that's what made me take JanBot seriously enough to push for the rewrite. Fast-and-fragile wasn't going to hold up under real use, so I moved the whole thing to Go. I could read Go, which mattered: I wanted the ability to follow and debug what the bot produced, because if I couldn't debug it, I couldn't trust it. That ruled out languages I was less fluent in, regardless of their other technical merits. The rewrite bought real concurrency and tighter control. It was also when I made regression testing a first-class part of the system, not because Go asked for it, but because I finally had the discipline in place to do it properly. That mattered more than it sounds: an agent's output isn't deterministic, so "does it still work?" isn't a question you can answer by eye. You have to measure it, which is the principle this series keeps returning to: what you measure is what you get.

‍

By then JanBot wasn't one bot but several, each with a narrow job: a reviewer for reviews, a worker for pull requests, a manager for Jira, and others besides. Small, single-purpose agents worked far better than one enormous prompt, and that structure stuck as the system grew.

‍

Bringing it to on-call

There was never a roadmap for any of this. I was building in an exploratory way, meandering from one problem to the next, and the problems that stuck were ones I knew firsthand. Once the reviewer worked, I got curious how much more I could hand off, so I pointed JanBot at my own workflow. It began managing my PRs and plans and working through them: picking up a task, making the change, opening the PR, while I steered from Slack. My favorite version of this was kicking off a fix from my phone between meetings, then getting back to my desk to find the PR already up.

‍

The timing was lucky. A broader push was getting underway at Vanta to bring more AI into on-call, and I volunteered, mostly because I'd already built much of the plumbing by accident. JanBot knew how to work with Slack, Git, and our internal data; on-call just pointed that at a new set of inputs: Datadog alerts, Jira tickets, and a steady stream of "is this a bug?" questions.

‍

My part of the org needed the help, too. On-call was drowning in alerts and tickets.

‍

JanBot could help partly because of where the useful context lived. On-call documentation was thin, so most institutional knowledge sat in Slack history and past PRs. Slack especially was a goldmine, full of old incidents and half-remembered fixes and people talking through problems as they happened. JanBot leaned on both, which let it investigate even without a runbook.

‍

Noisy monitoring is a common problem, and the trouble is usually the signal-to-noise ratio rather than the raw number of alerts: most need a human to investigate before you even know whether they matter, and that investigation is the real cost. So JanBot did the first pass. When an alert fired, it investigated automatically and spelled out the impact in plain language: what was affected, how badly, and whether it could wait. When the real problem was the monitor itself, it suggested tuning, feeding a broader monitor-health effort already underway. It didn't fix on-call on its own. Mostly it sped up work that was already going on.

‍

Keeping those investigations in Slack, out in the open, mattered more than I expected. Each one left a trail I could mine later, and because the reasoning played out in a public thread instead of a private tool, people chimed in and pushed back and added what they knew. Debugging became a group activity again. It also matched how investigations actually go: not one-shot, but a hypothesis worked through the evidence until you reach a root cause, in a place where anyone could follow along and correct it.

‍

Being wrong in public cut both ways, though. When JanBot slipped, people were quick to fix it; there's an old adage that the fastest way to get the right answer is to post the wrong one, and a public bot rides that reflex. But visible mistakes erode trust, and it's a short step from correcting a bot to writing it off.

‍

Rolling it out quietly

I rolled it out the way I'd shipped the code-review bot: quietly, with almost no documentation. This was deliberate. I wanted to see how people reached for an agent when no one handed them a manual, what they expected, where they got stuck. I leaned on the same observability I'd had from the start, so every interaction pinged me and I could watch the tool work in real conditions and step in when it went wrong.

‍

Watching so closely had an unexpected effect: it pulled my own attention toward on-call, and I ended up partly on-call much of the time, alongside whoever held the pager, reading the same threads and seeing the same alerts. I never really announced the tool, either. Instead of a rollout memo, I'd drop into a thread, point JanBot at whatever was happening, and let people watch it work. Learn by play, teach by show.

‍

If that sounds like product development, it was, even though none of this was a formal product effort: iterate, talk to people, watch behavior, ship small, get it wrong, adjust. The instincts were ones I already had.

‍

I asked a handful of teams to pilot it early on. Past that, though, I didn't have to push much: interest grew on its own, with teams asking for access before I ever reached out to them. That kind of organic pull was the most encouraging signal I got, not proof of anything, just a hint it was helping people beyond me.

‍

It wasn't all applause. JanBot got things wrong, a lot. Some people didn't mind: they'd correct it, try again, and end up getting real value out of it. Others asked it for something once, got a mediocre answer, and moved on. Which is fair. If you're busy, coaching someone else's bot is nobody's idea of a good time.

‍

I won't pretend the problem is solved; it's still early, and there's plenty of work left. But every so often you catch a glimpse of where this is heading. My favorite came from a Slack alerts channel where someone asked, in genuine disbelief, "how is it possible that there are no alerts today?!" I caught myself asking JanBot to double-check the monitors, on the assumption that something must be broken.

‍

‍

Nothing was broken. It was just quiet for a day. And that day was proof we can get there.

‍

Cutting it back

The last part of the story is the one most engineering write-ups skip: what happens after an experiment works.

‍

JanBot went wide on purpose. The original premise, after all, was to replicate me and my responsibilities in their entirety, so I let it try to do many things and learn what it was actually good at. Honestly, it was also just fun to push the boundaries and see what these tools could do. Once on-call triage emerged as the thing that genuinely moved the needle, the work shifted from adding to removing. We trimmed features and cut the experiments that hadn't earned their place, focusing on making the core reliable enough to depend on. Going wide to find the fit, then narrowing once you have it, turns out to be a reasonable way to build with agents.

‍

Not everything I built made the cut, and I don't consider that time wasted. The code-review side, where this whole thing started, never got fully productionized on its own, though a few people still ask for JanBot's reviews specifically. But it didn't disappear: the learnings from it are feeding into Vanta's general AI code-review work in CI. This isn't specific to agents, either. Whatever you build, a failed or abandoned attempt still leaves behind something the next attempt can use.

‍

Part of the appeal of those early days was how big a swing I could take. AI made a full rewrite feel almost casual: moving the whole thing from JavaScript to Go would have been hard to justify on a normal project, but nothing depended on it yet, so the bet was cheap. Deploys worked the same way. I could ship changes at will and let downtime fall wherever it landed, because nobody was counting on uptime yet. The more people rely on the tool, the less room there is for that kind of swing: deploys now have to be safe and close to invisible, not just fast. It also meant growing up the contribution model. This couldn't stay something only I touched, so other people needed a real way to both operate it and add to it themselves. I could rewrite everything, or take the whole thing down, on a whim in the first month. I can't now.

‍

It also meant letting go. JanBot started as a personal experiment and grew into something bigger than I'd planned for. The on-call side is now being productionized and handed to a dedicated team, which comes with a more official name, and that feels right; a shared piece of infrastructure shouldn't be named after one person.

‍

What I took away

If there's a larger point here, it isn't "build an on-call bot." Nobody mandated any of this, and in my experience nobody has to: give engineers real problems and the space, sometimes the funding, to chase them, and they will experiment on their own. It was true long before AI. All it took here was an offhand comment, some curiosity, and the ordinary instinct to scale yourself and help the people next to you.

‍

The people are what make that possible. Experimentation only works in a culture that's open to new things and forgiving when they go wrong, and agents make the point louder, because they go wrong constantly. JanBot happened to find a fit; the next experiment might not, and that has to be okay too, or nobody takes the swing. The instincts behind it aren't new or AI-specific: reviewing a teammate's work with care, sharing context generously, being patient with someone who's still learning. AI didn't replace any of that. If anything, it raised the value, because now some of your colleagues are agents, and they need the same patience and judgment the human ones do.

‍

So mostly I feel grateful: to the colleague whose throwaway comment started this, to the engineers whose review instincts JanBot is quietly made of, to the people who volunteered their own time to contribute to it, and to everyone who used an imperfect tool, corrected it, and kept coming back. Encouraging feedback and hard feedback alike, since that's how any of this improves. I'm far from the only one experimenting here, and comparing notes has been the best part.

‍

JanBot was just one experiment. The culture that made it possible, open and patient and generous, is the part worth protecting.

‍

Want to work on experiments like this?

The Vanta engineering team is growing. See open roles.

‍

More on how we build AI at Vanta: How we build AI products at Vanta, Where thinking in engineering happens now, Vanta's agent development principles, and The Vanta AI quality evaluation maturity model.

‍