The hallucinating bot with Alex Stamos, CPO at Corridor | The Tabletop

Follow The Tabletop on YouTube or wherever you get your podcasts.

‍

In this episode

What does a security leader actually do when an AI chatbot starts confidently revealing customer data that was never supposed to see the light of day?

‍

Alex has spent his career at the intersection of security and the hardest problems in tech—Chief Security Officer at Yahoo, Facebook, and SentinelOne, founder of the Stanford Internet Observatory, and now Chief Product Officer at Corridor, a startup focused on the security and safety of AI coding agents. If anyone knows what it looks like when AI ships faster than security can keep up, it’s him.

‍

In this episode, host Khush Kashyap drops Alex into a fictional scenario: He’s roleplaying the CISO at Buzzmatrix, a fictional 400-person AI SaaS company with a lean security team and no in-house incident response. Late on a Friday, the VP of Marketing pushes a new AI chatbot live without security review. By Monday, customers are posting screenshots of the bot serving up order histories, account details, and support summaries—some real-looking, some hallucinated. Across a series of escalating injects, Alex has to decide what to take down, who to wake up, when to bring in legal, and what he owes customers.

‍

Alex walks through engaging outside counsel, the GDPR clock, forensic vendor management, and the leverage a CISO actually holds over an uncooperative vendor. Along the way, he makes the case that the existential threat in most incidents isn’t the lawsuit, it’s the erosion of customer trust. At the end, he renders his verdict: real incident or constructed fiction?

‍

Time stamps

[00:00]  “Our lawyers will destroy you guys in court”

[00:46]  Welcome to The Tabletop: Meet Alex Stamos and Today’s Scenario

[01:06]  Setup: CISO at BuzzMetrics, a 400-Person AI SaaS Company

[01:24]  The Friday Launch: Marketing Ships an AI Chatbot Without Review

[02:11]  Inject One: The Screenshots: Real and Hallucinated Customer Data

[03:57]  Pull It Down or Assess First? Triaging the Marketing Bot

[04:27]  Inject Two: The Conversation: Customers Claim the Data Is Theirs

[05:03]  Why You Don’t Post Before Legal Signs Off

[07:10]  Do You Owe a Response When the Data Is Fictional?

[07:48]  Inject Three: The Training Set: Real Tickets, No Scrub, No Consent

[08:14]  Now It’s a Real Breach: Outside Counsel, Regulators, and the GDPR Clock

[10:52]  When Does the Investigation Itself Become a Liability?

[11:47]  Forensics Inside the Vendor and the Hugging Face Nightmare

[12:51]  Vendor Leverage: “You guys are showing up in our write-up”

[14:55]  Over-Legalization: Why Lawyers Shouldn’t Drive Incident Decisions

[15:29]  Litigation Risk vs. the Existential Risk of Lost Trust

[16:43]  Root Cause: Launch Calendars, Data Controls, and a Culture Not Paranoid Enough

[20:03]  The Wake-Up Call: Staffing a Product Security Team After an Incident

[20:15]  The Moment of Truth: Real Incident or Fiction?

[21:16]  Off the Table: The One Question to Ask Before AI Touches Customer Data

[21:32]  The Privilege Differential and Why You Can’t Trust AI Access Control

‍

Transcript

Khush: I'm Khush Kashyap, and this is The Tabletop. Alex Stamos has spent his career at the intersection of security and some of the hardest problems in tech. CSO at Yahoo, Facebook, SentinelOne, founder of the Stanford Internet Observatory, and now Chief Product Officer at Corridor. If anyone knows what it looks like when AI ships faster than security can keep up, it's him.

‍

But today, he's in the hot seat, role-playing the CISO at Buzzmatrix. A scenario unfolds with multiple injects. He gives his live reactions. At the end, he'll decide real incident or fiction. Alex, are you ready for The Tabletop? 

‍

Alex: Absolutely. 

‍

Khush: The clock is ticking. Let's get into it. It's late on a Friday afternoon when you get a Slack message from Jen, who's your VP of Marketing.

‍

She wants you to know, and not ask, that the team has just pushed a new AI chatbot live on the website. It's part of a campaign they have been building for two weeks. She's excited. Any initial reactions? 

‍

Alex: Yeah, so I, this is an initial control failure in that people are able to push anything dynamic especially something that's AI powered, without the security team being involved.

‍

So my initial reaction would be one of dread. Of course, it's a Friday is always when things kick off. , so if it was possible and I had an Op Sec team, I'd try to have them do a really quick review as, as, as quick as possible.

‍

Khush: Mm-hmm. Inject one, the screenshots By Monday, users are posting screenshots of the bot responding with specific confident answers, including customer order histories, account details, and support ticket summaries.

‍

Some of it appears real, and some of it looks like it's hallucinated. What do you do? 

‍

Alex: Um, can you tell me a little bit about this company? Like, what our capabilities are? Is this- 

‍

Khush: Yeah. Let's think about it as a four hundred people AI SaaS company- 

‍

Alex: Okay ... 

‍

Khush: with a security team. Right. But for four hundred people, we are talking 

‍

Alex: about- Four or five people, something like that.

‍

Yeah. No in-house incident response, no in-house real investigations capability. 

‍

Khush: Yes. 

‍

Alex: Right. So,  these reports come in. You obviously, you... It... Hopefully, we're at the point where we have the ability to call a security SEV, right? Um, you know, it hasn't been verified yet, so it's probably not a SEV zero or whatever the equivalent would be.

‍

But you'd wanna get your security team engaged immediately on trying to investigate. , that'd be from two ways. One, pulling the logs on these individuals. , and so from the reports, we should hopefully be able to roll back through the logs. , if my team is not yet engaged with the people who wrote this bot, they're gonna have to figure out where that source code is, where the logs are stored, and such.

‍

This is always a problem, in these situations, that if the security team was not engaged in the creation of the, the service or the product, they don't know how to do incident response. I've seen this over and over again, where it can take days to get access to that data. So you try and get them as quickly as possible working with the developers, wake those folks out of bed get them into the office.

‍

The moment that that turns, it turns out that these reports are accurate, and especially if it's data that's been leaked that's across users then I think you, you upgrade that to your, your top-level severity and hopefully engage the kind of executive incident process. 

‍

Khush: So the bot is already live. Do you pull it down immediately or try to assess the blast radius and the repercussions first?

‍

Alex: I mean, this is just, like, a marketing bot. This is not the production product, so I think you would take it down immediately. Yeah. 

‍

Khush: Okay. Got it. So Jen, 

‍

Alex: you're- And I'm already, I'm already ticked that Jen has shipped this without asking. Yeah. So I'm looking for an excuse to take this down. , and, - , you know, so it, you know, you're gonna be on a hair trigger since this is not something that turns out to actually be that critical.

‍

Khush: Inject to the conversation. Your data teams confirm that the bot isn't pulling from a live database. It hallucinated all of it. But the outputs are specific enough that two customers are publicly claiming the information is about them. 

‍

Alex: Hmm. 

‍

Khush: Jen has drafted a social media response calling the bot creative and conversational, and the output's not representative of real customer data.

She wants to post it in the next hour. Legal hasn't seen it yet. You haven't signed off yet. What do you do next? Who needs to align internally first? 

‍

Alex: Yeah, you-- So you absolutely don't post anything like that, that comes to conclusions until, one, you're absolutely sure, and two, you-- for something like this where customers are now making claims publicly, you have to have legal sign off, right?

‍

Um, at this point, you know, hopefully we've engaged kind of an executive level incident process, right? This is where it's interesting to figure out, like, the actual source of this tabletop because certainly it could be close enough if it's either been prompted or if it has memory about the user that from the user's data or it could be a public training set, or we could be leaking data.

‍

If it's data that's specific to us as a company, we could be leaking it somewhere else. And so one, you're gonna want the, you know, probably the engineering team that worked on this with oversight from the security team to start trying to investigate, well, what's actually, you know, caused this. A lot of that's just gonna be testing, right?

‍

We've taken the bot hopefully off the public side, but it's still gonna be working internally. We should be able to prompt it to see with our own, you know, internal endpoints, for example, to see where it's getting the data from and, and, to see whether it lines up at all. Is this from public sources, the internet, stuff like that.

‍

Um, so you have one person do that. 

‍

Khush: Mm-hmm. 

‍

Alex: Um, I think you tell Jen she has to cool her jets and then if, if that's a problem, then at this point you have to get the CEO involved because this has become like a, a real serious, um...  impact, you know, potential impact to the company's reputation. Um, you know, hopefully that's part of a standard mechanism by which you've already notified a bunch of members of the executive team.

‍

They know that there's an update coming, so you probably have a crisis management call that includes the CEO, and lay out, "This is where our investigation has shown so far. We are not totally sure about this. till our recommendation of the security team would be to put out a holding statement that it's down, we're doing an investigation and such, and then not to put out anything publicly until it...we're absolutely a hundred percent sure." 

‍

Khush: That makes sense. 

‍

Alex: Like, when you have to undo those things, it is much, much, much worse. You, you, you can turn what was a small mistake into a huge scandal if you end up putting out a statement that you, you have to retract. 

‍

Khush: I wanna go back to the customer claims. If the data is fictional, but a real customer believes it exposed them, do you, as the CISO of this company owe them the same response as an actual breach victim?

‍

Alex: Well, do you owe them a response, like, from a legal perspective? No. I think the important thing would be, once you know for sure what's going on, to, to, to communicate with them. If these customers are complaining so much they might get litigious, so again, I, I would not have any kind of communication with them other than, "Thank you for the report. We are investigating," , at this point until you're a hundred and ten percent sure of you know exactly what's going on. 

‍

Khush: Inject three: the training set. So your team completes a full review. The bot was trained on a corpus that includes real customer support tickets, identifiers intact, no scrub, no consent.

‍

There is no way to know with certainty that the model retained or what it might surface if it went back online. And you now have confirmed that real customer data was part of the training set.

‍

Alex: So I'm glad we didn't say anything. 

‍

Khush: Which changes what you owe to people and when. What do you do with that information, and who do you tell first?

‍

Alex: Okay, so now you probably have a real data breach situation. Mm. And so this is where first you talk to your general counsel, and you have to inform the CEO and general counsel in a small group of, of the findings and then bring that to the larger executive group. At this point, because it's a real breach, a company this size, you need to get outside counsel involved, right?

‍

So there's almost certainly gonna be litigation. There's gonna be regulatory hits. Um, and the general counsel at a company this size is not gonna have an internal team that can handle that, right? There's not gonna be an AGC of security like you would at a big public company. At a big public company, you could probably handle an incident like this internally.

‍

But a four-person company, it's just not gonna be possible. So you're gonna recommend the GC. I think we need to bring in specialized counsel. I, as an experienced CISO, have some recommendations, but I'm sure you, you know plenty of people you want to call. Um, and you probably are looking for an outside firm because this is gonna be the kind of thing, again, if it leads to litigation or if there's any doubt of where the data came from, you're going to need an outside firm to do that.

‍

You're also gonna need to get the GC immediately involved in looking at the contract and the relationship with that subsidiary-- with that company that did the work because that relationship is gonna be a really big deal. Um and so, you know, at this point, you're now bringing up to that level of incident, and I think what you do next is really dependent on where are these customers located?

‍

What are your regulatory hits, right? So do you have a GDPR hit? If so, you have a forty-eight-hour clock that's ticking. Um, even in the United States, depending on state laws, there's a bunch of different clocks. , and so you're gonna have to start really scraping through the logs, and you can start that with your team.

‍

If you can get good external incident responders , quickly, then you can have them start to assist in that. But you get your team as part of their analysis to really start , indexing you know, who, who possibly searching for situations in which people might have had some kind of data exposed.

‍

So there are good data scanners out there that will find we haven't really talked about exactly what the data is, but if it's an address or Social Security number, phone number that you can classify for those kinds of things. So you need to start looking through all the logs. Fortunately, it was only running for a weekend, so it shouldn't be that much.

‍

And then finding any customer that was possibly affected. 

‍

Khush: Yeah. 

‍

Alex: Um, and then you know, you're, you're not gonna be able to do an announcement for quite a while now. Um, but you definitely need the CEO to tell everybody, "Hey, nobody is allowed to say anything publicly without me personally approving it, and I'm not gonna personally approve anything until the general counsel signs off and probably outside counsel."

‍

Khush: That makes sense. There are two follow-ups that I want to know your thoughts on. One is, um, you now know that the real customer data was in the training set. 

‍

Alex: Yeah. 

‍

Khush: Every hour you spend confirming scope is an hour customers don't know. At what point does the investig- investigation itself becomes a liability?

‍

Alex: Oh, I don't think... I mean, the s- system is down. We're not exposing further. Okay. I don't think this-- it becomes a liability until you have a legal responsibility to disclose. And the legal responsibility is based upon what you know. I mean, this is a, a, a game unfortunately less reputable companies than us at Buzzmatrix, which we're a very reputable company.

‍

We wouldn't play this game, right? But you know, those like forty-eight hours are based upon who-- what you know and what information you're able to gather. Um- but I don't-- I would not think of it that way. I think at this point, since we know for sure that customer data was exposed to this company, was pu-pulled into this, that it's going to be a big deal, and, and we need to have a real incident response firm.

‍

Um, the other reason we need a third-party firm is we're gonna need them as a, you know, first and non-testifying expert that's engaged by our outside counsel to start doing forensics inside of that company, right? And that we're gonna have to approach and have probably our CEO and general counsel call this vendor and say, "We're sending Mandiant or whomever.

‍

They're gonna be looking at all of the logs there and figuring out how this data got in here, um, because we're gonna need to have a forensic report that we can reply on in future litigation, as well as to figure out for ourselves how this mess-up happened." 

‍

Khush: That makes sense. 

‍

Alex: And, and also make sure they're not using that, that model anywhere else.

‍

Um, which hopefully this model was built just for us, and they're not, now vendor... You know, they haven't uploaded this to Hugging Face, in which case this turns into a real disaster. 

‍

Khush: Yeah, for sure. Now we are thinking about a supply chain risk for many more companies than just ourselves. There's another factor that I want to get your input on.

‍

The vendor trained the model and holds the logs. If they're uncooperative or slow-walking your audit and your investigation, what leverage do you actually have, and what does that tell you about how AI vendor contracts need to be written going forward? 

‍

Alex: Yeah, I mean, so part of the leverage is just what's in the contract, although in the end, um, if you're in a contract dispute, things get incredibly slow, right? Um, and so, you know, I think, y-you know, part of your leverage is the reputation of this vendor is gonna be very much d- based upon their ability to help us through this incident. If it comes out that the incident... No matter what, they're pulled in. They're part of it. And probably what I'd explain to their CEO and their CISO if they have one: "No matter what, you guys are showing up in our write-up. What we'd like to say in our write-up is..." What's the name of our vendor? Do we have a made-up name? Is it- 

‍

Khush: Let's say, um, Vendor Z. 

‍

Alex: Okay, Vendor Z. It's a great name, guys. First off, I think your naming's fantastic for Vendor Z. , um, really cutting edge. , what we'd like to say in our write-up is that Vendor Z was our partner in this and worked side by side with us to help make sure that customer data was protected.

‍

What we'd hate to say is that we're doing our best even though Vendor Z has been resisting our ability to do this, which not only creates more legal risk for you, but probably completely destroys your market to selling your services to companies like ours. I could do that with one blog post. I'd really rather not do that.

‍

I think that's probably a, enough leverage to get them to help us through this, because I would say as an experienced CISO, this could be a learning opportunity for you, just as it's a learning opportunity for us, and we can all come through this together better. If you make this hard, then we will eventually get those logs.

‍

We will sue you, and we will get access to them, or they will come out in the litigation by states' attorney generals and such, in which case we all lose. 

‍

Khush: Hmm. 

‍

Alex: All you guys are doing right now is making our lives harder to, to help protect you. Why don't you get over that, and let's work together to help keep people safe, to keep their data safe, and to protect both of our companies.

‍

Khush: I love this. I don't think there is any masterclass out there which can help us learn from CISOs like yourself and your critical thinking hat and how you think through scenarios and how you think through all the repercussions and your relationships and what you offer and also what you take from your vendors, et cetera.

‍

This is really good. 

‍

Alex: Yeah, I mean, I think in the end, it... You have to make sure part of the problem you have in these situations, I've, I, I mean, I've done a lot of incident response, right? As, a bit as a CISO, a lot as a consultant, right? I've worked on dozens and dozens of incidents. And you see an over-legalization of incident response, and I, I obviously I'm calling it...

‍

I'm- I've already engaged the general counsel and outside counsel and such. But having outside counsel there and the general counsel to both maintain privilege but also to give advice is different than letting them drive the actual decisions. Um, and I see way too many situations in which the lawyers are driving the decisions.

‍

And in the end, for the vast majority of companies, if you have a cyber incident, whether a full breach or data, you know, this might be considered a da- data spill, kind of an AI data spill or some kind of privacy scandal, the litigation risk is almost never existential, right? The amount of money you will lose in litigation is almost never so much that you will go out of business.

‍

What is this existential is the trust of your customers. And, and that is something that I would explain to this other company, as somebody who has seen this before many, many times. You guys need to... I understand your lawyers are telling you not to cooperate with us. But what you should tell them is that you're less concerned about specific litigation, which, by the way, our lawyers will destroy you guys in court, so that, that you, you have a litigation risk on both sides here.

‍

Um, but t- putting that aside, what you should be more concerned about is acting in a way that at this moment you are taking steps to do the right thing, both for your direct customer, which is us as a company, and your indirect customer, which are the people that we support and whose data we have. And if you're seen as doing the right thing, then in the long run that will be way better for your reputation, and that is the actual existential risk you have to worry about, not litigation risk.

‍

Khush: Hmm. Fast-forward to three weeks, the bot has been taken offline. Good. A full audit of the training data is underway. Legal is reviewing notifications, um, obligations, et cetera. Looking back at the full arc, the Friday launch, the hallucinated outputs, the training data, what was that single decision or non-decision that put Buzzmatrix on this road?

‍

Alex: Yeah, I mean, so the... There's a couple of big picture things. One, like I said, the launch calendars know i- if, if we're a company where people can just get user data, give it to a vendor without any kind of oversight, and then launch a public service without any oversight, then we obviously are lacking some serious internal control.

‍

So those are two things that should be effectively impossible to do technically without the security team knowing, especially the data access, right? If your employees are just able to grab possibly gigabytes or terabytes of customer data and then give it to a vendor and the security team never notices, then that is a, that is an internal control failure because that-- there's no difference of what that looks like f- when an attacker does it versus your marketing team in this case.

‍

In this case, the marketing team was unfortunately one of the adversaries. Um, so I think there's an internal data Control failure. Um, there's the, the launch failure, which is a big one because this would also have been caught if you had a launch calendar, if you had a security review. This seems like a pretty simple thing that you would have done.

‍

Um, and then, you know, you talk about AI governance. I, I think like at this size, you're not gonna have like a big AI governance council, but what you-- but there is obviously a big failure of just the overall culture, right? Hmm. That people are not paranoid enough, that they're not thinking about security or safety.

‍

Perhaps they're still thinking like we're a twenty, thirty, forty-person company working out of a garage, and we're not a four hundred person company. Buzzmatrix is an important company, right? People rely upon Buzzmatrix. The, the, the, the brand of Buzzmatrix means something. Um, and, you know, Jen from marketing should, should know that more than almost anybody else, right?

‍

Um, and I think that lack of paranoia is a big deal. I'd also... You have to think about, as a security team, did we do something to fail in that not just not having the controls, but perhaps we were not easy enough to work with, that people didn't come with us. So, you know, not knowing the history here, perhaps it's just the team is too small and we don't have those processes yet and the company's too, too young.

‍

Or perhaps, you know, either under my leadership or previous leadership- Hmm ... the team was prickly and difficult to work with, and people avoid, right? Like th- this is always the line you have to walk in as security leadership is you have to be paranoid. You have to think about all the crazy things that can happen, and then you also have to enable the business to make money, right?

‍

Um, and so, if you create a situation where people are like, "Oh my God, I don't want to talk to security," then they will eventually work their way around your controls. Um- And, cause these kinds of incidents. And so you wanna make sure to build a culture where people are paranoid, and they also feel like they can engage with security without it becoming a huge problem for them.

‍

Khush: Got it. And such a great point of reflection, too. Like, just being able to remove yourself from the situation and think from the angle of the business. 

‍

Alex: Yeah. 

‍

Khush: Do they have everything they need in terms of policies and guidance and culture, and have we built the right relationships with them? That's a great point.

‍

Alex: Yeah. And you might-- Y- maybe you'll go back to the CEO and say, "Okay, great. If, if, you know, we're gonna do this launch calendar, and I've just estimated, though, for us to do a good enough job and then not slow people down, which we don't wanna slow people down, this is how I'm gonna have to staff it," right?

‍

Al- almost certainly I have seen over and over again of a company that size, when you have one of these incidents, it becomes a big wake-up call, and you end up significantly increasing the size of, of something like a product security team so that you can have those relationships, and you can support those people at the speed at which they wanna move.

‍

Khush: So the moment of truth. 

‍

‍Alex: Okay. 

‍

Khush: Based on everything you have just walked through, do you think the scenario is based on a real incident, or is, is it something we constructed? 

‍

Alex: I mean, it's, it's-- I think it's highly realistic. It could definitely happen. I would not be shocked if it ends up happening. I haven't heard of it, so I'm gonna say you constructed it out of the parts of, of incidents that perhaps would-- I wouldn't be shocked if it has happened. I just haven't heard of this one. 

‍

Khush: Well, the scenario is actually fictional. 

‍

Alex: Okay. 

‍

Khush: We have heard of certain things which we cannot say more of. Yes. But it's definitely fictional. 

‍

Alex: Yeah. Um, but a very realistic... I mean, certainly, I, I've heard a number of situations where people roll out, like, AI chatbots, and they end up with, with problems.

‍

It's usually 'cause of much more traditional prompt injection such. AI systems love it when you just ask them nicely to do things. And, but for the training set, that's a, a, an, um... Yeah, that's an interesting wrinkle. I haven't heard that one yet, but totally realistic. 

‍

Khush: Alex, now we are getting to the next segment of the show, which is called Off the Table. It's our chance to ask you some off-the-cuff questions. One question security should ask before any AI tool goes anywhere near customer data. 

‍

Alex: Oh. Well, okay. So I mean, the biggest question is, is there a differential in a privilege differential between the data it has access to on the back end and the front end, right?

‍

Almost all AI systems, you cannot trust them on a privilege differential. If, if they have access to data that the end user does not have access, that is not safe. You, you have to do full impersonation all the way through. That the only way you can safely do this is if you create a completely new context window, a completely new sometimes container.

‍

You instantiate context and identity within that, and then you pass through the credentials of that user, and then you have deterministic access controls on the back end that are deciding what data you have access to. If any-- even the, the smartest foundation models, those models have access to all customer data, and then you tell them in a prompt, "Make sure to only tell this user s- only give Sally Sally's data," you're toast because they are not built to be able to do that access control.

‍

Way too many companies thinks you can do that. That's the most important thing is like, is there a privilege differential or is this AI system impersonating the, the end user deterministically? If so, that doesn't mean you're out of the woods, but it means that at least your controls are deterministic and not in this AI model that you can't understand.

‍

Khush: That makes sense. Alex, you survived the tabletop. 

‍

Alex: Okay, great. 

‍

Khush: Thanks for being here.

‍

Alex: I'm not fired by Buzzmatrix.

‍

Khush: Oh, no. You're Alex Stamos. I think it was a wonderful tabletop exercise. Thanks. Our users and our listeners will have so many data points and so much insights from your experience and the way you explained everything because as I said before as well, nothing like this exists out there.

‍

We, we go through it real life with incidents, learning from our CISOs, but never in a setting like this. So- Yeah ... thank you so much for being here and going through this with us. 

‍

Alex: Thanks for having me. Yeah.