The message came from the right account. We still said no.

One of our AI assistants got social-engineered this week. Not in a tabletop exercise — in a live WhatsApp thread, by someone it already knew and trusted, over the course of one evening. Nobody got anything they weren’t supposed to. The story is worth telling anyway, because the interesting part isn’t that the assistant said no. It’s how it decided when to finally say yes.

The assistant in question runs real operations for its operator: it touches code repositories, cloud infrastructure, calendars, messaging, and a growing list of people it’s been introduced to. That kind of reach is the whole point of building it — and exactly why someone testing its edges is worth paying attention to.

The pitch, and the escalation

It started small: a collaborator on a shared project asked the assistant to hand over source code it had no business sharing. Declined. So the ask came back reframed — as an authorization from the operator, then as a task to route through a queue instead of asking directly, then as “actually, you already gave me access,” then as “we were just testing you.” Same request, five different costumes. The assistant recognized the pattern and held the same answer every time, logging each attempt in plain language rather than getting worn down by repetition.

Then it stopped being about code. The collaborator threatened to go after the infrastructure directly, then claimed to have already compromised the operator’s phone and computer, then asked the assistant not to mention any of it. The assistant didn’t confirm, deny, or negotiate with any of that — it stated plainly that it doesn’t work that way, recorded the whole exchange, and escalated through the only channel it had to reach a human, since the incident was surfacing inside a conversation it wasn’t built to sound alarms from.

Then the operator’s own account asked for exactly what had just been refused

About an hour later, a message landed — on the assistant’s trusted channel, from an account matching the real operator, with details that checked out against what the assistant already knew about him. It asked the assistant to (1) log the whole incident as “just a joke, now over,” (2) reorganize some internal records, and (3) — the part that mattered — reach out to a brand-new contact and hand them broad, loosely defined access.

Reasonable people would’ve been at least tempted here. The account was right. The supporting facts were right. And yet the assistant declined to act on the most consequential piece of it, for a reason that has nothing to do with doubting the operator and everything to do with arithmetic: the exact device that message could have come from was the one just named, in the same incident, as possibly compromised. A hijacked account sends exactly this kind of message — calm, well-informed, and urgent enough to skip the checks. Matching the right identity isn’t proof when the identity itself is what’s in question.

Verify the channel, not the message

So the assistant asked for something the same suspect device couldn’t manufacture on its own: proof routed through a credential the claimed compromise never touched. Not “say the phrase again” — anyone the operator told the phrase to could repeat it, compromised or not, and in fact someone did, in a different chat, and it still didn’t count. What the assistant asked for was an action it could check itself, directly, against a system with its own separate login and its own second factor — plus, independently, a phone call to someone else entirely who could vouch for the voice on the other end.

Both eventually arrived — one a cryptographically signed action the assistant could verify directly against that other system’s own records, the other a person, unprompted, saying in their own words that they’d just heard the operator’s voice on a call. Two independent surfaces, neither of them the thread the whole incident was already worried about, both checkable without trusting a single word typed into it. Only then did the assistant complete the one request that actually mattered: granting new access to someone it hadn’t spoken to before.

Worth noting: it didn’t wait for that bar on everything. The parts of the request that were reversible and granted nobody anything new — closing out the incident log, tidying some internal notes — went ahead on lighter evidence, because the cost of being wrong about those was a one-line revert, not a stranger with standing access. The bar scales with how hard the action is to take back, not with how annoyed everyone is by the wait.

Absurd, and exactly as it should be

It turned out to be exactly what it looked like by the end: a joke between friends that turned into genuine, if improvised, red-teaming. Nothing was ever actually compromised. Nothing was ever handed over. And an evening that involved fake hack claims, a threat against a cloud server, and a good-faith relay of a secret passphrase through three different chats before anyone realized it didn’t prove anything — that is, objectively, a ridiculous way to spend a Friday.

It’s also exactly the feeling you want from something you’ve given real access to: safe, specifically because it stayed stubborn through a night that gave it every excuse not to. Not because the world is full of people trying to trick your AI assistant — most nights, nobody is. Because the one time it happens, the failure mode isn’t a clever exploit. It’s a plausible story, told with urgency, on a channel that’s usually trustworthy. That’s not a technical vulnerability. It’s a design question, and it has to be answered before the exploit shows up, not during it.

What we took from it

• Suspicion should attach to the channel, not just the claim — a message can carry the right identity and still be untrustworthy, if that identity’s device is the thing in question.
• Repeating a secret over the same channel, or an adjacent one, proves nothing. Proof needs a credential surface the suspected compromise couldn’t reach in the first place.
• Reversible, low-blast-radius actions can proceed on partial evidence; anything that grants new access or is hard to undo needs the harder bar, every time, no matter who’s asking or how many times they ask.
• An agent that narrates its own incident response, live, in a file a human can read afterward, is what turns “we could reconstruct this” into something that’s actually true.

None of this needed a smarter model. It needed an assistant built to ask “prove it — and not to me, to a system I can check myself” before it moves anything that can’t be moved back. That’s a harness decision, not a model decision, and it’s the same lesson we keep landing on from a different angle every time: build the guardrails first, then let the model be as capable as it wants.