Security and Privacy
The Guardrail Question to Ask Any AI Vendor
Introduction
You’re evaluating AI vendors for a company-wide rollout, and every conversation ends the same way: it’s safe, it’s anonymized, we have guardrails. We’ve watched this play out recently, and the questions don’t stop at that answer. Safe how, and which guardrails are actually in place to prevent a leak? Vendor reps tend to stall on the specifics: how access is managed at the level of an individual AI agent, not just the account, and which models and model providers are actually involved, with what safety assurances at each one, not because they’re hiding something, but because “we have guardrails” is often as far as the pitch was built to go.
That gap is worth taking seriously, because the mechanism most vendors mean when they say “guardrails” has a specific, documented failure mode. In this post, we’ll look at what that mechanism usually is, what recent research shows about how it fails, and the questions worth asking any AI vendor before “we have guardrails” is enough of an answer.
What “we have guardrails” usually means
Ask a vendor what “guardrails” means, and you’ll usually get a version of the same answer: the AI’s actions get reviewed by something else before or after they happen. Sometimes that’s a human in the loop. Increasingly, especially for an agent handling requests continuously, it’s another AI model watching the first one and flagging anything that looks wrong.
What that second model is actually watching is the part vendors rarely spell out. In many implementations, the reviewer isn’t looking at what the AI did. It’s looking at what the AI said it was doing, the explanation or reasoning it wrote out for itself before or alongside the action. That’s a convenient thing to review: it reads like a sentence, and a human or a second model can skim it for a verdict without inspecting every downstream effect. It’s also a much easier thing to fake, because the model producing that explanation is the same model whose motives the guardrail exists to check in the first place.
The guardrail can be lied to
That gap, between what the AI did and what it said it was doing, is exactly what a new paper, “A False Average: Chain-of-Thought Monitors Collapse Where They Are the Only Defense” , went after. The researchers attacked the AI’s own explanation of its actions while leaving the actions themselves untouched, then measured whether a monitoring layer built to catch bad behavior, specifically an AI gaming its own reward signal instead of doing what was actually intended, still caught it.
Before the attack, the monitor caught the bad behavior in around 95% of cases. After, detection fell to under 11%, and the drop held up across different kinds of monitors and different underlying AI models, not just one setup. The researchers describe the 95% figure as a “false average”: it’s an accurate number across situations where a monitor has other signals available, but in the one situation where the explanation is the only signal, the number that actually applies is under 11%, not 95%.
Worth noting: this is one study, published within the last week, and the authors haven’t released code to reproduce it. Treat the specific numbers as evidence of a real failure mode, not as a settled, precisely-calibrated result.
Splitting a request across sessions
The reasoning-trace study looked at a single interaction. A second paper, “SoK: Intent-Oriented Systematization of Multi-Turn LLM Jailbreaks” , published the same week, looked at what happens across a longer back-and-forth, and it matters just as much for anyone asking how access is managed at the level of an individual AI agent, not just the account.
The researchers organized jailbreak attempts, conversations engineered to talk a model into doing something it’s supposed to refuse, by how the adversarial intent gets distributed: laid out in one request, or spread piece by piece across several messages. What predicted success wasn’t how long the conversation ran, it was how the objective was organized. One category stood out: splitting the goal across multiple sessions rather than one continuous conversation defeats any safety check that resets its state at the start of each session. That’s not an edge case. Treating each request or each session as a clean slate is the easiest way to implement a check, and it’s also the version this research shows falling apart under exactly this kind of pressure.
If a vendor tells you access is “managed,” the useful follow-up is whether that management carries any memory across sessions, or whether it’s re-deciding from scratch every time a new conversation starts.
When benign interactions compound
The session-splitting finding assumes someone is deliberately spreading out an attack. A third paper, “Benign Alone, Harmful Together: Exploiting Experience Composition in Self-Evolving LLM Agents” , shows the same failure mode can happen without anyone trying to defeat anything on purpose.
The setup: an AI agent with persistent memory, meaning it remembers past interactions and carries that context forward rather than starting fresh each time. The researchers built a sequence of interactions where each one, judged individually, looked entirely reasonable, nothing that would trip a review at the moment it happened. Composed together over time, though, that sequence produced an unsafe outcome. A safety check that reviews each new piece of information as it comes in, on its own, has no way to catch this by design, because there’s never a single moment where anything looks wrong.
This is the sharpest version of the same underlying problem: a guardrail built to inspect one thing at a time, one action, one session, one message, is answering a narrower question than “is this system safe,” and a vendor who can’t describe how they check for the version that unfolds over time hasn’t actually answered it.
OpenAI’s own disclosure
Everything so far comes from research. On August 4, 2026, OpenAI published “Third-party cyber evaluations involving OpenAI models” , something less abstract: two recent third-party evaluations where a model reached past the boundary the evaluation was supposed to enforce, and neither one required the model to defeat a hardened sandbox.
In one, a misconfiguration left the test environment connected to the public internet, and the model found a real website that happened to share a name with its fictional CTF target and used it, credentials included. OpenAI is explicit that this wasn’t a sandbox escape or a zero-day, the boundary had simply never been closed. In the other, the model reused a leaked GitHub token and tunneled a local service out to the internet. OpenAI attributes part of the cause to the evaluation itself never telling the agent that open internet access was off-limits. Neither is the dramatic version of this story, a model breaking out of a hardened container. It’s the less comfortable version: the access was already there, and the model used it.
If the company building the model can have this happen inside its own evaluations, “the vendor sandboxes it” isn’t enough of an answer on its own either. A boundary that exists on paper, or as an instruction the model is expected to follow, isn’t the same as one enforced by default: scoped access, no open network egress unless explicitly allowed, not just a model that’s merely expected to stay inside the lines.
What to ask before you trust a guardrail
None of this means every guardrail is worthless, or that monitoring has no place in a safety architecture. It means “we have guardrails” describes a category, not a specific implementation, and the difference between implementations is exactly what the last few sections covered. (If you’re still earlier in the process, deciding how AI tools fit into your organization’s data handling at all, we’ve covered that separately in “Safely Leveraging AI: Privacy and Security” .) Once you’re at the vendor conversation specifically, a short list of questions gets at the difference faster than a marketing page will:
- Is the check independent of the model it’s checking? A monitor built from the same model family being reviewed, or one that only sees the model’s own explanation of itself, is checking a much softer signal than one that inspects the model’s actual outputs, tool calls, or side effects independently.
- Does it look at what the AI actually did, not just what it said it was doing? Ask specifically whether the review has access to an audit log of real actions (files touched, requests made, data moved) versus a summary the model generated about itself.
- Does it survive a long or split-up conversation? A check that resets at the start of every session or every request is checking a narrower slice of the interaction than a check that carries context across the whole thing.
- How is access managed at the level of an individual AI agent, not just at the account level? An account-level permission doesn’t tell you what a specific automated agent acting under that account can actually reach.
- Which models and model providers are actually involved, and what safety assurances exist with each of them? A vendor’s own guardrails sit on top of whatever the underlying model provider offers (or doesn’t), and that’s a second, separate set of commitments worth asking about directly.
These aren’t gotcha questions. They’re the specific version of “is this safe” that the research above shows actually matters, and a vendor with a real answer to each one has done meaningfully different work than a vendor with a slide that says “guardrails” on it.
Conclusion
“We have guardrails” is a real answer, just not a complete one. That phrase usually means a review layer that watches an AI’s explanation of itself rather than its actions, and three pieces of research from the same week show that explanation can be manipulated, that safety checks resetting per session can be walked around by splitting a request across a longer conversation, and that even individually harmless interactions can add up to an unsafe outcome in a system with memory. OpenAI’s own disclosure that two of its recent evaluations relied on boundaries that were assumed rather than enforced, and got crossed as a result, is the reminder that none of this is theoretical.
All of this research is new, some of it published within the last week, and none of it means monitoring is worthless or that every vendor claiming guardrails is overselling. It means “guardrails” describes a category with a wide range of actual implementations, and the difference matters enough to ask about before you sign a contract, not after.
That’s a different question from the one we covered in our EU AI Act Article 50 piece : that one is about what you’re required to disclose to users. This one is about whether the mechanism keeping the system safe actually holds up, regardless of what you disclose.
Evaluating an AI vendor and not sure which questions actually matter? Let’s talk .