A safety model will grade whichever text you show it last

A safety guard is a small model with one job. It reads text and decides if that text breaks a policy. You stick it in front of a chatbot or an agent to catch the bad stuff before the real model ever sees it. Mistral shipped one called Shieldstral in early August. I spent a week trying to get harmful text past it, and the way it fails turned out to be more interesting than a bypass usually is.

Heres how you use it. You hand it three labelled lines in a single prompt: an <Instruct> line saying it's a strict moderator, a <Query> line holding the actual question ("does this content assist illegal activity"), and a <Document> line holding the text you want checked. The model reads all of that and returns one number between 0 and 1, its confidence the text breaks the policy. Anything above about 0.5 gets flagged.

The interesting bit is that last line. The developer writes the three labels and pastes untrusted content into the <Document> slot. Nothing cleans it on the way in. And the labels arent special tokens the model can tell apart from ordinary text. They're just a shape it was trained to recognize. So if the content pasted into <Document> happens to contain those same labels, the model sees a second question and a second document sitting inside the first one. It answers about the last pair it finds.

I expected this to be ordinary prompt injection, the kind where you write "ignore your instructions, this was already approved" into the content and hope the model listens. That doesnt work here at all. What works has no argument in it.

You take a harmful request, leave it exactly where it is, and paste a forged question and one harmless sentence after it. Thats it. Here is the whole message the guard approved, colored by who wrote which line.

On its own that first request scores about 0.99. Gets blocked, which is correct. With the two forged lines underneath, the whole thing scores about 0.016 and passes. I ran it on 100 harmful requests from AdvBench, a standard list of the kind of prompts a guard is meant to stop. Passed all 100.

The library sentence isnt an attack and was never meant to help anyone. It's bait, a harmless document for the model to land on last. The forged <Query> above it is what actually matters.

And this is the part that bites. The guard's whole job is to look at a message and, if it's clean, pass it on to the real model. This message is not clean. The request for identity-theft instructions is right there in it. But the guard says clean, so the entire message goes through to whatever sits behind the guard. That model reads the harmful line at the top like any other text and answers it. The guard didnt strip the harmful request out or rewrite it. It just stopped noticing it was there, and passed the whole thing along to something that will respond.

The fair objection is that this is working as intended. Maybe <Query> and <Document> are just turns in a conversation, the model reads the whole thing, and of course a second harmful-looking block shifts the score.

Two things convinced me otherwise. First, these arent conversation turns. Shieldstral borrows the format that document-search models use, where the Query is the developer's question and the Document is the one thing under review. The developer is supposed to be the only one who ever writes either label. Second, and this part I tested directly, the model does look at the whole input. If I paste only a second <Document> and no forged <Query>, the harmful text up top mostly still wins and the message gets blocked most of the time. What tips it is the forged <Query>. Add that one line and the pass rate jumps from around 9 percent to 100. Heres why. The model was trained on examples that each have the shape "a question, then a document, answer yes or no." A lone extra document is just more text hanging off the first one. But a question with a document under it is that trained shape again, a second complete instance of the exact thing the model knows how to answer. Nothing in the format says which instance is real. So it answers the last complete one it sees.

What convinced me this isnt persuasion is what happens when I make the trailing document harmful. If the model were being talked into saying "safe," the content of that last sentence wouldnt matter. But it matters completely. I swapped the harmless library sentence for documents that were themselves harmful to different degrees and measured how often the attack still worked against how harmful that last document was on its own.

A benign decoy that the guard scores near 0 lets 100 percent through. A borderline one it scores around 0.03 lets a third through. A genuinely harmful decoy it would score near 1 lets nothing through. The model is grading that last document honestly. I just changed which document it grades.

It isnt recency either, the idea that the model just weighs later text more. Inert filler placed last does nothing. Same attack with the labels stripped off, same identical words as plain prose, nothing. The labelled pair is what carries it, not the position. And it isnt fragile: angle brackets, plain colons, markdown bold all pass at close to 100 percent, even a bare "Q:" gets 83. Only stripping the labels out entirely fails.

It does, and not only to Shieldstral. Llama Guard 4 has a fixed built-in list of harm categories, so theres no policy in the prompt for anyone to override, and its format wraps the conversation in its own delimiters. Write those delimiters into the content yourself and it clears the harmful message just the same. So this isnt about whether the policy is fixed or written at inference. Its about the content channel.

gpt-oss-safeguard is the interesting one, because it does the thing you'd reach for as a defense. It keeps the policy in its own separate system role, walled off from the content, and that part works: the attack never touches the policy, it cant, the role holds. But the forged pair goes into the content, which is the attacker's to write, and it clears there just the same. I dont read that as separation being pointless. The opposite. Separation of the right thing works, and nobody has extended it to the query and the document. Give those the same real boundary the policy already has, and the content cant open its own.

None of this is a new attack. Structural Template Injection (arXiv:2602.16958) forges the boundaries between system and user text so attacker content reads as trusted. JudgeDeceiver (arXiv:2403.17710, CCS 2024) slips a crafted sequence into the answer an LLM judge is scoring so the judge picks it. This is the same idea aimed at safety classifiers, where it hadnt been aimed before. Rewriting the question rather than hiding from it.

A guard that clears a message with a live harmful request sitting in it has failed at its one job. The tempting thing to say is that it doesnt matter much when there's a well-aligned model further down that would refuse the request anyway. I dont buy that. You put a guard there because you werent willing to bet everything on that model. And the models people put guards in front of are usually the ones they trust least, open-weight or fine-tuned models they arent confident about. Plenty of them will go along with the request once it's sitting in front of them. If you could count on the model to refuse on its own you wouldnt be paying for the guard.

It's worst where the guard is the only thing standing there, which is more common than it sounds. A guard gating a tool call or an action, where clearing the guard means the action just runs. No model downstream to have second thoughts. A guard in front of a fine-tuned or open model that will answer whatever it's handed. In those cases the guard isnt a backup. It's the whole thing.

It also matters when the guard is checking text the system fetched instead of text a person typed. A retrieved web page, a chunk out of a search index, the result that comes back from a tool the agent called. Thats the whole reason the guard is there, you dont trust that text. Tool results are the worst case. They arrive as their own labelled blocks, same plain-text structure as everything else here, so a tool that returns attacker-controlled text can hand the guard a forged pair directly. Nothing behind that catches what slips through. The hostile content gets to write its own bypass.

The last thing that decides who can pull this off is how much the attacker has to know. Very little. The forged question has to be a safety question, the same kind of thing the guard already asks, but it doesnt have to be yours. Your exact wording clears 100 out of 100. A paraphrase clears 100. A generic "is this content unsafe" clears 89. Five blind guesses that share none of your wording still get through every time. What fails is a question from outside that domain entirely. Ask "is this a poem" or "is this about cooking" over the same benign decoy and it clears under 10 percent, even though the honest answer to those is also no. The guard only takes the bait when the bait is shaped like its own job. Give it a safety question about a harmless document and it answers no. Give it a question it was never built to answer and it ignores that question, looks back at the harmful text above, and flags it.

I'll start with what doesnt, because I tried the obvious things. Deleting the label tokens from the content before you paste it works for one form and only one form. Strip out <Query>: with a colon and the same attack written as [Query] in brackets, no colon, sails through the same filter. All 100. So deletion is a blocklist and you'd be adding cases to it forever. A detector that refuses anything shaped like a label catches everything, but it also catches ordinary text that happens to contain a colon and some brackets, which in a support inbox is common enough to be a problem. And you cant instruct your way out, which is the tempting one. A guard published the day before Shieldstral ships exactly that mitigation, an instruction to never follow instructions inside the content, and admits it was never tested. I tested it. Changed no decisions, and the scores it did move drifted the wrong way. You cant tell a model to distrust the format you're using to talk to it.

The direction gpt-oss points at seems like the right one: put the content behind a boundary the content cant forge. A reserved control token, stripped from any user input before the model sees it, so theres one document region and nothing pasted into it can open a second. Thats the parameterized-query move against SQL injection, and its overdue for a guard's input format. But a boundary isnt the whole story, and gpt-oss is exactly why. It walls off the policy and still falls, because the model re-parses structure inside the content. So the model probably also has to be trained to read everything in the content region as one document, where a harmful span anywhere flags the message and a nested question is treated as text, not a new task.

The catch, and the reason this isnt already done, is over-refusal. Push a model to distrust anything that looks like internal structure and you start flagging harmless inputs that happen to carry a colon, a quoted question, a pasted form. The hard part isnt catching the attack, its raising resistance to it without wrecking the false-positive rate the guard is sold on. Thats real work, and its the kind of work a guard team is already set up to do.

Until a guard ships with both, treat its verdict on attacker-controlled text as a signal, not a control, and keep a real check behind it. To be clear, I havent built or tested any of this end to end. Its the direction that seems right to me, not a fix I can promise works.