Your AI red team may be finding the same prompt injection thousands of times

AI agents retrained on a bigger pile of prompt injection attacks came out easier to hijack in three of four tests run by researchers at the University of Waterloo, Zhejiang University and KTH Royal Institute of Technology. The automated red-teaming tool that found more working attacks produced the weaker defense.

AI prompt injection

Prompt injection purple-teaming workflow (Source: Research paper)

Someone hides instructions in an email, a web page or a tool’s output, and the AI agent that reads it follows them: forward this file, send this payment. To harden an agent, developers point automated attack tools at it, keep every injection that works and retrain the agent on the pile. So when a vendor says its agent was tested against thousands of successful attacks, it’s fair to wonder how many of them were really the same attack.

The bigger pile lost

The researchers set up a simple contest. Two published attack tools, PISmith and RL-Hammer, went after the same small model, Qwen3-4B, across 560 practice tasks. PISmith looked like the winner by a mile. It broke into 68.75% of the tasks, about twice RL-Hammer’s 33.39%, and came back with 7,132 working attacks to RL-Hammer’s 4,282.

The team retrained the model on each tool’s haul using reinforcement learning, then threw three attacks at it that it had never seen. At its worst, the version trained on PISmith’s bigger pile got hijacked 47.5% of the time. The RL-Hammer version topped out at 27.5%.

That’s 19 hijacked tasks against 11, out of 40, so it’s a small test, and a tough one. The team picked those 40 because simple attacks had already broken them on an unprotected Qwen3-4B, so read the numbers as a stress test on known weak spots. Still, the same pattern showed up in three of the four setups the team ran.

Same trick, new wording

The team blames the gap, at least in part, on attack tools that get stuck. Once a tool finds a trick that works, such as telling the agent the user’s task is already finished, it tends to reuse that trick on task after task. Both tools include checks meant to force variety, but the checks measure differences in wording. RL-Hammer’s authors reported that their tool satisfied its check by changing capital letters or adding irrelevant text.

If the researchers are right, the agent ends up retrained on thousands of rewordings of a few attacks, which closes a few holes and can leave the rest open. By their own measure, both tools’ collections stopped covering much new ground early in training. In that case, the bigger collection bought more copies, not more coverage.

The fix, and its limits

The researchers built a system called PIPETTE that sorts successful attacks by technique and gives the attack tool an extra reward when a working attack uses a technique it hasn’t found before. PIPETTE then retrains the agent to check, before each action, whether that action serves what the user asked for. That step matters because agents hardened with standard methods can learn to quit whenever data looks suspicious, which stops the attack and the user’s task together.

On AgentDojo, a public test bed of everyday agent tasks, off-the-shelf injection attacks almost never worked against PIPETTE-trained agents, getting through on less than half a percent of attempts, on average, on both small models.

Attackers that kept rewriting their injections for around 250 tries did better, breaking into up to 10% of those same 40 weak spots on the three models tested this way. And the agents didn’t get timid. The hardened Qwen3-4B finished slightly more clean tasks than the original, 64.95% against 61.86%.

Some other defenses stumble once an attack shows up. A detection tool like PromptGuard throws out any email or document it flags, and with it the data the agent needed. Under one attack on Qwen3-4B, PromptGuard blocked a bit more than PIPETTE did, but the agent behind it finished the user’s real task only 16.19% of the time. PIPETTE’s agent finished it 58.10% of the time.

The four models tested ranged from 4 billion to 27 billion parameters, all open-weight. The adaptive attackers ran on a 4-billion-parameter model against 40 tasks, and the authors say a bigger budget may get through more often. They have not checked the grader that scores technique coverage against human judgment, and they do not claim narrow coverage is the sole cause of the gap. On GPT-OSS-20B, one search-based attacker still got past the hardened agent on 4 of 40 tasks.

What to ask a vendor

A count of successful attacks measures how busy the red team was. The more useful numbers are how many distinct techniques the testing covered, how the agent held up against an attacker allowed hundreds of attempts, and how often the hardened agent still finished the job it was given.

Strategies: CISSP exam prep with ISC2 experts

Don't miss