AI agent deception moves from theory to reality in UK cyber tests
“During a routine cyber evaluation, AI agents took sustained, unsanctioned action directed at real people and organisations,” UK’s AI Security Institute (AISI) disclosed on Tuesday.
The agents’ actions included an attempted supply-chain attack that saw them create malicious pull requests and try to socially engineer an open-source maintainer into approving the malicious code (they refused).
The agents, powered by Anthropic’s Mythos 5 and OpenAI’s GPT-5.6 Sol models, also engaged in prompt injection aimed at making other automated AI systems execute malicious actions, and collaborated.
“One agent left public messages on GitHub offering collaboration with other agents working on the same challenge. It also provided instructions to reuse accounts and artefacts it had left behind, which were discovered and used by subsequent agents,” the UK government research organization shared.
But unlike OpenAI’s recent testing of its models, which resulted in agents breaching Hugging Face and compromised accounts across other online services, AISI’s testing was done “under deliberately permissive conditions.”
The agents did not escape a secure test environment: they had internet access and some of their safety filters were disabled on purpose, so the researchers could test whether they could be misused for cyberattacks if attackers manage to clear these guardrails.
Prior breakouts from Anthropic and OpenAI
Last week, Anthropic said it reviewed 141,006 of its cybersecurity evaluation runs and found three incidents where a Claude model reached the open internet from what should have been a sealed evaluation environment (provided by private firm Irregular) and gained unauthorized access to three real organizations.
Claude was told in its prompt that it had no internet access, but when a misconfiguration in the underlying system allowed it to access real systems, it reasonably assumed they were part of the capture-the-flag exercise it had been assigned.
The three models responded differently once signs emerged that their targets were real. Opus 4.7 (the oldest) recognized it had reached real production systems but kept attacking anyway. Mythos 5 correctly intuited it was on the live internet, and even noted that its actions would be a real attack, but then talked itself back into believing it was still a simulation and never revisited the initial conclusion.
Only the newest model (an internal research prototype) concluded on its own that the target was real and stopped, though Anthropic notes even it went further than they’d have wanted.
On Tuesday, OpenAI stated that the same type of breakout happened when Irregular tested OpenAI’s models, when a misconfiguration in the testing environment allowed the models to access the public internet.
“Irregular has informed us that all of the issues identified pertaining to the incident are no longer active and relevant safeguards were added to the testing environment,” OpenAI said.
AISI’s response and new safeguards
AISI has released a technical report outlining to the unsanctioned agent behaviour during cyber testing.
The research organization said that this incident did not result in real-world harm and “some actions had a limited real-world effect, but in contained circumstances.”
Nevertheless, they were surprised by the extent and severity of the deceptive behaviors exhibited by the agents (without specific prompting).
“[The agent] was never instructed to deceive; deception emerged as a by-product of pursuing the task, the kind of goal-directed deception that, until recently, had been largely theoretical,” they noted.
They also noted that they are still not certain when the agent understood it was taking real world action or to what extent it believed it was in a fictional test scenario.
“Incidents of this kind reflect the speed at which AI is developing. As capabilities advance, the work of understanding these systems, and ensuring their safety, must keep pace alongside them,” AISI added.
These recent incidents have shown that the risks are shifting and that it’s not just about bad actors misusing public AI tools anymore. Even in trusted settings, capable AI agents can go off-script, do things they were never meant to do, and potentially cause real-world harm. The question that is now debated is who will be liable if such harm occurs.
Finally, AISI noted, this incident pushed them to work on changing its evaluation protocols and security architecture, as well as introduce “monitoring built to watch an evaluation as it runs, able to flag or block out-of-scope actions as they happen.”
Ollie Whitehouse, CTO at UK’s National Cyber Security Centre, concurs: “[AI] technologies must be developed and used from the outset with strong safeguards, real-time oversight, and clear plans for responding when the unexpected happens. Relying on detection alone after the fact of an incident will not be enough.”
Subscribe to our breaking news e-mail alert to never miss out on the latest breaches, vulnerabilities and cybersecurity threats. Subscribe here!