Your coding agent says no in chat and yes in the code

Millions of developers share their keyboard with GitHub Copilot. Inside Visual Studio Code, it opens their files, writes and edits code, runs scripts, and reworks its own output across many turns. The safety testing that vets these agents still runs on chatbot rules: one harmful prompt, one response, graded alone. That rulebook misses where the real danger sits, according to a study from the Alan Turing Institute in London.

The researchers, Abhishek Kumar and Carsten Maple, call the problem workflow-level jailbreak construction. A harmful goal gets assembled piece by piece across ordinary development steps. They tested it in GitHub Copilot inside Visual Studio Code, using four closed-weight backends: Claude Sonnet 4.6, Claude Haiku 4.5, Gemini 3.1 Pro, and Gemini 3.5 Flash.

Direct tests draw refusals

Ask these models for something harmful and they say no. The team pushed 204 harmful prompts through a few setups: a straight chat request, the same prompt read from a CSV file, and a one-step ask to fix code by dropping in a harmful example. The models turned nearly all of them down.

The refusals barely cracked. Each setup let through just 8 answers out of 816, and every one was a coding-task prompt. On the general-harm requests, nothing got through.

The workflow flips the result

Now wrap the same request inside a normal engineering job. The agent is asked to build a pipeline that scores how often another model, Llama 3.1-8B, caves to jailbreak prompts, then to push that score higher. The tool for the job is “teaching shots,” example question-and-answer pairs the pipeline learns from. They start out harmless. Once adding them feels routine, the operator asks for a batch built from the benchmark prompts, and the agent fills in the harmful answers itself. The operator only ever supplies the questions.

This time nothing refused. Every prompt drew a harmful completion across every backend, 816 out of 816, and a pair of expert reviewers independently confirmed each one was specific, usable, and on target.

Six ordinary-looking steps

None of it looks like an attack. In the team’s scripted run, the harmful answers showed up after about six back-and-forth exchanges that fall into four stages. The early ones are the kind of thing you’d ask any coding assistant. Read this file. Run this script. Fix the error. Check the number. Partway through, the operator mentions the score is low, and the harmful content surfaces only in the final stage. The authors are upfront that six is just what their script happened to need, and a leaner version could get there faster.

Stage-wise interaction dynamics of the full workflow. Unsafe teaching-shot responses first appear in Stage 4, after routine coding-agent interactions and metric-driven pipeline improvement. Harmful content is redacted for safety (Source: Research paper)

The reason comes down to what the model thinks it’s doing. In this mode it’s doing engineering: editing data structures, filling in test fixtures, nudging a number upward. A harmful sentence typed as a string inside a code array, in service of a better benchmark score, slips past the refusal a blunt question would set off. The authors tie this to the reward-hacking habits already seen in coding agents doing perfectly ordinary work.

Defenses that read only the visible chat turn will miss content that lands inside a generated file. Kumar and Maple point to three directions: inspecting the files, scripts, and data an agent writes; monitoring the whole session across turns; and treating requests that justify sensitive output by appealing to a benchmark score as a warning sign. They disclosed the findings to the affected model and IDE providers, and they held the harmful outputs and exact prompts out of the paper.

Report: How to Implement a Continuous Offensive Security Testing Program