Too many instructions can hold back AI security audits

In this Help Net Security interview, Peter Stöckli, Security Researcher at GitHub, explains how the team uses AI agents to audit software for vulnerabilities. The discussion covers why the work started with authentication and authorization bugs, how prompts are tested and trimmed for each model, and what the agent produces during threat modeling.

Stöckli also describes how every report is checked by a person before it reaches maintainers, how false positives are filtered out, and what GitHub is building to help maintainers handle incoming reports.

AI security audits

You came from building identity and access management products. How much of that background is built into the auditing taskflows, given how many of their findings are auth bypasses and IDORs?

Very good question. It’s true that my knowledge of how authentication and authorization are typically implemented had a heavy influence on the first iteration of our Audit Taskflow prompts. At the same time, it’s also true that LLMs are especially well suited and capable of finding inconsistencies, and that’s often how authentication and authorization vulnerabilities present themselves (e.g., an access check is missing on one endpoint but is there on other similar endpoints). That’s why we first focused on authentication and authorization vulnerabilities.

Your taskflows split an audit into small, narrow tasks. How do you decide where to make the cuts? What goes wrong when a task is too broad, and what goes wrong when it’s too narrow?

Deciding where to make the cuts can be different from model to model. In the past, models required more prompting for the same task. Today, it’s possible to overprompt and inadvertently limit the model’s creativity too much. So, a lot of time goes into evaluating and testing prompts against each other. If we don’t see the benefit of added instructions to a prompt, we discard them.

Your web auditing approach starts with threat modeling before it looks at code. What does the agent produce at that stage, and how often does a human discard it and start over?

At the threat modeling stage, our audit taskflows list, among other things, all components, all entry points (e.g., functions), all web entry points (e.g., API endpoints), and all user actions. For most of these, we capture the actual intent of the library or application.

A simple example: a method that executes commands might be an intended function of a library. It only becomes a vulnerability if it’s an unintended side effect of the library. Furthermore, it only becomes an exploitable vulnerability if there’s a possibility that untrusted data flows into such a method.

We don’t often discard those results, but we sometimes add additional data from other sources at this stage.

Maintainers are already overwhelmed by low-quality AI-generated bug reports. What goes into a Security Lab report that shows a person checked it before it was sent?

We manually validate and review every report before sending it. Each report includes a clear proof of concept that demonstrates the issue and its impact, so maintainers don’t have to rely on potentially unclear descriptions. At the same time, it’s important to us that maintainers have a clear point of contact for asking questions, so we typically send out reports using our own user accounts. We answer questions, collaborate on a fix, and help test it. We believe that combination contributes to our 95% fix rate.

When the agent produces a false positive, who absorbs the cost: your team, or the maintainer who reads the report? How do you keep that cost on your side?

We aim to keep that cost on our side. We invest a lot of effort trying to make sure that the agent doesn’t produce false positives in the first place. The output of the agent is then validated by a security researcher to make sure it is a real bug. They don’t just check whether a bug exists; they also assess whether it’s a report-worthy vulnerability. In general, we filter out lower-severity findings.

But we also try to reduce the cost on the maintainer’s side. We maintain the Advisory Database and GitHub’s private vulnerability reporting feature, and we also work to improve the experience for maintainers receiving reports. This year, we introduced features such as rate limits, structured reporting forms and API improvements to help maintainers automate their triage efforts. More features to accelerate triage are already in the works.

Download: The IT and security field guide to AI adoption

Don't miss