GitHub Agentic Autofix and the question of who owns a bad AI patch

In this Help Net Security interview, Marcelo Oliveira, VP of Product Management at GitHub, talks about who is accountable when an AI-generated patch fixes one flaw and introduces another. He explains how Agentic Autofix re-runs detection engines and project tests, and what a CISO should change before pointing an agent at thousands of old alerts.

Oliveira also discusses which metrics show a codebase is getting safer, how GitHub picks the models behind autofix, and whether developers lose security skills when they stop writing fixes themselves.

If an AI-generated patch closes one CWE but quietly opens another, whose incident is that: the developer who merged it, the AppSec team that switched the feature on, or GitHub?

GitHub is accountable for the quality of the fixes we suggest, and customers decide what gets merged, just as they would with any code change. Copilot Autofix in PRs and Agentic Autofix both take care to effectively remediate most of, if not all, vulnerable code paths flagged in alerts.

For our first-party tools, such as CodeQL and our new AI scanner, we run regular tests against real-world vulnerabilities to monitor and continuously improve fix effectiveness. Our next-generation remediation agent used in Agentic Autofix autonomously re-runs our detection engines and the project’s tests to ensure fixes are effective and free of breaking changes.

Together with our enterprise customer feedback, we’re continuously improving our autofix workflows, and we’ve observed a four times higher security backlog fix rate compared to human-only baselines based on the previous version of Autofix, where all we provided were suggestions.

Plenty of organizations sit on thousands of historical alerts. If an agent can open 2,000 pull requests overnight, the bottleneck moves to review. What should a CISO change in their process before turning this on across the company?

The agent doesn’t have to dump 2,000 PRs. Campaigns group the fixes, and the customer sets the pace.

A CISO should establish a risk-based remediation process, not simply convert an alert backlog into a pull-request backlog. The team should validate and deduplicate historical findings, then prioritize using evidence of exploitation, reachability in the deployed application, and business criticality: whether the affected system handles sensitive data, supports essential operations, or exposes a path attackers can use. That follows the contextual approach behind CISA’s vulnerability-prioritization guidance. This is something our customers do with GitHub.

Every proposed fix should explain the risk it addresses and include evidence that it removes the weakness without breaking expected behavior. The team should expand when the process demonstrates reduced high-risk exposure without unacceptable regressions, not merely because the agent can generate more changes.

Clearing the historical backlog is only half the job; the other half is making sure it doesn’t build back up. As the volume of code is growing exponentially every year, the name of the game must be prevention. GitHub Advanced Security catches issues across every surface where code is created: agent loop, IDE, CLI, Copilot App, and in each pull request before any code is merged, and Agentic Autofix fixes can be batched for more efficient reviews. In 2025, Autofix saved developers more than 1.5 million hours of security review time.

Vendors love mean time to remediate. Which metric would you trust more to show that a codebase is getting safer, not just that alerts are closing faster?

Rather than relying on mean time to remediate, I would look at whether the codebase is carrying less high-risk vulnerability exposure over time—and whether fewer vulnerabilities are being introduced or reaching production. Those measures focus on the security of the software, not just how quickly alerts are closed. To make the trend credible, keep scanning coverage and scope consistent, and distinguish verified fixes from suppressed or reclassified alerts. That approach aligns with NIST’s Secure Software Development Framework, which emphasizes reducing vulnerabilities in released software and preventing recurrence, and prioritizing vulnerabilities based on exploitation and impact.

We’re applying that same thinking to our own products. Copilot memories written by Agentic Autofix, combined with the security checks that run by default on every Copilot cloud agent task, enable the entire agentic software factory to become more secure over time, so vulnerabilities aren’t introduced in the first place.

The model behind autofix will change over time. How do you decide when a new model is good enough to swap in, and do customers get a say in which model touches their code?

For Copilot Agentic Autofix, we transparently document the model choice in our responsible AI documentation. New models are enabled only after careful evaluations for safety, alignment, secure code generation, and other company-wide standardized tests. Agentic Autofix optimizes model selection to achieve the best possible cost vs. capability trade-off, as indicated by our internal benchmarks.

We’re continuously refining our approach to model routing and selection in cooperation with our internal research teams and our AI lab partners. Our approach is based on continuous evaluations as new models become available, and trade-offs between precision, recall, and tokenomics.

If junior developers never write the fix for an injection flaw themselves, does the organization lose that knowledge? How do you keep autofix from becoming a crutch?

From our three-plus years of experience with fix suggestions in the Pull Request, we see developers take advantage of the contextual explanation provided by Autofix. Together with the suggested code changes, these become a key component for developer education, and several enterprise customers reported fewer similar vulnerabilities introduced over time. We’re leaning into this learning effect by giving developers even more self-serve security tools where they work today: in the CLI, their IDE, and the Copilot app, where they can explore and collaborate on security reviews in a safe space before sharing code with their team for review.

Download: The IT and security field guide to AI adoption