Code review used to be the only way to catch these bugs
An automated system called NOVA read the source code of 3,915 open-source projects over two months and came back with 14,090 vulnerabilities, each one confirmed through the system’s validation pipeline.
Vulnerability researchers at Palo Alto Networks’ Unit 42 built the system and checked its output against the public record afterward. Only 85 findings matched anything already documented, and most of those were published two to eight weeks after NOVA had found them.
The count is not the part that should worry you. Machines have been finding bugs in bulk since 2016, when Google’s OSS-Fuzz launched. By August 2023 it had helped identify and fix more than 10,000 flaws across 1,000 projects. What changed is the kind of bug. Only 8% of NOVA’s findings fall into the categories fuzzers have handled for years. The other 92% are bugs that never announce themselves with a crash.
Fuzzers find crashes. This found broken permissions.
A fuzzer throws malformed input at a program until something breaks, which makes it good at memory corruption, integer overflows and null dereferences, and blind to anything that keeps running.
“In the taxonomy we used for our frontier AI findings, the closest category, memory and calculation issues represented 557 findings or only 4.0% of the total. Even if resource-management and denial-of-service issues are added, these fuzzing-friendly categories total 1,121 findings — still only 8.0%,” Xu Zou, SVP of Cloud Delivered Security Services at Palo Alto Networks, explained.
Everything else came from categories where nothing visibly breaks. Access control flaws, where software checks who you are but not whether you’re allowed to do what you asked. Path traversal, code injection, prototype pollution, server-side request forgery. Catching these means reasoning about what the code was supposed to do.
The mix varied by language in ways that track how each ecosystem is built. C and C++ findings clustered in memory safety. JavaScript and TypeScript findings clustered in injection, prototype pollution and SSRF. PHP, Java and Python findings were dominated by broken access control, the recurring weakness of applications that have to track many kinds of user.
Vulnerability type by language shows that each ecosystem has a distinct weakness profile (Source: Unit 42)
One dependency flaw, four thousand exposures
In the package ecosystems examined, the pipeline produced 5,421 supply-chain findings. Of those, 1,280 were flaws in dependency packages themselves. The remaining 4,141 were downstream exposures, cases where an application’s code reaches a vulnerable dependency.
That ratio is the number to hold onto: roughly 1,300 flaws in packages generated more than 4,000 downstream exposures. And 2,776 of those were validated with a working proof of concept launched from the downstream application, not inferred from a dependency graph. Software composition analysis tools flag the presence of a vulnerable version. This established that the flaw could be triggered from where the application actually sits.
The gap between a patch and an exploit is the whole problem
“Our experience with NOVA highlights a clear structural change: the patch window has collapsed,” Zou wrote. The industry average for deploying a traditional patch, by his team’s reckoning, is 55 days.
An attacker does not need the latest frontier model to benefit from any of this. Once a fix ships, the diff between the old version and the new one narrows the search to a handful of lines, and working backward from a patch to an exploit is a far cheaper problem than finding the bug cold. Faster discovery means more patches, and more patches means more starting points.
No CVEs, no way to check
Unit 42 says it is working with open-source maintainers and with vulnerability clearinghouses, naming Lightwell and Akrites, to get the flaws fixed upstream. What it has not done is name a single affected project, publish a CVE identifier, or say when any of the 14,090 flaws were reported to the people who maintain the code.
The headline number also hides how unevenly it was earned. Four of the six ecosystem groups covered fewer than a hundred projects between them yet produced over half the findings. Large, dense codebases yield in clusters, so the total is not fourteen thousand separate surprises spread across the open-source world. It is a handful of big applications giving up a lot at once, plus a long tail of packages giving up a little.
Severity depends on which ruler you use. Under CVSS 3.1, somewhat over a quarter of the findings rated high or critical. Under CVSS 4.0, just under two fifths. Same bugs, different arithmetic.
Download: The high-performance team playbook