Your threat feed is someone else’s database: What ingesting malware intel at scale takes

The advice is to consume shared threat intelligence. Join the ISAC. Wire the community feeds into your pipeline. This looks like a fine advice and I agree to it. What nobody mentions you is the operating manual, because the access was never the hard part. A threat feed is someone else’s database. Someone else’s processes built it, someone else’s judgement calls shaped it, and someone else’s bad Tuesday is sitting it right now, waiting for the automation to act on it.

I lead the team that runs Dependabot at GitHub, which monitors more than 30 million repositories for vulnerable and malicious dependencies as of 2026. This year we extended malicious-package advisories from npm, where we had been flagging malware since March, to eight package ecosystems, by ingesting community intelligence from OpenSSF’s malicious-packages repository. Malware alerts are a brutal test case for feed ingestion. You cannot upgrade your way out of a malicious package the way you can with an ordinary vulnerability; you rip it out, then go rotate every credential the build touched. And the volume never stops. Over the year ending May 2026, we catalogued roughly 18 new malicious npm packages a day.

There are five lessons that survived production and none of them care whether you are ingesting package malware reports, OSV records, or ISAC indicators. If community data feeds your automation, they apply to you.

Provenance is your undo button

We should always ask the uncomfortable question first. Not how records get into your pipeline: how they get out. Upstream sources publish mistakes. A report lands on the wrong package name. A researcher retracts a finding. A bulk import goes sideways at 2 a.m. Since that is a when and not an if, every record you ingest needs a trail back to the exact upstream change that produced it, and imports need to revert as a batch, in one move, in minutes.

Provenance belongs on the same shelf as feature flags and rollbacks: an operational control, not record-keeping. When a suspect record surfaces, everything depends on whether you can trace it to the exact upstream change that carried it in. “Revert everything that came from that change” is a quick correction. “Search the database and hope” is a bad week.

You will drink your own water

If your organization both contributes to and consumes community intelligence, sooner or later your own findings will come back to you wearing a stranger’s timestamp. Community repositories aggregate reports from many contributors, and aggregation strips the context of who found what first. Ingest naively and you build an echo chamber where your own report re-enters the pipeline dressed as independent confirmation or collides with the original and fires a duplicate alert on the same package.

The fix is boring. Fingerprint what you contribute, check every inbound record against those fingerprints, and drop your own echoes at the door. Before treating any record as corroboration, a consumer should be able to answer one question: did this start with us? A team that can’t answer it is measuring its own voice and calling it consensus.

Normalization is where the budget goes

The threat data is the glamorous part. The work is that ecosystems agree on nothing. One registry treats package names as case-sensitive while its neighbor does not. Affected versions arrive as exact pins in one format and open-ended ranges in another, with boundary semantics that differ just enough to hurt. Severity vocabularies refuse to line up. And none of this fails loudly. A single mismapped version range either alerts thousands of people who were never affected or stays quiet for the ones who were, and both failures look like a working system from the inside.

So budget for it. The mapping and validation layer ate most of our engineering cost and carries nearly all the correctness risk. If your integration estimate is mostly transport and storage, the estimate is wrong.

Automation needs a blast radius, not a reviewer

At tens of records a day across eight ecosystems, per-record human review is theater. It cannot scale, and pretending it can just move the fatigue from your users to an internal queue that quietly backs up. Our malware advisories publish automatically. Nobody reads each one. That is a design decision, and I will defend it.

What makes it defensible is engineering rather than trust. A cap limits how many advisories a single import run can create, so a corrupted upstream batch cannot flood alerts before a person notices. Every batch reverts as one unit. Volume and shape anomalies get flagged against what an ordinary day looks like. The right question for any automated pipeline is not who approves each record. It is: what is the largest mistake this system can make before a human notices, and how fast can we un-make it?

Fail loudly, never repair silently

The most tempting decision in feed ingestion is the quiet fix. An upstream record ships a malformed version range, your parser could guess the intent, and the guess would probably even be right. Resist. A silent repair converts someone else’s error into your liability. The guess goes out under your name, downstream consumers inherit it, and the upstream source never learns their record was broken.

Invalid records should fail into quarantine and generate a report back upstream. Slower. Correct. It also makes you a better citizen of the commons, since shared intelligence only improves when consumers push quality signals back instead of patching around them in private.
Before you ship

A short interrogation for your next feed integration. Can you trace any record to its upstream origin? Can you revert an import batch in one move? Would you recognize your own contributions coming back around? Does normalization dominate your estimate the way it will dominate your reality? And when upstream data is broken, does your pipeline object, or does it guess?

Shared threat intelligence is one of the few ideas this industry has agreed on that deserves the agreement, and the community sources behind it need more consumers, not fewer. But a feed is not a newsletter subscription. It is a live replica of someone else’s database with automated consequences bolted on. Run it with that level of seriousness and it makes your users safer. Treat it as a checkbox and sooner or later you will automate someone else’s worst day into your own.

Download report: How security controls perform in practice