What AI-generated malware looks like under the microscope

In this Help Net Security interview, Lindsay Kaye, VP of Threat Intelligence at HUMAN Security, talks about what AI-generated malware looks like on the bench. She points to tidy comments, uniform logging messages, and emojis as signs a script came from a model, and notes that decompiled code strips much of that away. Kaye also covers how AI is squeezing the time teams have to patch, where it helps criminals scale fraud, and why her team keeps a human reviewing every conclusion before it goes out.

If you’re at Nullcon Berlin, Kaye is participating in the panel discussion “Agents Gone Wild” on November 5, 15:30–16:15.

You built your career taking malware apart by hand. When you reverse something you suspect was written with AI help, what looks different on the bench, and what looks exactly like it did five years ago?

As of right now, I have not explicitly reverse engineered malware written fully by AI in my current job – at least to my knowledge – which I’d consider something similar to the VoidLink malware disclosed by CheckPoint. Sometimes, I do come across scripts or pieces of code that look to be written using AI, and there are a few things that really stand out:

• In malicious scripts (so, NOT decompiled/disassembled code) you’ll see comments describing each function, often in the same type or format or sometimes with emojis. While this is certainly considered good practice in software development, this is not something a malware author would do. In decompiled code, as you know, comments would no longer be present, since they disappeared during compilation.
• Similarly, in decompiled/disassembled code, you might see logging messages that are all very similarly structured, with good spelling and grammar like you’d see in a professional codebase. You might also again see the use of emojis.
• In my current role, I focus extensively on advertising fraud which often materializes in mobile apps and HTML5 game sites. Before AI, we’d see tens of apps as part of campaigns but now, generating app content is so much easier that we see hundreds. Of course, now when we look at these apps/sites, they look mass-produced or share technical commonalities, and we see new ones emerge much more quickly than in the past.

While not EVERY piece of code that has these features is AI-generated, code generated using an LLM is “planned out” and will be structured according to “good” design patterns and coding behavior. However, when looking at decompiled/disassembled code itself (outside of the indicators I pointed out earlier), you do lose a lot of context, since decompilers can’t perfectly rebuild C/C++-like code. Add on obfuscation and it becomes even more challenging to discern. As for what has looked exactly the same, a few things I’ve continued to see are:

• Example code easily found on the internet still makes an appearance in malware. Whether this is copied by hand (as it was in the past) or surfaced using an LLM prompt, you’ll often see evidence such as use of “placeholder” names (such as “MyWindowClass”) that suggest the code was not generated by the author.
• We still see people use obfuscation toolkits, such as O-MVLL, or custom techniques to make code harder to reverse engineer. Both these toolkits and the custom techniques can be pretty advanced, and make analysis very challenging; however, now, discerning whether a custom technique is author-generated or created with the help of AI in some way is not entirely easy.

Ultimately, AI serves as a way to make malware developers’ jobs easier, allowing them to more easily put together code or scripts for a particular purpose. However, I’d say one of the most interesting – and potentially impactful – use of AI related to malware is when it’s used in a case like PROMPTSTEAL to alter its behavior mid-execution. I am certainly looking forward to seeing how AI changes how threat actors execute attacks, and understanding how we can more effectively defend against them.

If working exploit code can follow a disclosure within minutes, what happens to the patch window? Should defenders stop measuring response in days?

With the increased availability of AI, finding security flaws and vulnerabilities has not only become more accessible to and easier for people to do, but they can do it much more quickly and at scale. The PHP2Shell RCE vulnerabilities are a great example here, and had they been widely exploited by a threat actor instead of disclosed by security researchers, the damage could have been exceptional – especially before a patch was available.

Patching has been one of the most important tools for security teams to address vulnerabilities, especially those for which there are available exploits. It also brings with it some challenges that can often stretch the time teams take to implement them, notably versioning dependencies, ensuring the patch won’t break anything else, and downtime associated with putting the patch in. Time pressure around patching when a vulnerability becomes public (especially if it has publicly released POC exploits) has always been around, but now we’re facing more vulnerability disclosures on top of a very likely quicker and easier path to exploitability which serve to further narrow the time defenders have to mitigate it.

It’s challenging to give an exact timeframe for a patch window that would suit every organization, but we do know that we need to respond to the accelerating timelines created by AI-enabled threat actors.

Code was rarely the attacker’s bottleneck. Access, infrastructure and cashing out were. Where is AI changing criminal economics, and do you see it in dark web pricing for stolen accounts?

AI overall is lowering the barrier to entry for cybercriminals to conduct all kinds of fraud, including account-related fraud but a lot of the “benefit” we see today is in improving scalability, and increasing the efficiency of attacks. This materializes in a few different ways:

• Generation of “content”. This includes creating phishing pages/messages used to steal victims’ financial and personal information, conversing with victims of romance scams or pig butchering scams to reel them in, staging AI-written blog posts intended to hook a reader long enough to allow ads to load and reload, and audio tracks to enable streaming fraud.
• Going through large amounts of data to find data of interest. Data exfiltrated from stealers (which have been around long before AI, and remain popular methods of obtaining victim data), stolen from a company or scraped from a site contains swaths of information that AI can help structure or pick through to identify the most useful pieces. These might be usable session cookies and tokens from stolen browser data for session hijacking, valid credentials or files containing password data, any of which might be usable to escalate an attack, move laterally, or conduct a new one.
• Verifying data. One interesting case that we observed involved the use of an AI agent to conduct carding-like behavior. Traditionally, threat actors test credit card numbers to verify efficacy prior to selling them to other threat actors to conduct fraud. Verified cards can be sold for more than unverified ones.

In terms of pricing for accounts, I haven’t seen drastic changes that I can directly attribute to AI (pricing is largely based on supply and demand-related factors), but as noted, the economics of how these accounts are obtained and verified certainly represents an area in which AI provides a benefit to threat actors.

What should model providers share with threat intel teams that they don’t share now? Should abuse indicators from AI platforms flow the way IOCs do?

Honestly, I’d question whether “AI abuse indicator” is the right unit here. Similar to what I said earlier about reverse engineering, knowing a threat actor used AI is interesting context, but it isn’t usually the thing that helps me defend against them. What matters is what the activity does on the target.

That said, there’s real value in what already exists:

• Google and others have published research on threat actor usage of AI tools, which gives defenders a view into how these tools are being applied
• MITRE’s AI Incident Sharing Initiative, which sits within ATLAS, has stood up a STIX-based channel with a group of contributing organizations, and ATLAS has been adding generative AI techniques and case studies

To me, that’s a more useful direction than a separate indicator feed. An IOC is actionable because it’s specific and reasonably durable, a hash or domain. “A model was involved” is neither, and anything more granular, like a prompt pattern, tends to stop working the moment a model updates.

So if I had an ask, it would be about continuing to document the techniques, so they land in the frameworks teams already work from. That’s the format that reaches a detection engineer.

I’d also say this is a two-way conversation. Those of us seeing AI-enabled activity downstream have as much to contribute to that picture as platforms do.

How does your own team use AI in threat research, and where do you refuse to let a model make the call?

To date, my team has used AI as a way to augment our workflows. A great example is using AI to help summarize or ask questions from the data we collect from a wide variety of sources. In addition to data processing activities like extracting IOCs from reports that we may want to pivot on, we want to understand what tools are out there targeting our customers or products, whether they have changed since we last looked at them, and how these and other relevant topics are discussed in the underground forums we monitor.

AI can help us ask these kinds of questions and give initial answers to start from, but what it produces can’t be shared as-is without a human confirming the conclusions are correct and reviewing the findings from the lens of any additional insight we might have that the AI does not.

Another example is using AI as an aid for creating detection rules that we can use to hunt for additional malware related to samples of interest, or to help with clustering for samples or of disparate entities/groups based on commonalities in attributes. Again, AI serves as a way we can move more quickly towards developing our own analytical conclusions, but doesn’t replace that analytical layer provided by a human.

Virtual event: Top security leaders on governing AI agents