You can’t trust AI chatbots not to serve you phishing pages, malicious downloads, or bad code
Popular AI chatbots powered by large language models (LLMs) often fail to provide accurate information on any topic, but researchers expect threat actors to ramp up their efforts to get them to spew out information that may benefit them, such as phishing URLs and fake download pages.
Surfacing incorrect, potentially malicious URLs
SEO poisoning and malvertising has made searching for login pages and software via Google or other search engines a minefield: if you don’t know how to spot fake/spoofed sites, you’ll get your credentials stolen and your devices infected.
Partly because of this and partly because search engines have become bad at surfacing relevant information, users have slowly begun asking AI chatbots for information instead.
For the time being, their results may be more to the point and delivered more quickly, but the information provided can often be inaccurate, whether because the LLM got it wrong / was fooled, or because it outright “hallucinates” (i.e., invents) the answer.
Case in point: Netcraft researchers have recently asked chatbots powered by the GPT-4.1 family of models to surface login pages for 50 different brands across industries like finance, retail, tech, and utilities, and they got it right in 66% of the cases.
But 5% of the domains returned belonged to unrelated but legitimate businesses, and a whopping 29% – that’s 28 domains – were unregistered, parked, or had no active content.
“This means that 34% of all suggested domains were not brand-owned and potentially harmful. Worse, many of the unregistered domains could easily be claimed and weaponized by attackers. This opens the door to large-scale phishing campaigns that are indirectly endorsed by user-trusted AI tools,” they noted.
Popular search engines have also begun providing AI-powered search results that are equally dodgy and, on occasion, can lead users directly to phishing pages impersonating a popular brand (e.g., Wells Fargo).
To make the matter worse, AI-generated answers often strip away traditional indicators like verified domains or search snippets. “Users are trained to trust the answer, and the attacker exploits the user if the answer is wrong,” the researchers noted.
Attackers are actively trying to poison AI chatbots’ results
The surfacing of bad links might have initially happened rarely, but threat actors are doing everything to make it happen more often, and are actively trying to poison the results provided by generative AI chatbots.
They are designing pages that will be ranked high by chatbots’ language model: they look legitimate, provide correct documentation, pose as support hubs, etc.
“These sites are clean, fast, and linguistically tuned for AI consumption. They look great to humans—and irresistible to machines,” the researchers noted.
“And it’s not just phishing. We often see malware distributed via ‘cracked software’ blogs, tutorials, and discussion posts. As AI search gains prominence, these old vectors could see new life—surfacing not through keyword gaming, but through linguistic fluency.”
Despite safeguards to prevent sourcing bad/malicious codes, AI-powered coding assistants can be “gamed” in the same way by threat actors who not only publish bad code (e.g., a fake API), but also make the effort to accompany it with credible content (tutorials, forum Q&As) and use credible GitHub and social media accounts to promote it.
Subscribe to our breaking news e-mail alert to never miss out on the latest breaches, vulnerabilities and cybersecurity threats. Subscribe here!