AI agent observability is struggling to keep up with hallucinations
Sixty-two percent of IT and engineering professionals now rely on AI agents frequently to work out why their systems broke, and more than half have no way to measure whether the agents get it right. Splunk surveyed 2,050 of them across ten countries.
Source: Splunk
Agents diagnose, people still ship the code
Teams have been careful about where they let agents work. Finding root causes, tuning alerts, and drafting runbooks, the step-by-step guides engineers follow during an incident, are now common agent jobs. Writing new code sits at the bottom of the list, and about a quarter of respondents never hand it to an agent.
The reasoning is easy to follow. A bad alert threshold takes a minute to roll back. Bad code in production is a bigger problem.
For most teams, agents have made them faster. A large majority say agents help them spot problems sooner, and many say recovery is quicker too. But 14% say recovery got slower after agents arrived.
Sahinaz Safari, who leads AI for Splunk’s observability business, compares it to bringing on staff. “Like any new hire, they come with incredible potential as well as a period of more oversight as you integrate them in your environment,” she says.
That oversight has picked up a name, the verification tax. It’s the time engineers spend double-checking an agent’s reasoning before they trust its answer.
Everyone wants it watched, fewer can watch it
Most respondents call monitoring AI a priority. Hallucinations show how far practice trails that intent. Eighty-five percent say it’s important to see when a model invents an answer, but only 58% say they can reliably catch it.
Part of the problem is tooling. About half say their observability tools weren’t built to watch AI. The other part is ownership. Does a misbehaving agent belong to the data science team that built the model, the SRE team that runs the infrastructure, or security? Thirty-eight percent say their organization hasn’t decided.
“When agent observability is everyone’s job, it becomes nobody’s job,” says Mark Woods, field CTO and chief technical advisor for EMEA at Splunk.
Picture an agent stuck in a loop, hammering an internal API thousands of times a minute. Engineering sees a performance failure, and security sees something that looks like a denial-of-service attack. It’s one incident, and 81% of respondents say monitoring AI has pushed their teams to work more closely with security.
The bill arrives late
Money is where things get shakier. In 57% of organizations, observability now costs more than the infrastructure it monitors. Half got hit with a surprise telemetry bill this year.
Greg Leffler, Splunk’s director of developer evangelism, describes how that feels. “It’s like shopping for groceries where the price of a single tomato could jump from two dollars to ninety dollars overnight… and you don’t get the bill until the end of the month,” he says.
Download: The IT and security field guide to AI adoption
