How an AI meltdown could reset enterprise expectations

In this Help Net Security interview, Graham McMillan, CTO at Redgate Software, discusses AI, security, and the future of enterprise oversight. He explains why past incidents haven’t pushed the industry to mature. McMillan also outlines the structural shifts he expects once failures start to have business impact.

What type of AI-driven failure do you believe is most likely to force the industry into a maturity leap, and why have recent incidents not been enough to trigger that shift? When that failure happens, what is the first structural change you expect enterprises to make in how they build and govern AI systems?

There hasn’t yet been an industry overhaul for two reasons. Firstly, sometimes it’s hard to link AI to causation. We’ve seen hundreds of instances of hallucinations from models, but it’s hard to draw a straight line between AI and a result, especially where there’s been a catastrophic outcome. Can you prove that this person wasn’t going to do the said thing anyway?

Secondly, even in cases where the link is more obvious, such as when an AI system gives inappropriate advice like recommending a toxin as a medication, companies can still fall back on broad disclaimers that their answers may be wrong. An industry maturity leap won’t happen in reality until something high profile changes the public discourse and is a catalyst for change. Even then, legislation will only come in after wrangling with lawyers and it will be slow moving, because the pace of AI innovation is outweighing the pace of legislation creation.

Structurally, not much may happen next year, but where AI has been overpromised, there might be an overcorrection to reset expectations around what it can deliver for an enterprise. The AI bubble could burst, not because anything’s wrong with the tech itself, but because the hype is running ahead of reality.

How do you expect companies to rebalance model performance, safety constraints, and business speed once AI failures begin to create material financial consequences?

This ultimately comes down to trust and clarity around what data is being shared, and by whom. Is there governance on what information is going into models and how it hangs onto it in order to train itself? Requests from senior leadership to its employees to prioritize performance, speed and automation might be replaced by the need for caution and safety, especially if an enterprise starts seeing financial (or reputational) risks to its bottom line.

We’ll likely see a shift away from reactive to proactive governance, with enterprises asking important questions, has the model been instructed to hang onto the data so someone else can’t access it? Has a human been in the loop to provide critical thinking required? A safety net can’t always be installed with these models, so how is our organization incorporating human oversight?

Which traditional cybersecurity control, such as IAM, DLP, threat modeling, or logging, is fundamentally misaligned with AI-era risk, and what do you think will replace it?

I think there is always going to be a need to deploy traditionally good approaches to securing AI. IAM, DLP and logging remain essential, but threat modeling needs a significant update. The opportunities for breaking the integrity of the interaction with AI is exploding. It is no longer just about accessing the model.

We also need to consider whether the model should have access to all the resources required to augment a prompt, the integrity of the data used to train it, and whether the user has the right to view the responses. All this needs to be modelled and the “what-ifs” mapped out and dealt with. Then we can build security into the system by design, rather than patching after when attacks or hallucinations happen.

How do we track where AI has been used and what information its been trained on, and how do we negate any false assumptions its made based on old or imperfect data? The scope of the problem is so big that throwing out some of these well understood approaches is not what would we should be doing because we’ll need all these and more at our disposal to address the threats.

How will incident response evolve when the blast radius is driven by incorrect reasoning or manipulated training data rather than code execution?

This is an interesting question because if the responses are probabilistic then will the blast radius ever be greater than a few percentage points? The thought experiment I run is how much incorrect training data would you need to use such that the model gave a consistently bad answer. And so consistently bad that it created a meaningfully (i.e. for the lawyers to care) large blast radius? In which case, if the training data was so bad then wouldn’t there be a sufficient number of incorrect answers such that the model would lose credibility?

There is a significant possibility that “incidents” might be few and far between and could be very difficult to replicate especially if the AI has been told not to log any prompts or responses. The bigger risk is that a small number of plausible responses cause catastrophic damage because they’ve been used in an agentic context that could set off a chain reaction of incorrect actions.

Do you expect companies to begin treating AI models as internal critical infrastructure, and if so, what oversight mechanisms will emerge that do not exist yet?

I think enterprises will introduce the notion of the central AI team who will be in charge of its rollout and usage way before it becomes critical infrastructure. By critical, I mean that the business will fail if it’s not available. It will require special skills to deploy, and the costs to business are going to start growing in order to satisfy the crazy stock market valuations we see today.

In that way it’s deployment will mirror the use of IT over the second half of the 20th century. So managing all that centrally will be a natural reaction.