Turn it off and on again, but for critical infrastructure
Researchers at KTH Royal Institute of Technology built a container replica of a segmented industrial network, attacked it repeatedly across 14 days of running time, and used the captured traffic to train a defense agent that decides on its own when to intervene.
The agent sees six numbers per interval: packet counts crossing the network’s segments and moving to and from individual machines. From those counts it infers how far an intruder has progressed, and acts.
It can do nothing, reset one of three supervisory hosts, reset one of two water tank processes, or reset every host in the supervisory and control subnets at once. A reset reboots the target, renews its credentials, and changes its IP address. A reset can briefly interrupt industrial operations. An agent reading packet counts decides to bounce a running process, and the plant absorbs the outage.
The OT infrastructure of the intrusion response use case (Source: Research paper)
Most work in this area assumes the attacker is visible
Research on reinforcement learning for industrial intrusion response has mostly assumed the agent observes the system state or the attacker’s actions directly. The authors call that assumption unrealistic, and say the studies that do model partial observability tend not to explain where their observation model came from. So they measured one, running the emulated network in 30-second periods and collecting 40,000 of them. Even that was thin. Modeling how traffic varies with the full system state would take roughly 100 million measurements, so they estimated something simpler: how traffic varies with the attacker’s action alone. The packet counts are real. What the model can do with them is less than the math calls for.
The results, and the assumption underneath them
The best of the three agents the researchers trained keeps 500 running guesses about the state of the network, updates them every interval, and feeds a compressed version to its policy. It beat the two agents fed raw observation history and came close to a baseline agent handed full visibility into the system state. Feeding a policy four intervals of history instead of one made it worse.
There is a catch in how that estimate gets built. To update its guesses, the agent runs them through a model of how the system evolves, and that model includes how the attacker behaves. The agent performing closest to the full-visibility baseline is working from a description of the adversary it is defending against.
What transfers to a real network
The test network is three supervisory hosts, two PLCs, and two tanks. Both HMIs run HTTP with weak credentials, and the engineering workstation runs SSH, Telnet, and SMB, with weak credentials and exposure to CVE-2017-7494. The researchers did not study whether their model generalizes to other configurations or attack types.
The idea that transfers without any of the machinery is belief tracking. The learning agent and the paper‘s simpler threshold baseline both maintain a probability distribution over how far an intrusion has progressed on each asset, from undiscovered through scanned, exploited, and inspected, weighting each stage by what it would cost. A console showing the probability that a host has been exploited is buildable without reinforcement learning.
The researchers have released their implementation and plan to test the approach on an industrial testbed with a partner. They put operational safety in their future work, and note that it constrains what a defender strategy is allowed to do.
Download report: How security controls perform in practice
