Operational Technology (OT) networks are at risk from the recently-announced Apache Log4j (CVE-2021-44228) vulnerability. On the surface, it is not clear why this should be.
The vulnerability affects millions of web servers, allowing remote attackers to inject any code they wish into vulnerable Java applications on the Internet. The defect is being widely exploited in the wild, which is why security teams all over the world are scrambling to identify which of their web applications might use Log4j, and then working to rebuild or upgrade those systems.
But – nobody who knows anything about cybersecurity hosts OT applications that control pipelines or power systems or rail networks on the Internet. So, why are so many OT businesses so worried about Log4j?
The answer is the Industrial Internet, or some call it Industry 4.0. For the last half decade, critical infrastructures and manufacturers have been connecting their shop-floor systems straight out to the Internet. These are encrypted connections to web services and cloud services. Those connections punch through or simply bypass the half dozen layers of firewalls that are typically deployed between the Internet and most automation systems. Hence the problem – we may have done all of our new supply chain due diligence.
We may have convinced ourselves that we should trust our software and cloud service providers. But – even if we trust those providers – but should we trust their websites? The most trustworthy vendor in the world may still have a website and may expose web services that will be compromised in the next day or two. And – the Industrial Internet equipment in our OT networks is connected out to these at-risk cloud services.
Worse, once sophisticated ransomware groups or other attackers have a foothold in industrial vendors’ web services, those threat actors can be very difficult to detect or dislodge, even after the Log4j vulnerability is long since history. The big risk is that these attackers will remain embedded in the cloud services to which OT networks are connected.
Once these attackers have had time to look around and figure out how to take advantage of OT systems’ trust in these compromised cloud services, these attackers will be able to use these services to propagate their attacks deep into industrial infrastructures. Such attacks risk compromising thousands of industrial sites at once.
Cloud–based OT ransomware
Waterfall Security Solutions predicted the comingling of OT supply chain and cloud-based attacks in the OT/ICS Ransomware in the Supply Chain report. Unfortunately, recent events have proven these predictions. The Colonial Pipeline and JBS meat packing ransomware incidents demonstrated that critical infrastructures shut down by ransomware are more likely than not to pay seven and eight figure ransoms. This makes critical infrastructures much more likely to be targeted by ransomware actors in the future.
And the Kaseya incident showed clearly that ransomware groups are up to the task of using vulnerable cloud infrastructures to launch attacks simultaneously on thousands of targets. Hence today’s concern about Log4j – compromised cloud services from industrial vendors are a huge threat to industrial operations all over the world.
Securing OT networks
Most security practitioners’ instincts are to apply to OT networks the same tools and techniques that are used routinely to defend IT networks from ransomware, but this does not work. Why? Well, a big part of addressing ransomware on IT networks is NIST’s “detect, respond and recover” pillars. That is: identify the affected machines, isolate them, erase them, restore them from backups, and repeat.
The problem with this approach on OT networks is that cyber sabotage and uncontrolled shutdowns can be physically dangerous. Power plants have 100-ton turbines rotating at 1200 rpm, refineries have six-story catalytic crackers full of very hot, high-pressure hydrocarbons and even escalators in big buildings are dangerous for the people riding on the escalators if these things are sabotaged. The big issue with relying on “detect, respond and recover” is that human lives, damaged equipment, and lost production cannot be “restored from backups.” Yes, OT networks need incident response capabilities, but those capabilities only somewhat reduce the consequences of compromise – preventing compromise is and must be the top priority for OT networks.
To this end, the OT security solution that industrial sites are applying to this problem is the unidirectional gateway. Unidirectional gateways include hardware that is physically able to push information in only one direction – from our critical OT networks out to the Internet. The gateways are deployed between the Internet and vulnerable OT networks’ Industrial Internet equipment. The gateways work, because all ransomware and all other cyber-sabotage attacks are information – this is what “cyber” means.
So, when a gateway is physically able to push information out to industrial vendors’ cloud services on the Internet, and not able to let anything back in, then compromised cloud services are no longer a threat to safe or reliable industrial operations.
Such compromise may still be a threat to efficient operations, because industrial sites use industrial cloud services for a reason – to increase efficiency. But temporary reductions in efficiency are generally tolerable risks, whereas threats to worker safety, to public safety and to environmental safety are generally unacceptable.
Log4j vulnerability and OT networks: The bottom line
The bottom line is that the Log4j vulnerability is a huge problem. Cloud service providers, and especially industrial cloud service providers, need to look hard at any of their cloud and Internet-exposed systems that have ever exhibited the vulnerability. All such systems and anything connected to those systems is at risk of already harboring cyber attackers and ransomware groups.
Industrial enterprises need to shift focus as well. Today, such enterprises are busy asking every one of their software vendors if their products use Log4j or exhibit the vulnerability. A more important goal is to ask all industrial cloud providers if their cloud services have ever been vulnerable to this Log4j vulnerability.
And no matter how these questions resolve for today’s Log4j vulnerability, industrial enterprises who have not already done so should really look at deploying hardware-based, unhackable protections for OT systems, especially for those OT systems that are connected to the Internet. There will inevitably be other vulnerabilities and other compromises of those cloud services that are used by OT systems.
Remember that have already seen ransomware compromise 1500 Kaseya customers at once. We do not need ransomware, or anyone else, to cripple 1500 pipelines or power plants at once via Log4j or via any future cloud-system vulnerability.
To dig deeper into what secure OT sites do differently, you can request a copy of the author’s latest book Secure Operations Technology, free, courtesy of Waterfall Security Solutions.