Log4Shell is a dumpster fire that should have been avoided
On Thursday, December 9, 2021, my young, Minecraft-addicted kids were still completely oblivious of the Log4j vulnerabilities in their favorite game. Then again, so was every cybersecurity professional in the world.
That all changed when the Apache Log4j project announced CVE-2021-44228 (aka Log4Shell) – a zero-day vulnerability in Log4j’s standardized method of handling log files used by apps all over the world, from Microsoft’s Minecraft to Twitter to Tesla to Apple’s iCloud. This led to a blaze of stories about how the internet is “on fire.”
These screaming headlines make sense for us in the software and digital services industry. This vulnerability, which continues to be followed by others, is bad news. And it’s hard to find a metaphor to describe an easily exploitable zero-days vulnerability in a huge number of high-value targets.
Rationally, nearly every business, organization, and government that does anything related to software went into crisis mode. It is difficult to imagine anyone having been unaffected, and most probably many are still on high alert even after nearly two weeks.
However, for most internet users, life went on as usual.
If they have read about the Log4Shell firestorm, average users have probably concluded this story was all smoke and no fire: Minecraft works, Facebook works, their iPhone is charged. Who cares? Amazon’s services in the US have been a bit on and off lately, but these were unrelated outages.
Fortunately, my kids, like most people, likely have no idea how bad this could get—at least not yet. But if you work in cybersecurity, you know one thing is true: most of us have lucked out, so far.
We could have avoided Log4Shell
The truth is we have no idea how severely attackers have taken advantage of the vulnerabilities in Log4j. Attackers can obfuscate their intrusions relatively easily and it’s unlikely that the hundreds of thousands of companies that have been busy patching their systems have engaged in any sort of incident response to detect whether the vulnerability was exploited before the update.
Without a doubt, this is a dumpster fire. And mostly everyone in our industry is doing their best to make sure it won’t leap out and consume its surroundings. Perhaps our triage will pay off. Maybe consumers won’t even remember the hysterical Log4Shell headlines they glanced on between holiday sales. But for anyone who cares about cybersecurity, the stink is impossible to ignore.
And here’s the worst part.
I have yet to hear from a single service or product that would be negatively affected by configuring Log4j to disable lookups, the feature that exposed apps to the worst vulnerability disclosed.
This begs a question: where is it actually being used? And who needs it?
If a feature isn’t being used, any CISO or threat hunter or cybersecurity trainee will tell you that it should not be activated. Turns out, the vulnerable feature was added in 2013 as a “nice-to-have” addition, virtually without a debate. Even the current maintainers of Log4j dislike it and were quick to default it to a disabled state in an update released on Monday, 13 December.
If basic IT hygiene guidance had been followed, Log4j would have easily been immune to this type of vulnerability, but the internet has not exactly been built by way of hygiene.
Favoring features over security – even unnecessary ones
Microsoft Office macros, which were first developed in the 1990s, are a powerful series of commands that can be used to automate a repeated task. Still, nobody in their right minds would allow users to execute macros at will, and nobody should be allowed to send unsolicited macro-laden documents from untrusted sources. Yet this is exactly what keeps happening by default.
The way our industry treats Office macros represents the way we treat nearly all decisions where we need to choose between features and security.
The SNMP and ASN.1 vulnerability discovered by Oulu University Secure Programming Group (OUSPG) in 2001 and released in 2002 ended up forcing the US Navy to call all nuclear-powered submarine vessels back from seas for emergency software updates. The ASN.1 is an overly complex syntactic language used by basically all communications protocols. Find a bug in there, and you break all protocol implementations. This is essentially what OUSPG did two decades ago. And versions of the same thing keep happening.
The Heartbleed vulnerability of 2014 affected all OpenSSL-based Transport Layer Security implementations and allowed attackers to dump information from internet-exposed systems. Heartbleed has eerie similarities to the Log4j incident as both were open-source projects, both projects suffered from under-resourcing, and both had become bloated over time. Yet both were still being blindly trusted by thousands of projects and commercial products.
If you were around at Heartbleed, you cannot act surprised now with Log4j.
Once a supposedly harmless and innocuously funny feature gains large enough userbase, the potential fallout of a vulnerability suddenly becomes critical. And it happens over and over again.
No one can solve every cybersecurity problem alone
I imagine that many internet users —even my young kids—would be shocked to learn a piece of software like Log4j, which is deployed by some of the largest companies in the world who do trillions in dollars of businesses, is the product of a volunteer operation, where highly trained pros work countless hours for free—especially right now.
The fragile interdependency of the internet has rarely been more obvious as it is right now. We all use similar if not the same tools. We all are vulnerable to zero-days that threaten the core of our businesses. We must all learn from each other and build on the work others do, or we all risk failure—alone. This mutual reliance built the modern internet, yet it also seems to have led to a focus on short-term fixes that glide us from crisis to crisis without addressing the systemic issues we face.
Only a focus on outcomes—a fixation on the systems and results we want to create—can move us from an industry where we fill in each other’s gaps into one which truly collaborates for security by design.
So how do we shift left to better outcomes?
An obvious first step is to recognize that companies that use these freely available components should figure out ways to give back.
And by paying back, I don’t just mean cutting a check, as many have recently suggested in Twitter hot takes. Companies can test various components for security from our research time. We can contribute code for fixes, stability improvements and features from our research and development budget. And we can offer to take some of the load off project volunteers by allowing developers to spend time on projects while on worktime.
We are already doing all of this to some degree, of course. I’m not suggesting some new quantum theory here. However, we could do more—and we should do more. And not because we’re a charity, but because it’s a smart investment.
Just think about the number of billable hours (for lack of a better term) that have been spent on chasing the number of ways Log4j component has been incorporated in our deliverables, backend systems and in our supply chain. The severity of the risk greatly exceeds the so-called peacetime allocation of our contribution to open-source projects.
Wishful thinking isn’t enough
The old saying goes, “Don’t cry over spilt milk.” When it comes to Log4j, the cybersecurity industry served a lot of milk in a cracked glass. Once we realized the crack was there, we let out a primal scream. Meanwhile, we’re still serving new glasses to the onlookers, who wonder what all the fuss is about.
We broke this; we must fix it!
Our industry has generated trillions in value with the promise of being engineering marvels, offering well thought-out and smoothly functioning software and digital services. As this crisis shows, one only needs to scratch the surface and it becomes evident that much of that promise is built on wishful thinking and pure luck.
It’s time to start fixing technical debt that we have accumulated and start fixing the most critical parts of our digital infrastructure. And given the risk we all face, we’d be fools to not to step back and at least look at what we’ve been putting in our dumpsters.