Helping researchers with IoT firmware vulnerability discovery
John Toterhi, a security researcher with IoT security company Finite State, believes that many of the security problems plaguing IoT devices are solvable problems through transparency.
“Manufacturers who make their firmware public and follow GPL practices are doing themselves a huge favor: by making firmware public, manufacturers are enabling a world-wide network of the best security talent to find bugs, disclose them responsibly, and improve security for their customers. Without this transparency they exclude so many responsible researchers and enable threat actors who easily obtain their firmware through chip extraction, man-in-the-middling updates, and stealing firmware from update servers,” he told Help Net Security.
Breaking the barriers for security researchers
Toterhi and his colleagues have analyzed over 200,000 firmware images from 76 unique manufacturers across many different products: SOHO routers, cameras, televisions, enterprise network appliances, cell phones, medical devices, printers, home/building automation controllers, and more. Luckily, they had the company’s analytics system at their disposal to tackle such a mammoth task.
“The biggest challenge for most people conducting IoT vulnerability research is gaining access to the devices and firmware of interest. Also, the process of unpacking firmware or trying to gather data from a live device can be incredibly time consuming, and it’s the primary reason many security researchers aren’t comfortable targeting IoT devices,” he opines. “There is a fear that they have to deal with hardware, and that is intimidating.”
He and his colleagues acquired the firmware almost exclusively through various public sources, including vendor websites, and fed it into the system.
“We’ve built our firmware analytics system on top of a highly-scalable, cloud architecture, and it leverages a modern distributed systems software architecture that makes heavy use of microservices and serverless processing to enable low-latency, high throughput processing,” he explained.
“We segment our analyzers into two main types; binary and source code. Our binary analyzers handle multiple architectures including; x86, ARM, MIPS, and PowerPC by lifting binaries to Binary Ninja’s Intermediate Language (BNIL) and performing analysis on this layer. Our source code analyzers require significantly fewer resources than our binary analyzers, allowing us to deploy them as serverless functions which spinup on-demand, allowing us to process over 10,000 source files (shell scripts, JavaScript, ASP, PHP, etc.) simultaneously with no extra effort on our part.”
The difficult task of securing the IoT
Toterhi will share the novel processes they employed for performing semantic analysis on source code and binaries with the audience at the upcoming Hack in the Box Conference in Dubai. While his talk will be focused on how they went about quickly unearthing and analyzing backdoors, his team’s research wasn’t limited to that: they were also after memory corruption, command injection, authentication bypass, XSS, and a number of other common security issues.
Many of the vulnerabilities they uncovered are simple: hardcoded, default passwords, debugging tools that should have been removed before production, easily detected authentication vulnerabilities, trivially exploitable memory corruption bugs.
He appreciates the challenges IoT manufacturers face – writing secure software, verifying third party software from their supply chain, responding quickly to vulnerabilities disclosed against devices that are often hard to update – but believes they can do better.
“The single biggest step that manufacturers can take is to scan through their firmware before they release it. Whether they use an automated tool like what Finite State provides or conduct a manual analysis, they should check for these types of vulnerabilities,” he notes.
“There are open source tools that can find the low-hanging fruit. Engineers can clearly see when they leave a backdoor account in a device. Unfortunately, we find that manufacturers who try to implement security into their processes tend to focus only on the security of the application software they are writing themselves (using AppSec tools) and completely ignore the most widely exploited classes of vulnerabilities: third party software vulnerabilities, configuration vulnerabilities, and authentication vulnerabilities.”
Some other things that they discovered during the research have also been somewhat surprising.
The code duplication problem
For one, code duplication across IoT devices turned out to be significantly higher than they initially anticipated.
“The duplication rate is close to 80% in the 34 million files analyzed during this research. This is due to a significant amount of code reuse across product lines, code adopted by open source projects like OpenWRT, and code developed by major system on a chip (SoC) vendors like Broadcom, Marvell, and Ralink,” he says.
And this is one aspect where their firmware analytics system can be of much help to security researchers.
“We see vulnerability researchers analyzing the same libraries and executables over and over, just because they appear in different devices, but our platform automatically pulls analysis results in from across devices and looks for code similarity, so researchers benefit from the work everyone has already done. We are automating away all of the tedious parts of conducting security assessments on IoT devices, and we’re ensuring that security researchers using our platform can spend their valuable time finding and validating vulnerabilities that machines can’t find on their own,” he explained.
Secondly, non-binary code like server-side scripts account for a significantly higher percentage of attack surface than they expected (roughly 3 to 1).
“This prompted us to develop a number of source code analyzers to address the viable attack surface of these devices. While this increased the complexity of automating the vulnerability discovery process, it provides a more holistic view of the security posture of IoT devices.”