Dyn DDoS attack post-mortem: Users inadvertently helped

As StarHub, one of the three major telcos in Singapore, confirmed that they were the latest victim of “intentional and likely malicious distributed denial-of-service attacks” on their DNS system, Dyn has published a short post-mortem of the unprecedented DDoS attacks it suffered on Friday (October 21, 2016).

Dyn DDoS attack post-mortem: Users inadvertently helped

The Dyn DDoS attacks

Scott Hilton, EVP of Product at Dyn, confirmed that a Mirai botnet was the primary source of the malicious attack traffic. But the initial information about 10s of millions of IP addresses involved in the attacks were ultimately misleading.

Only 100,000 malicious, mostly Mirai-infected IoT endpoints took part in the attacks – most of the other requests to the DNS system were legitimate, from users trying and re-trying to access their favorite websites as the attack was ongoing.

“During a DDoS which uses the DNS protocol it can be difficult to distinguish legitimate traffic from attack traffic,” Hilton explained.

“For example, the impact of the attack generated a storm of legitimate retry activity as recursive servers attempted to refresh their caches, creating 10-20X normal traffic volume across a large number of IP addresses. When DNS traffic congestion occurs, legitimate retries can further contribute to traffic volume. We saw both attack and legitimate traffic coming from millions of IPs across all geographies.”

The attacks took the form of high-volume floods of TCP and UDP packets to port 53.

The first attack was initially leveraged against the company’s Managed DNS platform in the Asia Pacific, South America, Eastern Europe, and US-West regions, but very quickly concentrated on the US-East region. The second one, which begun later that day, was more globally diverse, but employed the same protocols as the first attack.

Both attacks resulted in disruptions of service, but were ultimately mitigated by Dyn’s Engineering and Operations teams. In their wake came several small TCP attacks, but did not affect the company’s services or customers.

“Early observations of the TCP attack volume from a few of our datacenters indicate packet flow bursts 40 to 50 times higher than normal. This magnitude does not take into account a significant portion of traffic that never reached Dyn due to our own mitigation efforts as well as the mitigation of upstream providers,” Hilton pointed out. “There have been some reports of a magnitude in the 1.2 Tbps range; at this time we are unable to verify that claim.”

He noted that these attacks have opened up an important conversation about internet security and volatility.

“Not only has it highlighted vulnerabilities in the security of Internet of Things devices that need to be addressed, but it has also sparked further dialogue in the internet infrastructure community about the future of the internet. As we have in the past, we look forward to contributing to that dialogue,” he concluded.

The company refused to speculate on the motivation or the identity of the attackers. US director of National Intelligence James Clapper said that preliminary indications point to a non-state actor.

The attacks against StarHub’s DNS infrastructure

According to the statement published by StarHub, there were two attacks: one on October 22 and the other two days later.

The attacks were unprecedented in scale, nature and complexity, the company said, and resulted in temporary web connection issues for some of their home broadband customers.

Apparently, the attack came from their own customers’ infected IoT devices like web cams or routers, and the company is planning on sending out its technicians to clean them up and secure them. The Mirai malware was not mentioned as the culprit.

What next?

A few days ago Hangzhou Xiongmai Technology, the manufacturer of several types of IP cameras, recalled some of the older models sold in the US that were vulnerable to getting infected with the Mirai malware.

According to information gathered by Flashpoint, it’s possible that a portion of the Mirai botnet that attacked Dyn consisted of devices manufactured by the Chinese company.

Xiongmai said that devices sold after April 15 are protected by new firmware that plugs the holes through which the malware can get in, and advised customers to change default passwords.

Cleaning up the various devices infected by Mirai is going to be a problem, as well as keeping them clean long enough to secure them against a new compromise (if not secured, it takes mere minutes for a cleaned device to get reinfected).

Unfortunately, we can’t rely on users to do it, as most are not even aware of this secret life of their devices, and aren’t tech savvy enough to perform the needed operations.

Well-intended individuals who would like to take it upon themselves to access these devices remotely and perform the cleanup are conscious that they would be breaking laws (not that that stopped the group behind the Wifatch software/malware).

An organized, globally sanctioned effort mounted by law enforcement or security experts might be the right way to go about fixing this, but there are still many things to consider before starting in that particular direction.