Rackspace Hosted Exchange service outage caused by security incident
Cloud computing company Rackspace has suffered a security breach that has resulted in a still ongoing outage of their Hosted Exchange environment.
“In order to best protect the environment, this will continue to be an extended outage of Hosted Exchange,” the company said on Sunday.
The Rackspace outage
The connectivity issues for Rackspace Hosted Exchange customers – mostly small to medium size businesses – started on Friday (December 2), with users experiencing errors when accessing the Outlook Web App (Webmail) and syncing their email clients.
It took 18 hours for Rackspace to realize that the problem will not easily be fixed and to decide to offer another option to disgruntled customers: a free Microsoft Exchange Plan 1 license on Microsoft (i.e., Office) 365.
They provided customers with instructions on how to make the switch and said that their support team is available for assistance but, apparently, the self-migration process is not as simple to pull off as they hoped and their customer support got overwhelmed.
“Since our last update, we have mobilized roughly 1000 support Rackers to reduce wait times and address ticket queues,” the company said on Sunday, then followed up with the decision to contact every Hosted Exchange customer by phone or email.
The company also offered a stopgap solution until the M365 onboarding can be performed. “You can also implement a temporary forwarding that will allow mail destined for a Hosted Exchange user to be routed to an external email address,” they said. Unfortunately, this option also requires the help of the support team and comes with some limitations (e.g., email sent before the rule is put into place will not be forwarded).
Despite Rackspace saying that they’ve “successfully restored email services to thousands of customers on Microsoft 365”, Twitter is still full of customer complaints about the wait times and/or inability to get help from the support team.
According to the Rackspace system status page, Rackspace’s other offerings – Email, Administrator Tools, and Apps, have been and are operating normally.
A service-destroying event
What caused the incident is unknown, but security researcher Kevin Beaumont has noticed that, a few days ago, the Shodan search engine was showing Rackspace’s Exchange clusters running an Exchange version from August 2022, which means that they did not have the patches for the ProxyNotShell vulnerabilities.
Whether or not the security incident happened due to the exploitation of these vulnerabilities, I think many Rackspace Hosted Exchange customers will end up not switching back after the outage has ended and are now worrying whether they will be able to recover their legacy email data from Rackspace.
Beaumont urged MS Exchange administrators to implement the ProxyNotShell patches because the mitigations provided by Microsoft can be bypassed.
“If you run Exchange in Hybrid mode or have on prem servers for management — which is extremely common for almost every Exchange Online customer — you still need to take action. The Microsoft blog says no action is required for Exchange Online. This is wordsmithing by Microsoft, and is wrong — those on prem Exchange Servers you forgot about and rely on still need fixing,” he warned, and offered advice on checking whether their servers have been compromised in the meantime.
UPDATE (December 6, 2022, 07:10 a.m. ET):
The Rackspace Hosted Exchange infrastructure and service is still down.
The company says that getting in touch with their support team is still a time-consuming affair – call queue hold times are 2-3 hours – and advises customers to request a call via their callback feature.
They also pointed out that customers who had previously subscribed to their Archive service for their mailbox and users can “recover their archived mail as a .pst and import into their new M365 profile.”
Unfortunately, there is still no word on how the security incident happened.
UPDATE (December 6, 2022, 02:40 p.m. ET):
Rackspace has finally confirmed the cause of the ongoing outage of is ransomware.