Rackspace Blames Zero-Day Exploit for Ransomware Hit SuccessHosting Giant Says Microsoft's Patch Notes Didn't Detail Remote Code Execution Risk
Rackspace says the ransomware-wielding attackers who disrupted its hosted Microsoft Exchange Server environment last month wielded a zero-day exploit to gain remote access to servers, multiple news outlets report.
Cybersecurity firm CrowdStrike has described the underlying flaw as being "a previously undisclosed exploit method for Exchange." It says in a Dec. 20, 2022, blog post that while Microsoft released a patch for Exchange last November to address ProxyNotShell, it didn't detail - or know - that unless organizations installed the patch, rather than using recommended mitigations, this new exploit method could be used to gain direct, remote access to Exchange servers via the Outlook web client.
Microsoft didn't immediately respond to a request for comment.
Rackspace, based in San Antonio, Texas, says the FBI and CrowdStrike have attributed the attack, which it first disclosed Dec. 6, 2022, to the Play ransomware group, San Antonio Express-News and other media outlets have reported.
Thousands of Rackspace customers globally have been affected by the attack. Beginning on Dec. 1, 2022, it left them unable to access their hosted Microsoft Exchange email.
As part of its response, Rackspace has ceased offering hosted Exchange services and is helping customers migrate to Microsoft 365, although it has offered other approaches too. These migration efforts, as well as getting pre-Dec. 2 email restored for some customers, remain ongoing.
Rackspace is the world's largest managed cloud provider, counting more than 300,000 customers worldwide, including two-thirds of the world's 100 largest publicly traded businesses. The firm says its hosted Exchange business comprises about 1% of its total annual revenue and is primarily compromised of small and medium-size organizations.
The Play attack appeared to leave Rackspace's hosted Exchange servers either encrypted or otherwise disrupted due to the company's response to the attack. The company didn't immediately respond to a request for further details.
Rackspace didn't immediately apply the Exchange patch released last November by Microsoft because of multiple user reports that the patch was causing errors, including leaving Microsoft Outlook Web Access - OWA - inaccessible, San Antonio Express-News reports, citing an unnamed, outside adviser to the company. Pending a fully working patch, Rackspace reportedly instead used mitigations recommended by Microsoft.
'Proactively Shut Down'
Rackspace previously reported that on Dec. 1, 2022, it "experienced a significant failure in our Hosted Exchange environment" and in response, "proactively shut down the environment to avoid any further issues while we continue work to restore service" and identify "the root cause of the issue." The company also brought in Austin, Texas-based cybersecurity incident response firm CrowdStrike to assist and investigate.
"Due to swift action on the company's part in disconnecting its network and following its incident response plans, CrowdStrike has confirmed the incident was quickly contained and limited solely to the Hosted Exchange Email business," Rackspace reported in a Dec. 9, 2022, Form 8-K filing to the U.S. Securities and Exchange Commission.
Rackspace has apologized for the disruption to customers. It has also warned investors that the attack could affect its hosted Exchange revenue, which accounts for about $30 million annually, as well as lead to unexpected costs. But the firm says in its filing: "Rackspace maintains cybersecurity insurance commensurate with the size of its business, and is confident in its ability to absorb potential financial costs associated with the incident and fulfill its obligations to other customers."
(2/7): As the process remains underway, we want to remind customers that due to the nature of the incident, certain elements of email and other data may remain unavailable to our customers.— Rackspace Technology (@Rackspace) December 27, 2022
Rackspace stock closed at $2.91 per share Tuesday, down 41% from $4.94 on Dec. 1, 2022, the day before service disruptions tied to the attack began to manifest.
Just days after Rackspace warned it had been the victim of a suspected ransomware attack, it was hit with a class action lawsuit accusing it of negligence.
Recommended: Microsoft 365 Migration
Rackspace says it is continuing to restore customer emails but that process is not yet complete. The company has detailed a five-step process it's following: restoring old systems in an isolated environment, recovering data from affected servers, prepping a new environment to handle data extraction, extracting data and, finally, recovering the email data.
Rackspace continues to recommend that all affected customers move to Microsoft 365 and says it has been providing them with "Microsoft Exchange Plan 1 licenses on Microsoft 365" at no additional cost, at least for now.
Two alternatives being offered by Rackspace are to move Hosted Exchange customers' mailboxes to Rackspace Email or to "use internal relay to maintain mail routing for your hybrid setup."
"Crucially the status page shows they will indeed entirely abandon their Hosted Exchange service - customers will be given a PST export of mailboxes at some point in the future, and that's a wrap on the product," says British cybersecurity expert Kevin Beaumont in a post to Mastodon. A
.pst file is an Outlook Data File export.
CrowdStrike Details Zero-Day Exploit
The Exchange patches released by Microsoft last November included fixes for a pair of zero-day vulnerabilities that were being used for in-the-wild attacks first spotted in September 2022.
The pair of flaws are together known as ProxyNotShell, for their resemblance to the trio of Exchange flaws found in 2021 known as ProxyShell.
The first ProxyNotShell flaw - CVE-2022-41040 - is a server-side request forgery vulnerability attackers can use to access back-end servers. The second flaw - CVE-2022-41082 - allows remote code execution when Remote PowerShell is activated. By exploiting the first flaw, attackers can trigger the second.
CrowdStrike reported in its blog post on Dec. 20 that after it investigated "several Play ransomware intrusions where the common entry vector was confirmed to be Microsoft Exchange," the suspicion was that attackers had used the two ProxyNotShell flaws in concert. But it says that turned out to not be the case.
Instead, via an exploit method it has dubbed OWASSRF, attackers were able to pair a Microsoft Exchange Server elevation of privilege vulnerability - CVE-2022-41080 - with the second part of the ProxyNotShell attack chain, CVE-2022-41082, "to achieve remote code execution (RCE) through Outlook Web Access (OWA)."
CrowdStrike adds: "After initial access via this new exploit method, the threat actor leveraged legitimate Plink and AnyDesk executables to maintain access, and performed anti-forensics techniques on the Microsoft Exchange server in an attempt to hide their activity."
While CrowdStrike's blog post does not name Rackspace as being one of the organizations attacked in this manner, the hosting giant's CSO, Karen O'Reilly-Smith, has confirmed that the CVE-2022-41080 vulnerability was used.
"Microsoft disclosed CVE-2022-41080 as privilege escalation vulnerability, and did not include notes for being part of a remote code execution chain that was exploitable," O'Reilly-Smith told Cybersecurity Dive.
CrowdStrike recommends all organizations that use Microsoft Exchange apply the November 2022 Exchange patch - labeled KB5019758 by Microsoft - "to prevent exploitation since the URL rewrite mitigations for ProxyNotShell are not effective against this exploit method."
"If you cannot apply the KB5019758 patch immediately, you should disable OWA until the patch can be applied," it says. Anyone running Exchange servers should also "follow Microsoft recommendations to disable remote PowerShell for non-administrative users where possible," it adds.
CrowdStrike has released a script via GitHub for Microsoft Exchange server logs that organizations can use to watch for signs of the OWASSRF exploit method being used.