Rackspace internal monitoring web servers hit by zero-day

Intruders accessed machines via tool bundled with ScienceLogic, 'limited' info taken, customers told not to worry

Exclusive Rackspace has told customers intruders exploited a zero-day bug in a third-party application it was using, and abused that vulnerability to break into its internal performance monitoring environment.

That intrusion forced the cloud-hosting outfit to temporarily take its monitoring dashboard offline for customers.

Reading between the lines, it appears Rackspace was hosting a ScienceLogic-powered monitoring dashboard for its customers on its own internal web servers, those servers included a program that was bundled with ScienceLogic's software, and that program was exploited, using a zero-day vulnerability, by miscreants to gain access to those web servers. From there, the intruders were able to get hold of some monitoring-related customer information before being caught.

"On September 24, 2024, Rackspace discovered a zero-day remote code execution vulnerability in a non-Rackspace utility, that is packaged and delivered alongside the third-party ScienceLogic application," a spokesperson for Rackspace told The Register Monday.

Yes, it not only discovered that flaw in the third-party utility, it realized it had been exploited.

Rackspace uses a ScienceLogic stack internally for system monitoring and providing a dashboard to users. ScienceLogic, which supplies IT infrastructure observation software, did not immediately respond to a request for more information about the exploitation.

Abusing this zero-day vulnerability gave the criminals access to three of Rackspace's internal monitoring web servers, "and some limited monitoring information," a Rackspace spokesperson told us, adding:

Customer performance monitoring was not impacted by this event. The only impact to customers was the inability to access their associated monitoring dashboard. There was no other customer service disruption as a result of this event.

A letter sent to Rackspace customers and shared earlier with The Register by a reader provides additional details about what the crooks accessed. It notes that "limited" internal monitoring information included: Customer account names and numbers, customer usernames, Rackspace internally generated device IDs, names and device information, device IP addresses, and AES256 encrypted Rackspace internal device agent credentials.

We've asked Rackspace for more details, such as how many customers were affected, regarding this cyber close shave. 

The letter customers received also says there is no need for them to take any remediation steps, but "in an abundance of caution, we commenced rotation of the Rackspace internal device agent credentials."

"There was no other customer service disruption as a result of this event," the biz told its clients. "No other Rackspace products, platforms, solutions, or businesses were affected by this event. We have actively notified all affected customers and are updating customers as appropriate."

Rackspace also assured us that upon spotting the security breach, it immediately isolated the affected equipment, took them offline, and then worked with ScienceLogic to develop and apply a patch. 

"ScienceLogic has notified their customers, and we have actively notified Rackspace customers utilizing this third-party monitoring service," the spokesperson said.

In December 2022, the IT provider's hosted Microsoft Exchange service was hit by a ransomware infection, which shut down email services to thousands of customers, most of whom were small and mid-sized businesses.

The company's expenses related to that cyberattack, also a result of a zero-day exploit, hit about $11 million, Rackspace said in a 2023 regulatory filing. ®

Updated to add at 0100 UTC, September 30

While we continue to press ScienceLogic to identify the third-party application that was exploited, the biz has told us the vulnerable program was bundled with its SL1 monitoring product, and that it is pushing out a fix to its clients.

"We identified a zero-day remote code execution vulnerability within a non-ScienceLogic third-party utility that is delivered with the SL1 package, for which no CVE has been issued," a spokesperson for ScienceLogic told us.

"Upon identification, we rapidly developed a patch to remediate the incident and have made it available to all customers globally. We are focused on assisting our customers in implementing the fix to minimize their risk. We will continue to update customers as appropriate."

Updated to add at 0400 UTC, September 30

ScienceLogic has declined to identify the exploited bundled software. A spokesperson told us: "At this point, we are not naming the third-party utility to diminish potential risk to our customers. We are aware of only one instance when the vulnerability has been exploited and a patch was quickly provided."

That one instance being, so far, Rackspace.

Speaking of which, the cloud virtual server host has been in touch to stress that it was its self-hosted ScienceLogic dashboard that was hit via the zero-day and that while that web interface was taken offline, its monitoring services continued to run.

"Rackspace’s monitoring functionality is not dependent on the ScienceLogic dashboard," a spokesperson informed us.

"Our Rackspace monitoring functionality was not impacted and there was no interruption to our monitoring and alerting services for our customers."