City of Dallas Ransomware Incident Report

Dr. Srinidhi Varadarajan, Chief Scientist, Elastio

ITS Risk Management, Security and Compliance Services Report, Sep. 2023 Synopsis

The following is my synopsis of an excellent report from the City of Dallas ITS Risk Management on a significant ransomware attack this year. In contrast, most such incidents involve private entities with buried analyses that are heard through the grapevine. The rest of the article breaks down the phases of the spread. Please note all sections below are taken verbatim from their report.

Threat Actor: Royal

Statistics

In the year 2022, ransomware victimized over 70 percent of organizations, marking a surge compared to the preceding five years and establishing the highest recorded proportion to date. The incidence of ransomware exhibited a noteworthy annual growth rate of 13% during 2022, surpassing the cumulative increase of the preceding five years. Furthermore, the number of public ransomware victims escalated by 38% when compared to the initial quarter of 2023 and demonstrated an astounding 100% surge from the second quarter of 2022. This denotes a substantial 75 percent upswing in the mean count of monthly attacks in the United States between the initial and latter halves of the preceding 12-month period.

Spread Phase

The initial entry point was established through the utilization of [a] service account that connected to a server. Leveraging this initial access, the threat actor cleverly navigated the internal infrastructure of the City by exploiting legitimate third-party remote management utilities. The Royal group constructed what is typically known as “Beacons” using remote management utilities and legitimate pen-testing technologies to traverse the City’s internal network. These actions provided staging for Royal to exfiltrate an estimated 1.169 TB of data through the initial impacted server. In addition to data exfiltration, the Threat Actor’s credential harvesting techniques provided a list of users, accounts, and devices.

Timeline of Activities
Intrusion or Recovery Action/Activity Approximate Date
Likely entry into network April 07, 2023
Intrusion Surveillance Phase April 07 – May 03, 2023
Account Credentials First Obtained April 07 – May 03, 2023
Attack Beacons Installed April 07 – May 03, 2023
Identification of Possible Attack May 03, 2023, at 2:54 am
Security Team Begins Analysis May 03, 2023, at 2:54 am
Likely Lateral Movement through Network Begins May 03, 2023, at 2:54 am
Attack Mitigation Procedures Initiated May 03, 2023, at 5:00 am
Bridge Call Opened May 03, 2023, at 5:23 am
Sanitation Servers Identified as affected May 03, 2023, at 5:32 am
Set of Servers Identified as Infected May 03, 2023, at 6:00 am
Team expanded to include additional IT disciplines May 03, 2023, at 7:00 am
Disaster Recovery Manager Notified of Ongoing Incident May 03, 2023, at 7:46 am
Citywide Announcement of Widespread Service Outage Made to City Staff May 03, 2023, at 8:05 am
IT Executive Leadership Notified of Ongoing Incident (CIO, CFO) May 03, 2023, at 8:22 am
Incident Response Plan (IRP) Initiated May 03, 2023, at 8:30 am
Communication to Federal Authorities May 03, 2023, at 8:30 am
City’s Office of the City Attorney (CAO) and Office of Emergency Management (OEM) Notified of Ongoing Incident May 03, 2023, at 8:30 am
Preservation of Evidence Procedures Initiated May 03, 2023, at 8:31 am
Notification to City Mayor and Council of Ongoing Incident May 03, 2023, at 9:05 am
Preservation and Restoration of Public Safety CAD Services Set as a Priority May 03, 2023, at 9:35 am
Infected Server Inventory Tracking Initiated May 03, 2023, at 9:44 am
Content of Royal README.txt Message Shared with Incident Team May 03, 2023, at 9:45 am
Critical Public Safety Servers Infected May 03, 2023, at 11:10 am
Begin Disconnecting Servers May 03, 2023, at 11:00 am
Rebuild of CAD Servers Begin May 03, 2023, at 12:00 pm
News Outlets Announce the Attack to the Public May 03, 2023, at 12:30 pm
New Servers Become Infected May 03, 2023, at 1:22 pm
Infected Databases Identified May 03, 2023, at 1:30 pm
Print Servers Are Disconnected May 03, 2023, at 2:11 pm
Initial Analysis Determines 173 Servers Are Impacted May 03, 2023, at 2:15 pm
Multiple Domains Impacted May 03, 2023, at 2:15 pm
Assessment that Multiple Departments Impacted May 03, 2023, at 2:15 pm
Server Reinfection Confirmed May 03, 2023, at 5:00 pm
Additional Blocks by CrowdStrike May 03, 2023, at 5:30 pm
Confirmation of a Development Services server infected May 03, 2023, at 6:00 pm
Confirmed Database GIS Servers Infected May 03, 2023, at 6:09 pm
Malware Execution Extinguished by the City May 04, 2023, at 5:58 am
Incident Support Team (IST) Activation May 08, 2023, at 9:00 am
Incident Support Team (IST) De-activation June 09, 2023, at 5:00 pm
Ransomware Detection | Elastio Software

Impact

As required under federal law and using different metrics for the inclusion of individuals, the Department of Health and Human Services (HHS) was notified that the Sensitive Personal Information (SPI) and Protected Health Information (PHI) of 30,253 individuals were potentially exposed by the activities of Royal. The OAG’s website indicated that personal information such as names, addresses, social security information, health information, health insurance information, and other such information was exposed by Royal.

To date, The Dallas City Council has approved a budget of $8.5 million in computer-based interdiction, mitigation, recovery, and restoration efforts directly tied to the Royal ransomware attack. The City has dedicated a total of 39,590 hours towards the comprehensive remediation effort, of which ITS methodically documented 14,158 hours.

Recovery

Recovery endeavors necessitated a temporary pause due to the incomplete neutralization of the malicious executables through EDR and its ability to propagate throughout the network ecosystem.

In the final analysis, it was ascertained that the event led to the impairment of 230 servers, necessitating comprehensive endeavors for their complete restoration and recovery through available backups. Among these affected servers, the City successfully retired more than 100 surplus servers [there was some goodness here out of all this], hosting outdated applications, unsupported operating systems, or deemed non-essential for crucial municipal services. The cumulative count of 1,398 endpoint devices went through reconstruction directly due to the effects of the Royal ransomware infection.

About Elastio

Elastio detects and precisely identifies ransomware in your data and assures rapid post-attack recovery. Our data resilience platform protects against cyber attacks when traditional cloud security measures fail. Elastio’s agentless deep file inspection continuously monitors business-critical data to identify threats and enable quick response to compromises and infected files. Elastio provides best-in-class application protection and recovery and delivers immediate time-to-value.  For more information, visit www.elastio.com.