Tech+ Blog: Bay Area IT Security & Consulting News

A Bay Area Law Firm Client of Ours Got Hit with Ransomware

Jan 22, 2019

This is the story of how cyber criminals in China attempted to take down and extort for ransom an Oakland Law Firm, how us the Tech+ IT team reacted, and what we can learn from it — the importance of cybersecurity in 2019.

It was Monday, January 7th; the first full week after the holidays. The day began like any other Monday — we gathered over coffee, discussed the craziness of our weekends, and after the team meeting Sean began a routine review of out client servers. A case had come in over the weekend that some documents were slow to load, and we noticed a small spike in hard drive activity.

What soon entailed was a series of frantic phone calls between Sean and myself followed by gut-wrenching pain as the gravity of the situation sunk in. The day I had been dreading since I founded the company in 2014 was finally here — we had a full-fledged ransomware attack underway. Unfortunately as hard as we try to prevent attacks with security measures there is no surefire way to prevent ransomware attacks. In the worst cases, the bad guys succeed, data is lost, and ransoms are paid out. Fortunately for us, we were well prepared as the client utilized a reputable cloud-based disaster recovery system sodata loss was limited.

The first thing we did once we confirmed the attack was have everyone power off their workstations asonce ransomware compromises one machine it immediately spreads and that’s when things get worse and even with backups in place recovery takes time. Restoring a single machine can take 1-2 hours and when handling dozens of machines that can easily turn from hours into days.

With everything powered off we started slowly checking each server one by one taking samples of the encrypted files so we could send them out for analysis. After submitting the samples to IT Security Researchers we quickly discovered we were dealing with something incredibly nasty: The Dharma -Adobe variant of Ransomware (.cezar family of attacks)[1] This strain is extremely problematic.

Only 1 in 67 antivirus engines could even detect it.

In fact it successfully made its way through the client’s Cisco Meraki Firewall with Advanced Security License, the Mimecast email security filtering, Microsoft Office 365 mail scans, and past our anti-virus protection. 
Even with recent backups available checking each system individually, completing the restores and testing to determine which backups weren’t compromised while watching in real-time as Chinese cyber criminals attempted to login to our servers (we blocked some 7,000 attempts per hour at the height of it) took three people in our team in excess of 100 hours of work that week. It was 14+ hour days and a ton of stress. Both Sean Carroll and Milan Trivedi went above and beyond showing up early and staying late.

In the end:

• We caught the attack and powered down all workstations before any systems were encrypted and before we even got the ransom demand messages.
• Our client lost 1-2 days of data (Chinese hackers infected the systems but waited a few days to detonate the payload, so we chose to restore from a backup image taken when we knew 100% of the data was unaffected.)
• Our client had only a single day of complete downtime, followed by another couple days of interrupted workflow as we got them up and running on temporary systems while we rebuilt the infrastructure.
• Our team put in 12-14 hour days all week working round the clock to recover from backups and prevent further attacks.
• Sean Carroll will be taking a well-deserved personal leave of absence starting today.
• We learned that Cybersecurity user training is more critical than ever (stay tuned for an email with cybersecurity tips as well training offerings for clients not already using our preferred eLearning platform: KnowBe4.)
• We saw firsthand how valuable proper disaster recovery backups are; a file/folder backup is not enough if you want to be able to recover from an attack swiftly. Without recent system images of the servers the backup could have taken 1-2 weeks to rebuild all the infrastructure rather than a day.
• We want to remind our clients just how important good passwords are. If you don’t have a password policy in place we will be touching base shortly to offer one.

We want to thank our client for all their patience and understanding during the attack. It was a huge inconvenience to not be able to work for a day and not once did anyone yell, curse, or make unreasonable demands of our team. Instead we worked together and ensured a smooth recovery.