If you ask a sysadmin what annoys him or her the most about their job, chances are pretty high that you’ll get, in no particular order, answers like “users”, “faulty updates”, or “calls on a Friday afternoon”. Some will also give you random answers like “after hours’ work” or “having their systems breached”.
Having a ransomware incident on your production servers will tick almost all the boxes. Systems will be down, your users (or customers) will complain, and you’ll have to spend an inordinate amount of time getting everything back in working order.
Now picture this happening in December, and the procedures to get things back in working order taking an expected three weeks. Christmas spirit will be sorely missing.
This situation was precisely what UKG reported a few days ago. A ransomware incident impacted their Kronos Private Cloud systems, hitting everything from Workforce management to Healthcare and the Banking solutions they provide. The silver lining appears to be that their user data was not impacted and remains safe.
This is, unfortunately, occurring more and more frequently.
From their public announcement: “While we are working diligently, our Kronos Private Cloud solutions are currently unavailable. Given that it may take up to several weeks to restore system availability, we strongly recommend that you evaluate and implement alternative business continuity protocols related to the affected UKG solutions”.
Without looking at any other aspects, and purely from a professional standpoint, we’d like to extend our sympathy to the teams trying to get things back to a safe state and working again. We know it’s hard and delicate work, and it requires dedication and expertise to keep calm in a situation like this.
Getting back to the ransomware attack, and without knowing the full details of the situation, we can only talk about the publicly known outcome, which is a long period of downtime for the affected systems. At first glance, it may seem strange why it would take so long to recover all the systems, but there may be multiple reasons for this.
First, it takes some time before you fully understand the scope of an attack and regain the trust necessary to get systems back in production. Restoring backups and realizing the attack happened before the backups were taken just means you’ll be hit again. Alternatively, wiping everything and restarting the whole infrastructure takes time and is very work-intensive.
Additionally, restoring backups is a lengthy process in itself. The sheer amount of data to move from one storage medium to the production systems will bottleneck storage and network, further slowing things down. And for those backup solutions that promise zero downtime, what they do is present a view of the data in the backup medium, which means that being hit by ransomware again would cripple the backup data itself, and that’s a very dangerous proposition.
It all boils down to regaining trust in your systems. Beyond simply restoring data or bringing systems back online, you need to audit everything thoroughly, and again, that takes time to do properly - the last thing you want to see happen in a situation such as this, is restoring the backups correctly and some time later being hit again by some part of the infection you either missed in the initial assessment or having a reinfection because you missed the security gap that let the attacker have access in the first place.
With today’s cybersecurity threats, the best option is not to be attacked at all in the first place. Of course, hindsight is 20-20, and it won’t help UKG’s IT team get their systems back online faster, but it’s a stark reminder of the risks for everyone else. Going over the basic issues like having proper patching in place, ideally with a fast schedule (or even better, a live patching solution), can’t hurt. Make sure older systems are kept up to date (here, extended lifecycle support options are helpful). Keep multiple backups on offline storage to avoid being corrupted by ransomware while stored. Finally, proper regular auditing of everything in your infrastructure. After all, you can't protect what you don't know exists.