Skip Navigation LinksRPS 205 Home > 205 VIBE™ Home > What Happened? Details about the RPS 205 Ransomware and Tech Outage

 What Happened? Details about the RPS 205 Ransomware and Tech Outage

11/13/2019 12:00 AM

​It was a Thursday night – I remember because it was the night of the first Chicago Bears game of the season, and my family's year-old King Charles Spaniel, Baxter, had just peed on the bed.

Our systems at Rockford Public Schools are set up to share text alerts anytime a server or system is disconnected. Sometimes a system will go down and it'll need to be rebooted. Sometimes a system will return on its own. But that night a flood of texts came in around 10 p.m. We had 50 or 60 of our roughly 300 servers go down in rapid succession.

I looked at our server files and saw they had been encrypted. We knew immediately it was ransomware. In every place a file was encrypted, a ransom note was dropped in. We had millions of encrypted files – and the threat actors started encrypting our backups. In hindsight, my military background helped me prepare for this. My telecommunications work started in the U.S. Army. I served two tours of Iraq and spent time in Korea. I'm trained to keep a level head and problem solve one issue at a time. Of course it's a different type of battleground, but I knew this would be reconnaissance.

I got to the Administration Building around midnight to stop the attack and shut down our Internet service to stop any further damage to our servers. Fortunately unplugging the Internet helped isolate at least some of our information.

We did have measures in place to keep our information safe. Our antivirus company said they've never seen that many consecutive variations attack at once. The threat actors had 199 variants. It was like an automatic weapon. It didn't matter how good our antivirus was, we just couldn't keep up. 

That was Sept. 5, more than two months ago. We are still working our way through the effects of the systemwide outage.

This year has brought a significant increase in cyberattacks against school districts, government agencies and hospitals. Three hospitals were recently hit in Alabama and unable to take patients for days. Fortunately were able to continue to have school and employees were paid.

I don't want to scare anyone or make people uncomfortable. But this is an epidemic. It's a serious global issue. This isn't someone sitting in a basement with a grudge against RPS 205. This is organized crime.

Why target Rockford Public Schools?
Public schools, hospitals and government entities are a good target for several reasons. They typically have a low amount of cyber insurance and don't always have top-tier technology or security. They are also public bodies, so it's easier to access information that would assist in an attack. In our case, the attackers – we call them "threat actors" – knew how much our budget was. They knew this time of year – the first week of school – would be an inconvenient time to hit, as opposed to sometime over the summer.

We can't share many details of this specific attack on RPS 205, because it's part of an ongoing investigation. But the easiest and most common way to get a username and password is to send an email with malicious items and trick a person into clicking on an email or downloading what appears to be a safe PDF. If you're clicking on and downloading things daily, you might lose your sense of cautiousness, and some of the phishing emails look very sophisticated. A virus can install on a device with one click. Depending on that user's access level, it can spread like wildfire.

No evidence showing personal info was compromised
When the attack happened, employees were paid for their work the previous two weeks. This allowed us to focus our attention on BusinessPLUS and eSchool – our student database and operations systems. We were able to retain the data in those systems, and we pushed that data to the Cloud so we could continue to run business. As Dr. Jarrett shared, we questioned early on if we could even have school during the outage. Things could have been worse. Fortunately our backups were largely unaffected. Now we have more complex ways to back up our systems and information, and we're working on staff training to ensure we're as secure as possible.

Most importantly, I'm happy to share that we still don't have evidence to show that any personal student or staff information was compromised. That's key.

Where we are now
It's been a long two months. We've had a lot of frustrated staff members and parents. Ensuring that all devices are clean before they return our network is a massive undertaking. We took every single device off our network, then brought each device back onto our network one by one. We have more than 5,000 computers, 300 servers and 1,200 wireless access points, and we had to rebuild our systems from the ground up. Sure, we could have paid the ransom. But that would only unlock our files. Our systems would have remained infected, and we'd still have to go through this intense clean-up work. We would also be vulnerable to another attack.  

We communicated updates daily to our staff and the public for a month. Now we're sharing updates directly with principals to share with staff – and with students and parents when needed. Computers should be back to all teachers by mid-November. Access to personal drives should be restored around that time, too.

The cost of the ransomware attack
We don't have a final cost breakdown for the outage. Fortunately our insurance will cover some of the cost. But it's difficult to put a price tag on the amount of work we lost, or the cost of manually handling things like payroll or inputting data after tracking it manually. Our IT team has poured in several hundred hours of overtime. They have literally worked around the clock to restore systems as quickly and effectively as possible.

What's next for RPS 205
In addition to rebuilding our systems, we are prioritizing plans to better secure our systems. I can't share specifics, but our plans include a combination of training, internal and perimeter protection, backups and disaster recovery. We're researching ways to additionally strengthen our password and access protocols. Staff: Look for training details in the coming weeks. In short, we're aware of our vulnerabilities, and we're already more secure than we were a few months ago.

The Rockford School Board has also recently approved a refresh for our devices – this was planned ahead of the ransomware attack and subsequent systems outage. The refresh includes the following:
-1,315 laptop computers for teachers and staff
-3,556 desktop computers for staff use and student labs
-374 computers for Project Lead the Way
-44 laptop computers for student use for eSports
-3,456 computer monitors for staff use and student labs
-100 computer monitors for Nutrition Services staff

A majority of the district's Windows-based desktop and laptop computers have reached the end of their lifecycle. In May, the district's Information Technology team established a minimum hardware standard for the district, and this computer refresh plan will update desktop and laptop computers to align with that standard.

Computers will be leased for 36 months – or three years – for a total of $3,114,251.72. The bid came in more than $2 million less than the district's consortium pricing. At the end of the three-year lease agreement with CDW-G, the district will have the option to purchase computers. The goal is to have all computers in schools by spring, so stay tuned for upgraded devices. The team and I are looking forward to working more with staff and students on ensuring our systems are safe and secure.

/205VIBE/PublishingImages/Post%20Photos/JasonBarthel_TechOutage_860x365.jpg
No
To Top