Cybercrime , Cybercrime as-a-service , Endpoint Security
The Ransomware Files, Episode 5: Texas and REvil
The State Recovered Quickly But a MSP Was Irreparably Damaged Jeremy Kirk (jeremy_kirk) • February 22, 2022 40 MinutesIn August 2019, 23 cities across Texas were struck by one of the largest ransomware attacks ever in the U.S.
The attack, which involved the REvil/Sodinokibi ransomware, started after a small managed service provider's remote access software was compromised. While the cities recovered quickly, the managed service provider sustained irreparable damage, which shows the devastating consequences ransomware can have on a small business.
This episode of "The Ransomware Files" reveals never-before-public details about the attack in Texas, describes how the state recovered so quickly and explores the human cost of ransomware.
Rick Myers and his wife, Diana, run the MSP, which is called TSM Consulting and is based in Rockwall, Texas.
"We lost customers because of it [the attack]," Myers says. "And anytime you lose a customer's data, you stand a good chance of losing business. Many of the customers we lost have been with us for decades, literally decades. It has taken a toll on me that I don't know that I can recover from."
The cities couldn't run payroll, citizens couldn't pay bills and critical public safety records couldn't be accessed. But Texas had been planning and practicing recovering from a major cybersecurity incident for several years. Gov. Greg Abbott declared the incident a statewide disaster, the first such declaration.
The cities were up and at least partially running in just eight days due to a massive effort from Texas state agencies, Texas A&M University, the National Guard and vendors.
Andy Bennett is former deputy chief information security officer for the Texas Department of Information Resources. He's now vice president of technology and chief information security officer with Apollo Information Systems.
"There was a veritable army assigned to this incident," Bennett says. "We had a ton of folks out in the field. We had a ton of folks there in the Security Operations Center. We had analysts running from city to city."
And in November 2021, an unexpected development occurred. U.S. prosecutors announced an indictment against a Russian man allegedly responsible for the attack against Texas. The indictment marked an escalating effort by the U.S. government to hold ransomware attackers accountable (see: REvil Ransomware Suspects Snared in Global Police Crackdown).
"The Ransomware Files" is a podcast miniseries available on Spotify, Apple Podcasts, Google, Audible, Stitcher and more. I'm speaking with those who have navigated their way through a ransomware incident to learn how they fought back and what tips they can pass on to others. No ransomware infection is ever welcomed. But there's invaluable knowledge gained. There should be no shame in getting infected, and it's important to share the lessons.
If you enjoyed this episode of "The Ransomware Files," please follow it on a podcast platform and leave a review. Also, the show has a Twitter handle, @ransomwarefiles, that tweets news and happenings about ransomware.
If you would like to participate in this project and tell the information security community about your organization's brush with ransomware, please get in touch with me at jkirk@ismg.io or direct message me here on Twitter. I'm looking for other people, organizations and companies that can share their unique experiences for the benefit of all until ransomware, hopefully, becomes a thing of the past.
Credits
Speakers: Rick Myers, founder, TSM Consulting; Nancy Rainosek, chief information security officer, state of Texas; Mandy Crawford, chief information officer, state of Texas; Andy Bennett, former deputy chief information security officer and now vice president of technology and chief information security officer with Apollo Information Systems; Danny Miller, chief information security officer, Texas A&M University System; Jeremy Kirk, executive editor, Information Security Media Group.
Production coordinator: Rashmi Ramesh.
"The Ransomware Files" theme song by Chris Gilbert/© Ordinary Weirdos Music.
Music by Uppbeat. (Tracks and license codes here) and from Podcastmusic.com.
Sources
- Bankinfosecurity.com, Texas Ransomware Responders Urge Remote Access Lockdown, Sept. 6, 2019.
- The Daily Mail, EXCLUSIVE: REvil 'super-hacker' wanted by FBI for 'using ransomware to fleece millions of dollars' from Americans is unmasked by DailyMail.com in his plush hideout in Siberia as Kremlin turns blind eye, Nov. 29, 2021.
- NPR, Texas Towns Hit With Ransomware Attack in 'New Front' of Cyberassault, Aug. 20, 2019.
- Statescoop, How Texas used its disaster playbook after a huge ransomware attack, Oct. 15, 2019.
- Texas Department of Information Resources, U.S. Justice Department Announces Indictment Against REvil Ransomware Suspect Behind 2019 Ransomware Attack on Texas Municipalities, Nov. 8, 2021.
- USA Today/The Associated Press, Texas ransomware attack shows what can happen when whole towns are targeted, July 26, 2021.
- Victoria Advocate, Jackson County fights to recover as computers remain under ransom, June 22, 2019.
Jeremy Kirk: If you call TSM Consulting in Rockwall, Texas, there's a good chance Rick Myers will answer the phone. If he doesn't answer, his wife, Diana, probably will. They run a small IT services company. The two started around 1997. In the beginning, Rick says:
Rick Myers: “I fired her, she fired me, and probably for the first six months until we sat down on roles, and said, this is your job, this is my job. I don't mess with your stuff, you don't mess with mine, and we've kept that in place ever since. And it's worked. She does the accounting, and I handle the technical aspects of it.”
Kirk: TSM Consulting provides IT support to small cities across the vast state. It started off by installing network connections to access the Texas Law Enforcement Telecommunications System or TLETs. It’s a huge network that allows access to a variety of local, state and national databases. Those databases hold everything from driver's license records to criminal files. In the early years of his business, Rick's clients often asked him to help with other IT-related tasks. TSM evolved into a kind of light managed services provider. It was a successful small business that at one time had four or five employees. Rick and Dianna have now been together as a couple for 52 years.
Rick Myers: “We grew it up to a very nice, comfortable living.”
But two-and-a-half years ago, Rick was thrust into an unenviable position. His small company ended up at the center of, what was, at that time, the largest ransomware attack in the United States. The attackers gained access to TSM's remote access software that the company uses to manage the IT networks of those small Texas cities. Then, the attackers installed ransomware on the systems of 23 cities. Cities couldn't run payroll, citizens couldn't pay bills, and critical public safety records couldn't be accessed. The attack was shocking in scale. The attackers were demanding $2.5 million in Bitcoin, which was a near-record demand for the time.
It was also a sign of what was to come. Ransomware gangs have increasingly taken aim at managed service providers. It allows them to amplify the damage they cause by exploiting the remote access of those managed service providers have into their clients networks. In 2021, the same ransomware that hit Texas was also used against an American software company, called Kaseya. The group is known as REvil or Sodinokibi. And until recently, it was one of the most prolific ransomware groups. This and the next episode of The Ransomware Files are going to focus on the attacks against Texas and Kaseya. How organizations have recovered and the human cost.
But while the cities in Texas recovered with remarkable speed due to the state's diligent preparation, Rick and Diana's business sustained irreparable damage. This episode is going to reveal new, never-before-public details about the attack. Rick hasn't spoken publicly before to this extent about it and its effect on his business. There are lots of news stories about who was infected with ransomware and who might have paid a ransom. What you don't hear about as often is what ransomware did to real people, how cybercriminals thousands of miles away can take almost everything away.
This is The Ransomware Files. I'm Jeremy Kirk.
In this podcast mini-series, I'm speaking with those who have navigated their way through a ransomware incident and learn how they fought back and what tips they can pass on to others. No ransomware infection is ever welcomed. But there's invaluable knowledge gained. There should be no shame in getting infected, but it's important to share the lessons.
For small cities, managed service providers are an important part of how they maintain their information technology systems. These cities may have just a few people responsible for everything that touches on IT - from customer relationship management systems to billing systems to desktop computers. MSPs like TSM Consulting help fill that gap.
Rick and another TSM employee were on call in the early morning hours of Friday, August 16, 2019. Around 2 a.m., Rick received a phone call from a police department. The person said their system was acting slowly. Rick used remote access software called ScreenConnect to access the server. ScreenConnect is now known as ConnectWise Control. Rick says the server kept badgering him to install something, which he ignored and rebooted it.
Myers: “About 20 minutes later, I get another call from the same department. One of the officers in the field had a ransomware message on it.”
Kirk: “In their car?”
Myers: “In their car.”
Kirk: “Oh, my.”
Rick called his technician just before three in the morning. Other government departments were facing the same issues.
Myers: “And several machines showing ransomware had been installed on the machine, and the pop-ups said, send your Bitcoin to here and it will decrypt your data. So after some conversation, we determined that we needed to go into the office, figure out what was going on. And we realized that was a good possibility we were the source of the problem. So we brought down our ConnectWise server at that point in time.”
Just 13 minutes before Rick received that first call, the attackers had commandeered TSM's ScreenConnect software and were using it to deploy the ransomware. Within about 90 minutes, 740 computers in 23 cities were infected.
Andy Bennett: “These local governments are the lifeblood of Texas.”
That's Andy Bennett. He's the former Deputy Chief Information Security Officer for the Texas Department of Information Resources, known as DIR. The department helps state and local governments with strategic direction for IT, implementation, and cybersecurity. Bennett is now Vice President of Technology and Chief Information Security Officer with Apollo Information Systems, a consultancy. He described himself as a straight-shooting, boot-wearing native Texan.
Kirk: “So, are you wearing your boots now? Perfect, perfect.”
Bennett: “Let the record show, I presented my boots to the camera.”
Andy had a feeling that what was unfolding that day was going to be big.
Bennett: “It was either 8:31 or 8:36 in the morning when the first call came in. And it was somebody telling me that there seems to have been an incident of some sort and that it may impact eight municipalities. And I was like, well, that's no good. The calls started coming in and you know, the number started to climb.”
In the two years leading up to the August 2019 attack, Texas has been keeping a close eye on the changing cybersecurity landscape. And it had been preparing, which was good because:
Nancy Rainosek: “2019 was when ransomware started showing up across the state.”
Nancy Rainosek is the Chief Information Security Officer for the State of Texas.
Rainosek: “I was teleworking that day, and Andy called me at home, and a call from Andy on a Friday was never a good thing. And because it just always seems like it happens then. And so we put the plan into motion.”
The plan had been in the works for a while. In 2017, the state's legislature passed the Texas Cybersecurity Act. The Act aimed to bolster the state's overall cybersecurity readiness that included continuing training and education. Andy says the Department of Information Resources also created an incident response team, Redbook. Redbook contains the procedures an agency should follow after a privacy or information security incident.
Bennett: “And we had already been considering, well, what would happen if there was a massive cyber incident in Texas, right? How would we handle them? And then, the legislature added extra emphasis to it and mandated that DIR go forth and create an actual plan and be ready for this. And so we did that. And that's exactly what we did. And we built a coalition of all kinds of different agencies and capabilities and personalities that came from all kinds of different places across the state to try to be prepared for what might come.”
In 2017, authorities in Texas also gained the power to declare a cyber event as a disaster. It had been used earlier that year. Judge Jill Sklar declared a countywide disaster in Jackson County in May 2019. The county is about 100 miles southwest of Houston. The county's computers were infected with the Ryuk ransomware and the attackers demanded $362,000 in Bitcoin. The county didn't pay though, and with the help of the Texas Military Department and IT provider and other state agencies, it knocked out six months of recovery work in just 15 days.
Mandy Crawford is the Chief Information Officer for the State of Texas. She's a career lawyer who spent nearly 20 years within the Texas Attorney General's Office. On the day of the attack, she was scheduled to speak at a law conference about cybersecurity issues around midday. She had to cancel.
Mandy Crawford: “And as the morning progressed, I think around 8:30, we learned that eight governments have now been attacked by the same event. And so, throughout the morning, it increased by 10:30. We now knew there were 19. And at that point, that's when I reached out to the governor's office and described to them what was happening.”
There's a variety of factors that play into whether a cyber event can be declared a disaster. But what was happening was heading in that direction. One contributing factor was a SCADA system for the water utility that appeared to have been impacted. SCADA stands for Supervisory Control and Data Acquisition, and these types of systems link the physical valves and pumps with interfaces that are administered through software. Andy says there were two cities that early on reported possible SCADA implications due to ransomware.
Bennett: “They were unable to access their remote telemetry for their water systems. And so their gauging and flow and all that kind of stuff was now manual, which inherently puts them at greater risk.”
Ransomware and other kinds of attacks against critical infrastructure are worrying because of their potential to cause human harm either through interruption of service or malicious tampering. Later, Texas officials found that no harm came to pass, and both utilities were able to maintain service using their continuity plans. But at the time, those concerns were enough to make a call that public safety could potentially be at risk. Later on Friday, Governor Greg Abbott declared a “Level 2 Escalated Response,” which is just one level short of an emergency.
The declaration activated the State Operations Center. That's run by the Texas Department of Emergency Management and is part of the Texas A&M University System. It also meant the involvement of the Texas Military Department, which is the national and state guard. Private sector companies also jumped in to help. Rick says that StorageCraft, which provided backup services for most of the affected cities, helped enormously by offering quick technical support and helping to recover data. Overall, there was a huge marshalling of resources and services for the recovery effort. Here's Nancy:
Rainosek: “I will say that working in the State Operations Center, I've done a number of security incident responses over the years, and where you spend the night or multiple nights just working to respond, and you're eating your meals from a candy machine. And the fact that we had the tools that they use for floods and hurricanes and the communication tools, the tracking tools. They fed us. I probably ate better there than I do on a regular basis.”
It was a logistical challenge. Texas is big. And TSM Consulting had clients all over the state. One, in fact, was a 10 hour drive away. Andy says the city's needed help and support.
Bennett: “Sometimes, the most important aspect of incident response is showing up to help. It's being there, looking them in the eyes, and letting them know they don't have to go through this alone. They will not be overwhelmed. And we will be there until they are on their feet again. Now, it may be a while so they run, but they will be walking before we leave.”
Mandy says there's also pressure. Citizens expect their government to take care of them in crisis.
Crawford: “When governments are hit by some sort of cyber event, or any event when government services are offline. It's not like when you know your local grocery store or even a big chain store gets hit. And you can go to a different one, right? You can't pick a different government. It's like, when those services aren't there, they're not there. And you need them. And so, one, that's about the reliability and the availability of those services. But then, long term, you talk about reputational risk, right? Because there has to be that trust that a citizenry has in their government. And so that's one thing that we always try to keep in mind on these events and in planning and risk mitigation and then disaster recovery, and continuity, all of those things, those have to be top of mind is that Texans trust that their government's going to be there for them.”
Andy says the target was to restore a city's operations to a minimum viable floor. If 12 systems run, say water billing, three will be prioritized for restoration. That way, at least some business can be conducted. People won't be happy that there's not a higher level of service, but at least there's a path to normality.
Bennett: “Law enforcement systems were impacted, communication systems were impacted, billing systems were impacted. You don't think about all the things that are provided by a local government until they aren't working, until the computer that drives them is no longer working.”
Many of the cities did have decent backups. Andy says there's just a few vendors in the space that service governments and the software is fairly consistent. But he says there were underlying complexities.
Bennett: “But where they're not consistent on? Is it cloud-based? Is it on-premise? Is it up to date? Or is it five versions out? Because they haven't kept up with licensing? Are the backups cloud-based or are they local? Were there multiple copies kept offline like you're supposed to, or was everything just on a mirrored hard drive? And we saw the entire spectrum.”
I'd read that all the 23 cities were up at least partially running in just 10 days. Andy corrected me.
Bennett: “Eight days.”
Kirk: That's extraordinary. How did you do that?
Bennett: “Planning, man. Planning, planning, planning. There was a veritable army assigned to this incident. We had folks out in the field. A ton of folks out in the field. We had a ton of folks there in the Security Operations Center. We had analysts running from city to city, we had help desk set up overnight, taking probably hundreds, if not 1000s of calls. We used any resource we could get our hands on, and worked them into the plan. The plan worked because it was flexible. It wasn't a rigid plan, it allowed for a contingency. And it also allowed for the incorporation of new unknown resources. So when we wrote the plan, we knew that if the balloon ever truly went up, there would be a ton of things that we didn't know while planning. And so we planned for the known unknowns and the unknown unknowns. And I have trouble describing how you do that, except by having a whole bunch of really cynical, yet hopeful people in a room together, calling out every bad thing that could possibly happen. And then somebody else shouting and we could handle it with this.”
Media outreach was also a task that wasn't in the plan and done on the fly. Mandy did have quite a bit of experience in that area already. But she says it did require some improvisation.
Crawford: “It was a different side to it other than the technical piece that required that communication, relationship building, relationship management, media skills that were a whole level of challenge that we didn't necessarily have in our plan, and that we kind of did on the fly. And I'm also speaking of gratitude for this whole event. Gratitude to the impacted entities for the way they work together, and the way that they work with us, not just our partners who responded, but the entities who were impacted. It was really a great partnership.”
Nancy says the incident marked a high point for many who participated in it, including herself.
Rainosek: “And I've talked to some other people that were with the Texas Military Department that responded and that they're like, this is a highlight of my career, I would say it's a highlight of my career to be able to take part in a response event that was so successful, and there's a lot of pride in the pre-work that went into it and hopefully the post-work will make us even more successful, should this happen again.”
Kirk: “Hey, was it the day the Russians messed with Texas?”
Crawford: “That's right. They didn't get the memo about Don't Mess With Texas, right?”
Kirk: “You know, I asked Andy, is it too cliché to use the phrase Don't Mess With Texas in the podcast? And he said absolutely not.”
Crawford: “Yeah. Here in the Republic, it's a source of pride.”
Texas was ready for this. But the state also got quite lucky. And to find out why we're going to dig into the forensics of this. How did the attackers get into TSM Consulting Systems?
As I said before, TSM Consulting used ScreenConnect to connect to its clients' systems. A technician would log into the on-premises version of ScreenConnect. The technician would insert a Yubikey into the computer and enter the time-sensitive code that's generated by the device. A Yubikey is known as a hardware security key. So not only do you have to plug it into the computer, but you also need the code that it displays. This type of procedure is known as multi-factor authentication, and it's intended to thwart account takeovers. So if attackers do happen to capture a username and a password through phishing attempts or other social engineering schemes, they still won't have time-sensitive code and won't be able to log into the system. After the Yubikey's code was entered, the technician could access the systems of, say a law enforcement agency, for example. But not all of TSM's clients required multi-factor authentication, and it wasn't implemented amongst all of the cities. Rick says that now his company uses a much stricter procedure called virtual escorting for accessing client systems where the client has to approve the access.
The attackers first touched TSM Consulting Systems on August 2. Two weeks prior to the launch of the attack, they reached TSM's ScreenConnect server remotely from a Tor exit node. So Tor is short for The Onion Router. It's a system that uses a special browser and network to route your encrypted browsing traffic through nodes throughout the world. Using Tor anonymizes your traffic in a way that's very difficult but not impossible to trace. So how did they get to that ScreenConnect server in the first place? It appears they acquired the account credentials sometime prior to the attack and then used them. The alternative theory is that they exploited vulnerabilities in the ScreenConnect software itself. But that theory has never been born out in evidence. Rick suspects a technician's credentials were captured.
Myers: “My guess of what happened, and this is all a guess on my part, we had taken on several clients that year, till one client, specifically, the machines were full of keyloggers. All that malware. Just all kinds of stuff on it. My guess is my technician logged into one of those machines using ScreenConnect with the keylogger on it, and the keylogger then captured his credentials and also the IP address of where he was coming from. And that was the start of the process.”
The attack kicked off at 1:53 a.m. On August 16, 2019. They tested a PowerShell script on the ScreenConnect server and downloaded a second PowerShell script from Pastebin that was inserted into process memory. They were aiming to have the ransomware only run in memory to avoid triggering antivirus or other security software. But then something went wrong. The PowerShell script was buggy. Here's Andy explaining:
Bennett: “They were trying to do an elegantly scripted attack using PowerShell and more sophisticated systems administration to fully bomb everything that they could reach. And best we can tell it was either a typo or a bad script that they didn't take the time to fix. And they, when it didn't work properly, waited for the timeout to quit. They panicked and they used the MSPs tooling that they had access to.”
So there were a lot more systems and computers that TSM could access that could have been impacted. But after the attackers pivoted to using TSM's own tooling to distribute the ransomware, they messed up again. Using ScreenConnect, they ran the attack on 23 systems, one representing each city. But then, they attacked the 24th system, which was actually the ScreenConnect server they were using to push out the ransomware. Andy explains what effect that had:
Bennett: “They nuked their command and control. And that's the reason it wasn't worse.”
You heard that right. REvil stuffed up their own attack by encrypting the very server that was enabling the attack in the first place. It was a lucky break for Texas. But there still was a lot of damage. Rick says, however, that only computers running Windows 7 were affected.
Myers: “We had been working for two years to get our clients converted over to Windows 10. And in most all of our life, our law enforcement customers had moved. We had a couple that still had a few machines and we're still on Windows 7. But we had a whole lot of city employees that didn't want to make the move to Windows 10. It was like pulling teeth trying to get them to convert, and those are the ones that hit the hardest.”
There also were a variety of antivirus programs running on the Windows machines. But the one that consistently stopped the ransomware from running was Windows Defender.
Texas A&M University helped with forensics. Nancy said one computer in particular proved useful in understanding the technical intricacies of the attack. The computer had been disconnected from the network, but it hadn't been turned off, which meant that whatever code the attackers had deployed, bits of it, were still running the memory.
Rainosek: “We had a machine that had been disconnected from the internet, but had not been turned off. We had a real shot at giving the Justice Department evidence or them getting evidence from this machine through the memory that hadn't been erased because the machine hadn't been turned off. So this was one event where we really felt like we had a shot at them, and actually finding out who did it.”
Danny Miller is Chief Information Security Officer for the Texas A&M University System. The university helped with forensics. Danny says that valuable computer Nancy just referred to was running in a closet in one of the small cities. They sent out a state trooper with a USB stick to plug into the machine. That USB stick had Elastic's Endgame XDR software on it. XDR is short for extended detection and response and its software that's used for incident response detection and forensics. The machine was put into an isolated environment, and Endgame could observe how the ransomware worked. Here's Danny:
Danny Miller: “We got a trooper to take a stick drive. With the Endgame agent on it, we said, tell him to go out there, plug it in, and then let it install itself. So our CRT team is the one that did that. And they were able to get it installed, kind of put it in a stasis mode, and then connect to it, get the machine connect to the network, and then in an isolated environment, allow our CRT team to connect remotely from A&M. And then they were able to see it live. And that was kind of like, one of the first like, whoa, this is what's happened. And this is how it works. Oh, look at that, the PowerShell script crashed. Oh, look at this. It's got this signature, and so forth. And so they were actually able to even remotely take forensic images, and then copy those into an A&M-owned asset that we had an agreement with FBI, that actually let us let our folks copy it to our controlled storage facility, and then the FBI had the only other access to it, so they could do their analysis as well. That was really cool. We really felt like that was the beginning of a very close relationship with law enforcement intelligence agencies.”
He says that helped narrow it down, that it was, in fact, the REvil ransomware.
Miller: “There's lots of different ransomware variants out there. At first, some of the teams were saying it was Petya or NotPetya, or whatever it was. And I said, stop. Do not broadcast that. That is a terrible mistake, because we don't know for sure. Nobody telling, just wait until we have confirmation. And then we'll tell our entities out there what we think it is.”
Many of the computers were simply bricked due to the attack and had to be totally replaced. But Mandy says one computer is rumored to have met a different fate after being infected.
Crawford: “And thank goodness that there were machines that were there and unplugged. I will say this, remember, this is Texas, and we do things differently. And one of the entities that was hit on one of the calls when we were doing an assessment of asking the folks to run down equipment that was hit, what state it was and things like that. And one of the sheriffs in one of the town said he had taken care of the machine. I said, well, what does that mean? He said, well, we took care of it. Oh, what does that mean? Well, I took it out back and I shot it.
Kirk: “Oh, is that true?”
Crawford: “I don't know if it's true, or if he was just saying it for the phone call but we got a good chuckle and we said, okay, well I guess the threat’s been minimized.”
When you think of Texas, you think of ruggedly independent people who aren't pushovers. You think of people who can persevere in what can be an unforgiving environment. When Mandy joked that it was a republic, it really was. From 1836 to 1846, it was indeed the Republic of Texas. It wasn't part of the United States yet. And it was just kind of a roguish territory whose independence was disputed by Mexico, out there on its own, not to be messed with. So to pose the question: Did the State ever consider paying the $2.5 million to cybercriminals to get the decryption key? The decision was made long before the attack. Here's Mandy and Nancy:
Crawford: “We're firm believers in not paying and investing more of the money in defenses, responding, mitigation strategies, rather than paying the ransom so that you break that business model. What do you think, Nancy?”
Rainosek: “Yeah, I agree. And I think that when you don't make it profitable for them to continue doing this, that's a big way to slow it down. Now, if people spend the money on remediating and shoring up their defenses, and not on allowing these criminals to profit that, and use that money then to build more tools to attack us with, then we've all got to work together in a community to make that happen.”
But Andy says he sees how organizations can panic, but he tells him not to give in.
Bennett: “It's real easy to get into that panic. And there are organizations where they have given in the panic and it still hasn't helped. And you know what, I'll call them out because they went public. Colonial Pipeline, in particular, went ahead and paid that umpteen gazillion dollar ransom, I think was actually $4 million. And then it turns out, they were able to recover from the backups faster than they could decrypt.”
Andy says that Rick was on board from the start with the state's attitude towards paying ransoms.
Myers: “I feel for Rick. What happened to him could happen to literally any small business. It could happen to big businesses. Look at Kaseya. Throughout this entire process, during the incident and after the incident, Rick has conducted himself with a poise and grace that I don't think a whole lot of people could muster under similar circumstances. Despite the fact that it probably could have helped mitigate the damage to him and his business, he never once asked us to consider paying the ransom. That's not something that's ever been said. We've always let it be known that that was our stance. He did not break with that.”
I first called Rick last September, he answered the phone and we had a short chat. He was interested in telling his story, but he wasn't quite sure if it was the right time yet. Over the next few months, I'd email him links to new episodes of The Ransomware Files as they came out. By January, he was ready. He wanted to make sure his kind of experience with ransomware doesn't happen to anyone else.
Myers: “The trauma that thing is just incredible. You trying your best to see it's take care of your customers. Many of my customers have been with us over 20 years. They are more than just customers. They are people you know and care about. And to lose their confidence was a troubling thing. And that takes an emotional toll on you.”
Kirk: “Absolutely, absolutely. Can I ask how old you are?”
Myers: “In a few days, I'll be 74.”
Then in 2020, another extreme event: the COVID-19 pandemic. Rick says that hiring employees in order to rebuild became nearly impossible.
Myers: “While we lost business, we lost customers because of it. And anytime you lose the customers' data, you stand a good chance of losing their business. Again, many of the customers we lost, or some of the customers, I'm not seeing many, some of the customers we lost were all-time customers that have been with us for decades, literally decades. And a loss of trust, as I said, this bear a toll. And I've always felt like you can recover from anything. In real, our plans were never to recover from this. And I figured by March, April 2020, I could start hiring and start rebuilding the company. Unfortunately, about that same time was when COVID hit, and again, I told you my age, we had to be careful while bringing people in to try to do interviews. So hiring became impossible. We couldn't hire anybody. We couldn't find anybody to hire first of all, and then it was uncomfortable bringing people into our office to interview on a face to face basis.”
Kirk: “Yes.”
Myers: “So we just decided that I was going to take the business on my own and run it myself as I did when I first started the business years earlier. As a result, we decided we no longer needed this expensive, nice office. So we closed the office and moved the business back to where we started from in my house. In the course of all that, I do feel I have aged just significantly over the last few years. Between the ransomware attack and COVID, between the two of them, it's taken a toll on me that I don't know that I can recover from.”
Kirk: “I'm so sorry, Rick, I'm so sorry.”
Myers: “We're still in business. We still do business. I still love my customers. I still love what I do and as long as I'm able to, I'll keep it up. But we'll never get to the point we were before. It just won't happen. I say that but no one knows what tomorrow is going to bring.”
In November 2021, something remarkable happened. The U.S. Justice Department announced they charged Yevgeniy Polyanin, a Russian national, with executing the attack against Texas. Also that day, they announced charges against a 22-year-old Ukrainian, Yaroslav Vasinskyi, with the attack against Kaseya. Both men were accused of using the REvil ransomware. As a bonus, prosecutor said they had also seized 6.1 million in cryptocurrency from a company called FTX Trading Ltd. That money was allegedly ransom payments paid to Polyanin. The Department of Information Resources in Texas put out its own press release heralding the indictments the same day as the Justice Department. Mandy says the DIR was thrilled.
Crawford: “What we had been saying all along because we worked closely with our own Department of Public Safety and the FBI throughout the event and there were stuff that we just couldn't share that throughout the process and kind of our standing line is because we hope they catch the bad guys. That's why we want to keep this because maybe they'll catch the bad guy.”
Kirk: “Yes.”
Crawford: “When they actually did, at least name the bad guy that was really gratifying. Because so often these things you don't know, or you do know who did it but you can't put a name or that there's not even a sort of possibility of some sort of justice. So it was good deal.”
Not long after the indictment was announced, the UK newspaper, The Daily Mail, tracked Polyanin down at his home in Barnaul, Russia. He wouldn't speak to the publication, but the newspaper did manage to speak with his mother. Polyanin's mother said she'd been in touch with their son, the charges were fake, and they were all doing fine. The U.S. and Russia do not have an extradition treaty, so there's little chance of Polyanin ever facing trial unless he takes a risky vacation outside Russia.
I wanted to contact Polyanin. I wanted to see what he thought about the accusations levied against him. I wanted to ask him if he knew about ransomware and the destruction that this cybercrime wave has wrought around the world. I wanted to see what he thought of Rick's story. But a trip to Siberia and then winter from Australia wasn't on the cards for me for a lot of reasons. And I'll give it to The Daily Mail for their tenacity and tracking anyone down anywhere in the world, which I doubt I could match, but I had a source who might be able to help and perhaps the best way to reach Polyanin was simply online anyway.
The source is a cybercrime researcher who's been tracking the nicknames and aliases that Polyanin has used since 2013. Polyanin's accounts had been quiet for the last couple of years. My contact sent a message to one of Polyanin's Jabber accounts. After a day or so, there was no response. My contact also tried one of Polyanin's friends and did reach him, but he didn't want to talk. No luck. It's unclear where Polyanin may even be now. In mid-January 2022, Russia's FSB security agency said it arrested 14 men allegedly affiliated with the REvil gang. Some of the names of those arrested were released, but Polyanin's name wasn't among them. Rick says he thinks that ransomware criminals have no clue about how their actions affect other people. He says that even if he did have the opportunity to say something to Polyanin, he wouldn't say anything. He says he'd just ignore him. That's Rick, moving on, looking ahead, even in the dusk of his career.
Kirk: “You sound like what you've just told me, which is very, very difficult and sad. You sound like a very resilient person though.”
Myers: “I guess then this goes to faith. Yeah, I'm very active in my church. Matter of fact, I'm a deacon in my church. A strong blood and a strong belief in God. And I know that I'm going to get taken care of either way. In that regard, I do feel there are worse things that could have happened to me. And I do think there's better things to come.”
Kirk: “That's wonderful to hear.”
This episode is the first of two parts so keep an eye out for the second one, which will cover the ransomware attack against Kaseya. This episode of The Ransomware Files was written, researched, edited and produced by me, Jeremy Kirk. The production coordinator is Rashmi Ramesh. The Ransomware Files theme song is by Chris Gilbert of Ordinary Weirdos Music. If you enjoy The Ransomware Files, please share it on your social media platform of choice. Also, the series has its own Twitter handle @ransomwarefiles, which tweets news and happenings about ransomware. And I'm on Twitter @jeremy_kirk. If you'd like to participate in this project or have an idea for it, please get in touch with me. My direct messages are open on Twitter and I'm easy to find on LinkedIn. I'm looking for other people, organizations and companies who are willing to share their unique experiences for the benefit of all, until ransomware, hopefully, becomes a thing of the past.