This week, Prasanna and Mr. Backup (W. Curtis Preston) review a series of posts made by Snorkel42, who previously appeared on this podcast in the episode called “Security expert rips Okta for their response to hack.” Things were recorded out of order, so this is the episode where we discovered him on Reddit, and tried our best to distill several thousand words into about 30 mins of advice on how to protect against ransomware. We talk about how to prevent getting it in the first place, how to limit its damage if you do get it, and how to respond and restore your data once that happens. There is a ton of really good advice here, so check it out!
Here are the three posts:
Listen to it here, watch the video, or read the transcript below.
[00:00:00] Prasanna Malaiyandi: What’s a TLD for our listeners?
[00:00:02] curtis: Oh, top level domain. That’s like.com or dot ransomware.
[00:00:13] curtis: Hi and welcome to Backup Central’s Restore it All podcast. I’m your host, W. Curtis Preston, AKA Mr. Backup. And I have with me, my delayed shipment consultant, Prasanna Malaiyandi. How’s it going , Prasanna?
[00:00:25] Prasanna Malaiyandi: I’m good. Curtis, wait, what’s delayed.
[00:00:29] curtis: my, my, my flooring shipment, you know,
and I, I turn to you for.
[00:00:33] Prasanna Malaiyandi: what I thought you received one.
[00:00:38] curtis: I did I did, but . I ordered a big shipment of flooring, and then I ordered a much smaller shipment and I did that in two shipments because I couldn’t order all of it at once. And then I had to order like another 10% and the second shipment I received the second shipment like three weeks ago, I still haven’t received the first shipment.
And, um, I just turned to you for, for, you know, emotional support in this time of. I’m not doing anything until the entire shipment comes in., it’s just ridiculous. I ordered this,
[00:01:10] Prasanna Malaiyandi: Have you heard about supply chain issues? Curtis has this not.
[00:01:14] curtis: I gave them grace because of the supply chain, but here’s the thing. This is made right up the road from me. Well, it’s more like up the road from you, but it’s made in California. It’s vinyl. The manufacturing is happening in California.
But the problem is that they’ve lied to me. They lied to me before. They told me it’s in production because you know, they make several colors. They’re like, oh, that color, it was really in demand. It’s in production. Now. They told me that like three weeks ago, they said it’s in production. It should ship out any day now. They’re now claiming they’re out of stock.
[00:01:51] Prasanna Malaiyandi: Oh,
[00:01:53] curtis: Right. They’re like, oh yeah, we, we, we did it was in production. We didn’t lie to you. We just didn’t make enough. Well, why did you stop the production run before you made enough to fulfill back orders?
I mean, I get that. You’re behind. I get that you had a big promotion, but retooling, the production line is a pain. Right. So why would you retool it
[00:02:16] Prasanna Malaiyandi: Maybe they ran out of
[00:02:17] curtis: of color.
[00:02:18] Prasanna Malaiyandi: Yeah. Whatever.
[00:02:20] curtis: So this is why you’re here. You’re here to make me not so angry. That’s why I said you’re my delayed shipment consultant. All I know is it’s not in my hot little hands and I’m not doing squat in my garage until I get the entire shipment.
[00:02:36] Prasanna Malaiyandi: Just think though. How about delayed gratification? Once you finally get the pallets
[00:02:43] curtis: This is the ultimate in delayed gratification. I’ve never had so much trouble spending money in my life. Right. I mean, and that even includes the two recent, very expensive couches that we bought. There were way more expensive than this. Um, we ordered it and then they were like, it’s in a ship off long beach. If you want to see your couches go to the long beach Harbor and look out into the water and you can see, and that was, that was promised like four weeks. And it was more like eight, but at least there, I was like, well, I’m part of the whole, you know, shipment problem. And I just had to wait, but here it’s just frustrating because they, because they’ve miscommunicated,
[00:03:23] Prasanna Malaiyandi: Yeah. I think that’s the problem, right? If they had not given you any information that yeah. It’s in production, right. You probably would have been fine. Yeah. It’s just shipping delays. That’s fine. The fact that they told you now you’re annoyed.
[00:03:35] curtis: Hashtag
[00:03:35] Prasanna Malaiyandi: it’ll be
[00:03:36] curtis: #firstworldproblems.
[00:03:38] Prasanna Malaiyandi: Take a deep breath.
[00:03:43] curtis: Yeah, good times. Good times. Um,
Our disclaimer, Prasanna works for Zoom. I work for Druva and, uh, the opinions that you hear are ours. This is not a podcast of either company. And a rate us at ratethispodcast.com/restore, or just click on your favorite pod catcher. And, uh, click down to the bottom and give us some stars, or maybe even a comment. Talk about how much you love Prasanna’s beard. I’m good with that. And how it’s so much longer and darker than mine and. And, uh, you know, if you’re, if you’re curious about such things, if any of these things, we talk about excite you either way then, uh, you know, @wcpreston on Twitter or wcurtispreston@gmail and, uh, you’ll find me.
So I see. I sent you this, this post that I, that I saw on Reddit, which it’s well, it’s actually a series of three posts from a Reddit user called snorkel42. Don’t let his, you know, snorkeling ID fool you the, the person knows what they’re talking about. I don’t know. I don’t know anything about this person.
Other than that, they, they have, they post regularly in a subreddit called security cadence. Um, but he also posted he or she, I don’t know if I mistaken mistakenly called the person. He, I apologize in advance for my misogeny, so. The, it was about ransomware and, and they are a specialist in the areas of security and many people had asked them to post stuff about ransomware and they had continually sort of said, I don’t want to post about ransomware. And can you imagine why that would be
[00:05:58] Prasanna Malaiyandi: You’re just sort of propagate well, it’s ransomware you get hit with, because there were a bunch of gaps before ransomware got hit and it’s better to address the problem rather than trying to address sort of the outcome.
[00:06:13] curtis: Yeah. So ransomware to this person is the symptom of a whole lot of bad things that you were already doing or not doing. And they’ve spent their career helping to make sure you do those things. But with the, I think two things, one is that obviously the ransomware attacks are getting to a fever pitch and then two.
There is what we talked about on the previous episode, which was this concern about Russia and D w we did cover that. Didn’t
[00:06:51] Prasanna Malaiyandi: Yeah, we cover the Conti ransomware gang
[00:06:54] curtis: Yeah. Yeah.
Um, yeah, the, the, the Krebs on security post.
[00:06:58] Prasanna Malaiyandi: Yep.
[00:07:00] curtis: That the concern is that the level of the fever pitch that we’re experiencing might actually go through the roof. And so they said, Hey, I’m gonna finally, I’m fine, fine.
I’ll post about ransomware, but even in their post about ransomware, it really wasn’t that much about ransomware as much as it was about the things. Well, no, that’s not true. I’ll take that back. It was, it was here is the way ransomware works. And so I I’d say the first one, I’d say of the three series,
The first one was about here’s how to prevent it. Number one, like from getting in. The second was here’s how to prevent it from doing more damage once it’s in. And then the third one, it was okay. All right. You’re totally screwed. You’ve got to reach for your backups.
[00:07:47] Prasanna Malaiyandi: Yeah.
[00:07:47] curtis: that
[00:07:49] Prasanna Malaiyandi: The one thing I would add to that , is he also was careful saying, I don’t want to just focus on the Conti ransomware and provide you steps to prevent that because there are so many other ransomware flavors out there. If you build something for just one.
You’re not going to be protecting yourself. Let’s take a holistic approach. And like you said, let’s cover, how do you prevent it from getting in? What, how do you prevent the spread of it? And then how do you recover?
[00:08:15] curtis: Yeah. Good point.
[00:08:16] Prasanna Malaiyandi: The first one is called initial breach, I think is how he titled the first article.
[00:08:22] curtis: Right. So the phishing basically, they’re saying That That is the number one way that you get ransomware.
[00:08:33] Prasanna Malaiyandi: Yep. Someone accidentally clicking an email, opening up something, letting the attackers in, and they don’t even know about it. So how do you prevent your users from clicking on malicious links?
[00:08:45] curtis: now, now, it’s interesting. This goes, yeah. Sorry. This goes somewhat against what, some of the advice of one of the guests that we had on the podcast, which was, they basically said, look, your people are going to click on stuff, stop relying on, you know, I dunno. I dunno if it’s against, but, but he, he, de-prioritized training and, and like, uh, phishing assessments, didn’t you think.
[00:09:19] Prasanna Malaiyandi: Yeah. So. This author does say training can only help you so much? I think the couple things, the couple things though, that he did mention is, um, you do need some level of training, but you need to make sure people don’t feel like they’re being punished. When they do the wrong thing, right? You want that transparency. You want to be telling people it’s okay for you to say that I clicked the wrong thing because then the IT team can try to evaluate what’s going on and try to contain it. The sooner they know the better it is.
But if say someone’s afraid because they’re going to get in trouble. They might be fired, right. It becomes taboo then no one’s going to report it. And that’s actually really bad.
[00:10:04] curtis: Yeah. Um, they said to prioritize rewarding over punishment. Right? Make it, make it known. Like you said, that it’s okay to call in. We want you to call in, even if you messed up and, and then, and they also said consider doing your own phishing assessments. I read some of the comments and they talked about that.
They had a thing where you, you, you got some. You got some, it was sort of some strikes and it was like 10 strikes. It was like, you could click on 10 malicious emails. And, and then it was the 10th. When, and that they actually had a series of escalations where, you know, it started out, Hey, you know, we really told you kind of thing.
Um, I think you can do both. I think you can do both carrot and stick, right. Reward and punishment where yes. You want to reward people for calling in. Thank you for calling accidentally clicked and then. And then if the person clicks doesn’t know, but you know, because you did a phishing assessment, you do a series of escalating things where that ultimately you can have a person.
And this was discussed in the comments, not necessarily that you would fire somebody that, that keeps doing this, but you might say, okay, this person cannot be trusted with a straight internet connection.
[00:11:26] Prasanna Malaiyandi: Yup.
[00:11:27] curtis: Right. All email from this person will be monitored. Yeah. They can only open email that’s straight from our Exchange server or whatever stuff like that.
[00:11:37] Prasanna Malaiyandi: So phishing was sort of one way that people get in. Right. But I think once they’re in whichever mechanism it is, it’s like, okay, how do you detect that someone’s in? And I think Curtis, this is what you’re going to say, right. About sort of this notion of droppers.
[00:11:52] curtis: Yeah, I actually didn’t know this part. That’s I was fascinated that that basically that the actual phishing results in a very small piece of software whose job it is to install the actual piece of software
[00:12:08] Prasanna Malaiyandi: Yeah.
[00:12:09] curtis: and that he calls out a dropper.
[00:12:11] Prasanna Malaiyandi: Yep.
[00:12:12] curtis: Well, and so the idea is understand that that’s the way it works, that a piece of code gets dropped in, and then that piece of code executes, and the only purpose of that piece of code is to download the other piece of code. And so they said that you could, you could stop that. You could say, well, you can’t run arbitrary pieces of code in, in locations that are directly accessible by the end user,
[00:12:42] Prasanna Malaiyandi: Or you could restrict what applications are allowed to run on a laptop for instance,
[00:12:46] curtis: yes,
Whitelisting, I think whitelisting is it, I think it’s the, the best. The best way to stop stuff like this. It’s also the highest touch because it means that every new application that anybody has to install, they have to get approval.
[00:13:03] Prasanna Malaiyandi: Yep. think it’s a way to guarantee sort of legitimate applications have gone through some sort of validation process, security review, et cetera, before it’s being allowed to be deployed in your environment
[00:13:17] curtis: Right. And then the next thing it talked about was that a random file running should not be downloading files from the internet, right. That it should only be HTTP and HTTPS is downloading from the internet. And so. He said with exceptions, like, you know, um, uh, SFTP for example. So he talked about, he talked about, you know, again, accessing that also possibly blocking bizarre TLDs right. And unnecessary locations. You could just simply say, listen, uh, we don’t have anything to do with Russia. Why would we download anything from Russia?
And if there is somebody in our company that needs to download stuff from Russia, they will be, they will be accepted. That was a very running theme I heard was lock down everything and allow exceptions.
[00:14:08] Prasanna Malaiyandi: Yeah. And, uh, it was going to bring up two things. One was what’s a TLD for our listeners?
[00:14:15] curtis: Oh, top level domain. That’s like.com or dot ransomware.
There is no dot
[00:14:24] Prasanna Malaiyandi: And was it you, or was it one of our guests who were, who was talking about how they worked at a company that completely locked down their network and the network admin would never let them do their backups
and everything was by except.
[00:14:39] curtis: That was me. Yeah. Yeah. Uh, that was, I was a client of mine where they had internal firewalls and that’s an example of, you know, going to the extreme of, well, now you’re now you’re preventing core business functions,
[00:14:54] Prasanna Malaiyandi: Yeah,
[00:14:55] curtis: right?
[00:14:56] Prasanna Malaiyandi: but
[00:14:56] curtis: they also talked about local firewalls, right. Which is what we were just talking about, that the, and we’re going to get to that more in the next section is, so they’re just looking, he’s looking
for ways to stop the dropper from getting yeah, exactly.
[00:15:16] Prasanna Malaiyandi: Yeah. thought was an interesting point I’d never thought about is he does have a point about they block newly created domains. Right. Which I thought that had been dormant for a while and then are now active, which I thought was very interesting because it’s something I had never thought about, but it totally makes sense.
Usually when you get ransomware, right. These actors, they spin up domains and they start communicating, using that domain. So he’s like, yeah, you could have a policy to just block these domains. So they can’t actually reach back out to the command and control servers to be able to download from the dropper, the actual exploit.
[00:15:53] curtis: Right. And, and they said they weren’t aware of anything. Where that you can do this for free, but there are tools that are available to help you do This right. There’s
[00:16:05] Prasanna Malaiyandi: remember, uh, what are the D D. Uh, what were the initials?
[00:16:12] curtis: DDI.
[00:16:14] Prasanna Malaiyandi: right. And I think that goes to some of that as well. Right. Where it’s like, Hey, if you have some of those controls in place, you can now prevent unauthorized access to domains. They should not be having access to.
[00:16:27] curtis: Exactly. And then they started talking about preventing lateral movement inside. Think about the ways that people need to move within your organization and allow that, but block all other movement, right. Lateral movement between servers and I, and I think, again, going back to that company, that was a perfect example of, they had blocked all lateral movement between all servers and I couldn’t get my job done.
They’re only problem w and they should have done that. And, you know, they were forward thinking in that regard, but you do need to allow exceptions for things like backup, right. That is definitely a server to server lateral movement.
[00:17:11] Prasanna Malaiyandi: Yeah. And it’s also other simple things. Like one of them was your favorite topic, right? Locking down RDP and SSH. Right. If it’s not needed, then lock it down. Right. SMB is the same way as well for vCenter, right? Figuring out what actually needs access and what. Needs to be available to the internet. And one of the points he made is you should just assume that your inner internal network is as hostile as internet access. Right? So once an exploit happens, you can’t trust anything internally.
[00:17:44] curtis: They were also, I, you know, I didn’t necessarily agree with this one here. And that was it’s time to kill monolithic file servers. Right. Now I don’t have a problem with the file server. It’s just, I think when, when they mean monolithic file server, they’re just saying a file server where everybody in the company can access all the data. I would agree there anybody that’s doing that, you know, in a
[00:18:07] Prasanna Malaiyandi: Yeah. Segregate the data isolate to departments that need access. You use ACLs, make sure the people who need access have access and then monitor who’s accessing what.
[00:18:18] curtis: So they made a specific example of like, you know, just because just because accounts receivable gets attacked, something shouldn’t happen to payroll. These are, these are both finance functions, but they’re separate financial functions and they should have their own areas. Uh, and this is another one that I harp on is about protecting privileged credentials. And
[00:18:42] Prasanna Malaiyandi: don’t just have your password tattooed on your forehead, Curtis.
[00:18:47] curtis: They recommended implementing, uh, things like LAPS, which I had to look up, which stands for local administrator password solution.
[00:18:56] Prasanna Malaiyandi: Uh, setting a different random password for the common local admin account on every computer in the domain. So you don’t use one password for everything.
[00:19:07] curtis: And then MFA, I think, I think every system, you know, every, every privileged account needs to have MFA and, you know, I’m sorry, that’s a pain. I, you know, I use it all the time, but it what is
[00:19:24] Prasanna Malaiyandi: but wait, why do you need a privileged account? You should. Here’s the thing. Most times you should probably not need privileged accounts, so you do not need to access your privileged accounts.
[00:19:36] curtis: Agreed, but, but they have to exist. And so you have to lock them down this way. I think what you’re saying is MFA, shouldn’t be that big of a deal for you. If you set up modern administration.
[00:19:48] Prasanna Malaiyandi: yeah. And you should rarely be using that.
[00:19:52] curtis: Right. Right. And then very last on the list and I would have put it first, but you know, it’s just me and that was patching your stuff.
[00:20:00] Prasanna Malaiyandi: How many times does that come up on the podcast? When we talk about ransomware, you know,
[00:20:05] curtis: Yeah, exactly. So the next one is about. It’s like, okay, so you got some ransomware. Let’s talk about the things that they’re going to try to do. The very first thing they listed was deleting of shadow copies. And so I, and really shadow copies are basically like he’s talking about windows shadow copies.
[00:20:31] Prasanna Malaiyandi: Yeah, I think windows shadow copies. Yup.
[00:20:34] curtis: Right. And so there is a tool here, which I had never heard of called raccine. And it, it stops you from deleting shadow copies. He said it stops everybody from deleting them. So just realize that if you’ve got some regular thing that regularly deletes shadow copies, it’ll break that, but it looks it’s something on github. So it’s, uh, you know, it’s an open source tool.
[00:21:03] Prasanna Malaiyandi: And just reading that briefly, I think many backup tools when you’re backing up windows applications uses shadow copy. So be careful if you are using that because you may not be able to do your backups.
[00:21:16] curtis: Yeah, that’s a good question. I, I guess, you know, I would differentiate between shadow copies made just for the purposes of backups and shadow copies that are made and then left there. I don’t know if there’s like a different. I know that when you make a snapshot, you say why you’re making the snapshot.
[00:21:34] Prasanna Malaiyandi: Yeah.
[00:21:35] curtis: Um, but agreed that this is not something that you’re just going to download and just implement,
[00:21:43] Prasanna Malaiyandi: Yeah.
[00:21:44] curtis: might break all your backups. Well, what it might do is it might allow you to create that snapshot, but then it leaves all those snapshots around and let you delete them. and you might get an error on your backup because you can’t, it can’t delete the snapshot.
[00:21:55] Prasanna Malaiyandi: yeah. Or your production could run out of space and then your app dies.
[00:22:01] curtis: And then what’s the next one
[00:22:03] Prasanna Malaiyandi: So the next one is a common theme for us. Uh, when we talk about ransomware, it’s less about the actual encrypting of data. It’s the fact that these ransomware actors, especially the Conti group, they like to exfiltrate your data and steal sensitive data, and then hold you hostage and be like, Hey, you want to pay?
Then you have to pay twice once for the decryption key. And then once to make sure we don’t publish your data. And then sometimes they will still go and publish your data.
Right. So in this post, he talks about sort of, how can you make sure you can detect data exfiltration? And he talks about everything from, if you have, if you understand network patterns, you could look for anomalies.
Um, you can also look at other tools. To see when data is actually being read and sent. So there’s some interesting tools that he talked about. One that I never thought about, which was this mechanism called, uh, from things called Canary tokens,
where it basically creates a false file. And any time someone accesses it, it generates a token and sends it home. And then it’ll send you an email, say, Hey, by the way, someone accessed this file. So you can sort of get notified of, Hey, someone’s accessing something, which they probably normally never should be. Because most of this ransomware software and data exfiltration, it’s just programmatically reading, like scanning folders, reading files, right. Trying to figure out what to send.
[00:23:38] curtis: Right. And they mentioned both commercial solutions and open source solutions. Like the one you mentioned, they also mentioned something called, uh, Zeke, which, uh, And you know, that it analyzes NetFlow, but there are commercial tools, which we’ve mentioned on here. Um, and I, and I’d like to get, I’d like to get more of those companies on here.
And their recommendation was the same as mine, which is looking for something that uses behavioral analytics to determine what is, and is not a normal file transfer, right. That should be able to spot a massive, uh, exfiltration attack.. And then the response against encryption, they talked about the EDR XDR, which is I had to look that up. I was not in my, so this is what,
[00:24:32] Prasanna Malaiyandi: And point D
endpoint detection and response.
[00:24:36] curtis: right. Okay. So. The idea is that if you’ve got, if you’ve got the money to put something on each laptop that basically looks at and stops, massive file modifications, it would detect and stop those. Right. And then same thing with the, with the honeypot. I liked the idea with the creating an entire separate file server that has, has all the same file names, but just with junk data, watch for anybody doing anything there and then report on.
[00:25:07] Prasanna Malaiyandi: Yeah. And the interesting thing is when he was talking about honeypots, I didn’t know, this is, he was like, oh yeah. And then to make it more realistic, you, there are a couple things you can do. You can map those device shares to actual endpoint devices. So they show up there because if I’m a ransomware program and I’m just looking at all the devices attached, right.
I don’t know if it’s real or not. And the question came up, Hey, how do you hide it from your end users? Because you don’t want your end users clicking on it as well. And there are registry commands in Windows, so you can actually hide them. So your users don’t actually see those drives.
And instead he suggested you actually bookmarked. Shared drive letters with these honeypot shared drives because ransomware, uh, programs are either going to start from a and work alphabetically or start from Z and come backwards, to see what drives are available. And then they’ll just start looking that way.
[00:26:03] curtis: So, so put a honeypot at a and put a honeypot at z.
[00:26:07] Prasanna Malaiyandi: Yup.
[00:26:09] curtis: I like um,
[00:26:12] Prasanna Malaiyandi: were some really interesting things that he talked about.
[00:26:15] curtis: And we can only cover a little bit here. I just would highly recommend anybody that’s interested in this, which should be everybody go read this thread. It’s really well-written thread
[00:26:28] Prasanna Malaiyandi: It’s like how to trick ransomware and how to protect yourself.
[00:26:31] curtis: Right.
[00:26:36] Prasanna Malaiyandi: jump onto the third? Curtis.
[00:26:38] curtis: Yeah. Get up on the third? one.
[00:26:39] Prasanna Malaiyandi: Sorry, what is the third one about by the way?
[00:26:42] curtis: Oh, the third one well, basically it’s like, well, you’ve been infected. What are we going to do? Worst case scenario you’ve been infected and it’s spread, and now you need to reach for your backups.
So they mentioned go to the, the, the incident response plan. And of course that assumes that you have one, which we’ve said that you need to have one, right? We we’ve mentioned repeatedly that a ransomware attack is not the same as a disaster. There are elements that I’d say a disaster is a subset of. Uh, typical DR response is a subset of a, of a ransomware attack response.
[00:27:22] Prasanna Malaiyandi: Think people get confused because in the end you’re trying to do the same things, right. Get your data up. But I think the steps and the number of people, the different types of people involved are significantly different between just a normal DR. Versus a ransomware recovery.
[00:27:39] curtis: Well, you know, simplistically to me, the biggest difference between, uh, responding to a ransomware attack and a disaster, it’d be the equivalent of like, if you’re doing a DR and you’ve had a flood step number one is drain the data center,
right? Get all the water out of the data center. Well, a ransomware attack is like, you’re trying to drain the data center while you have a person standing there with a fire hose, it’s filling up your datacenter.
Right? the, that’s the difference between a disaster recovery and a ransomware recovery is that they are actively still attacking you. And you’re actively experiencing the disaster at the same time as you’re trying to recover from it. And so they’ve got a good thing here on what should be in an incident response, right?
Some things you have to have in your incident response plan got eight things about right. Procedures and policies and an incident firm. Right. You, you need, you basically get professionals, retain them now, right? Oh, by the way, I just, I just gotta throw out a really hilarious thing from, uh, my granddaughter Lily yesterday.
So we have a friend, a mutual friend that was in a car accident a while back. Not, not seriously injured, but injured enough that there is a lawsuit that our, that, that that’s going on. And Lily said, uh, she, you know, she, she mentioned that I couldn’t, she couldn’t pick her up because, you know, she was with her, she was with her lawyer and then she looks at me, we were just walking and then she’s like, do I have a lawyer?
I was like, no, I don’t think you have a lawyer. You don’t need a lawyer right now.
[00:29:35] Prasanna Malaiyandi: But, but you’re right. Most people don’t even think about that. Like even in like everyday, like normal situations, it’s like, if I, God forbid get arrested, right. Who am I going to call? It’s like,
[00:29:46] curtis: Right. Yeah. And so w what they’re saying here is, you know, go, go find who you’re going to hire
[00:29:55] Prasanna Malaiyandi: who are you going to call Ghostbusters?
[00:29:57] curtis: going to call? And, um, you know, and they got a policy, oh, a policy. This is interesting policy for informing partners and customers and the media. Right? Decision-makers right. All of that stuff. This should all be decided upfront. You should be deciding that now. I don’t know how many times we can say that.
[00:30:17] Prasanna Malaiyandi: Yep. And then they talk about restoring your data.
[00:30:20] curtis: Restoring your data. right? And I think how they said alright, three posts in and we can finally talk about backups. Right. It’s interesting here. Right? And he talks about, you know, the typical call-out is that ransomware’s going to target your backups. And so you need some sort of immutable backup solution. Right.
[00:30:39] Prasanna Malaiyandi: Um, he does also talk and I know Curtis, you’re probably going to have concerns with this, right? That you don’t have to be offsite to protect your backups properly. He mentions that you could use strict network segmentation or other mechanisms to ensure separation, which would protect you in the case of ransomware, but may not protect you from all disasters that could occur.
[00:31:05] curtis: Agreed. And, and, and I don’t, I don’t have an issue with that, right. Obviously, you know, I’ll say obviously I work at a service-based backup company. And we see that as the easy it’s easy peasy. All our backups are off site. I I’m not against, you know, as a backup expert, I’m not against onsite backups. There’s a lot of good reasons for an onsite copy, but I completely agree with this person that you have to protect that onsite copy from attacks. And there are a lot of very common backup designs, incredibly common backup designs that do not that the default installation of those products do not protect you. Right. And I, and I’ll, you know, I don’t wanna, I don’t wanna pick on our friends at Veeam, but that’s a perfect example. The guys from Veeam came on here and they explained to you, if you listen, if you, if you haven’t seen those episodes, go back and listen to them. Uh, about, you know, when they talked about the, the Conti ransomware attacks and how you can configure your Veeam backups to protect against that. My concern is that most of their customers are not listening to this podcast, by the way, they’re more than welcome. All 700,000 Veeam customers are more than welcome to come listen to the podcast.
But if, if you just do the default installation and you don’t take their recommendations on how to further protect your data, you know, it’s no different than any of the other products, right? So
[00:32:36] Prasanna Malaiyandi: Read
[00:32:36] curtis: you’ve got to stop doing that. Read the manual, read the best practices. Call Rickatron. Rickatron’ll sort, you out and. So he talks about that. He also talks about testing, testing, your backups. I’m editing right now, like literally in I’m in the middle of editing the podcast, the episode of the restore test gone horribly wrong.
[00:32:59] Prasanna Malaiyandi: backup.
[00:33:00] curtis: It’s going to be a great episode. The. Yeah, Schrodinger’s backup. Exactly. That’s going to, if, yeah, if you haven’t heard that episode go back and listen to it. It’s a, it’s a
[00:33:12] Prasanna Malaiyandi: Yeah.
[00:33:14] curtis: uh,
[00:33:15] Prasanna Malaiyandi: article also refers to it, right? Yeah.
[00:33:18] curtis: yes, he does.
Uh, did he, did he actually refer to Shrodinger’s
[00:33:22] Prasanna Malaiyandi: Yeah. Yeah. HInging your company’s future on a schrodinger’s backup thought experiment is a terrible idea. Don’t do that.
[00:33:30] curtis: Nice. So, and then why don’t you talk about the decryption part?
[00:33:36] Prasanna Malaiyandi: Yeah. So I guess the final part right. Is you’ve been hit with encryption, right? So now what do you do? And in most cases, it’s. You can try to get, like, if you’re lucky, there might be a free decryptor out there for your data. It’s just going to take a very long time. Right. And if you do pay the ransom and you have to understand that paying the ransom may be illegal to some of these groups, right?
They’ll give you back a decryption key. Hopefully it’ll work. It’s not, it’s in the ransomware. Group’s best interest not to cheat you there, but you’re taking a risk there as well. And then finally, Once you’ve actually decrypted your data. You’ve gone back up and running. There’s nothing that prevents them from either coming back and attacking you again, if you haven’t fixed anything right.
Or the next group coming back. Cause that’s another common thing you see is one group gets in encrypts your data. Another group figures out a different mechanism because they know now that you’re willing to pay. And so they might come after you as well. So even once you have your data decrypted, it’s not the end of the story.
[00:34:46] curtis: Right. And then the there’s a, there’s a what’s next and, and, and all of these words, and this is a really long series of posts, which I highly recommend you go look through. There’s one part where they typed in all caps, and this is it right when you’re done, whatever you did restore, pay the ransom, whatever it is.
It’s not over, you clearly have a serious gap in your defenses. You need to find these and fix them. And then this is all caps and you need to understand that those gaps are bigger than just whatever the initial breach vector was as highlighted in parts one and two of this series, there are several opportunities to stop a ransomware breach before it gets to this point. So, um, there, there was some other. It was another one that I read, uh, somebody, they said, well, if I, if I, if I was at a company that had a highly, I think it was actually in here. If I was at a company that a highly publicized breach does this hurt my chances of getting a job and the author of this article didn’t think so, because they basically said you now have experience
[00:35:51] Prasanna Malaiyandi: Yep. I think it was actually at the end of this article is where he wrote about that.
Yeah. He’s like, yeah. It’s something you should actually show that you’ve gone through this because for a lot of people it’s just theoretical. They’ve never experienced it. It’s like you Curtis. Right. I can sit here and talk about like how to back up your data, how to restore your data, ideally how it should be done.
Right. But I’ve never cut my teeth in a production environment, trying to do a restore with people, yelling at me over my shoulder or watching over my shoulder. Right. You have, and I think that’s sort of the difference, right? Is you have that experience because trial by fire.
[00:36:28] curtis: Yeah, I, you know, you just, you just reminded me of, and I know I’ve told this story before, but not everybody’s listening to every episode. My, one of my favorite restore stories was, was back at my first big job. And we had somebody in the NOC that was coordinating the various things that were happening of this big restore.
And we had another guy that was in the data center that was actually doing things and. He was talking to the person who was on the phone in the NOC. And he, he didn’t know that he was on speaker. And so he said, he’s like, oh, so you know where you are. I’m in the NOC. He goes, oh, so I suppose you have Tom and Tom standing on your left and right shoulder.
And he was referring to our boss’s boss and our boss’s boss’s boss. Right. And, um, the, uh, that would be Tom Thomaides and Tom Lackey. And they were indeed standing both on his left and right shoulder. And they said that when he said that, oh, so you have Tom and Tom standing on your left and right shoulder. He said they just both took one step back.
[00:37:38] Prasanna Malaiyandi: but it’s true, right? It’s a stressful thing everyone’s watching to make sure it goes perfect.
[00:37:43] curtis: Right. And, um, so, you know, we wish you all the best of luck. I continue to be concerned about our, our friends over there in the Ukraine. And, uh, we wish them the best of luck and. You should also be concerned about the potential ramifications that all of that has on continued further attacks on your data center and read this article, read every word of this article, not just this summary and, um, you
[00:38:13] Prasanna Malaiyandi: the three
[00:38:14] curtis: read all three parts and we’ll, we’ll put links to it in the show description so that you can easily find it.
Cause finding stuff on Reddit is not necessarily easy. So, uh, Thanks again Prasanna for your wise, uh, shipping advice and, um, you know, a good, good commentary on article well.
[00:38:34] Prasanna Malaiyandi: anytime Curtis and I hope I know, normally when we talk about ransomware, you get very depressed. So I, it feels like this isn’t a depressing article. It feels like here are things you should be doing. So
[00:38:45] curtis: Here are things that you should do now.
[00:38:47] Prasanna Malaiyandi: Yeah,
[00:38:48] curtis: Yeah, absolutely. So, all right, well, thanks to the listeners. Uh, you know, we’d be nothing without you remember to subscribe so that you can restore it all.