Check out our companion blog!
March 6, 2023

What to do with your network in a ransomware attack

What to do with your network in a ransomware attack

We have talked about this a lot on the pod, and now we have someone that can explain what you actually do with your network when you get a ransomware attack. It's Tom Hollingsworth from Gestalt IT, and we're excited to have him on the pod. Some of his recommendations of course, require some configuration in advance. We talk about VLANs, SEIM and access management tools, and why many networking admins are terrified of the "reject all" concept that would actually make your network much more resilient in an attack. There is some really good stuff in this episode.

Mentioned in this episode:

Interview ad

Transcript
Speaker:

I am super excited about this episode.

Speaker:

We have a networking expert coming on and we talk about what to do in your network,

Speaker:

when you get a ransomware attack, I bet you've been wanting to know that answer.

Speaker:

So stay tuned.

W. Curtis Preston:

Hi and welcome to Backup Central's Restore All podcast.

W. Curtis Preston:

I'm your host, W.

W. Curtis Preston:

Curtis Preston, aka a Mr.

W. Curtis Preston:

Backup and have with me possibly my Pex consultant Prasanna Malaiyandi.

W. Curtis Preston:

How's it going?

W. Curtis Preston:

Prasanna,

Prasanna Malaiyandi:

am.

Prasanna Malaiyandi:

I'm good Curtis.

Prasanna Malaiyandi:

And just for people that's p e x, not P E C K S.

W. Curtis Preston:

Yeah, this is the piping, the, uh, the modern

W. Curtis Preston:

piping alternative to copper, which I think is far superior.

W. Curtis Preston:

And, uh, You know what?

W. Curtis Preston:

Just, just for those that are watching this on on video, which is only a

W. Curtis Preston:

handful of you, but I'm gonna tilt my camera up and this is what my office

W. Curtis Preston:

looks like right now because I got yet another pinhole leak in my, um, second

W. Curtis Preston:

story bathroom water supply, which happens to be right above my office.

W. Curtis Preston:

And yesterday I was just sitting here at my desk and I get this

W. Curtis Preston:

drip drip on my face and I'm like,

Prasanna Malaiyandi:

you're like, am I sweating profusely?

W. Curtis Preston:

Yeah.

W. Curtis Preston:

And the pipe, the pipe is actually over there.

W. Curtis Preston:

The, the, the joint that's leaking, it's actually over there, but you know,

W. Curtis Preston:

the water finds its way, you know,

W. Curtis Preston:

along a drywall seam and then it just sort of drips

W. Curtis Preston:

down onto my face.

Prasanna Malaiyandi:

you know what though, Curtis, I have to say

Prasanna Malaiyandi:

congratulations on finally finishing your other project, which we should

Prasanna Malaiyandi:

tell our

W. Curtis Preston:

my other pro.

W. Curtis Preston:

Yeah.

W. Curtis Preston:

Yeah.

W. Curtis Preston:

For those who have been following along my other project is, is, I

W. Curtis Preston:

mean, it, it's still, you know, at this point it's 98% done.

W. Curtis Preston:

But, you know, I have put the, the stair, you know, the, the,

W. Curtis Preston:

the flooring on the stairs.

W. Curtis Preston:

Um, it looks really good.

W. Curtis Preston:

Uh, it looks way better than the ceiling in this room, I will say that.

W. Curtis Preston:

Um, and now there's the official first mess on the new floor

W. Curtis Preston:

. There's just, as I

Prasanna Malaiyandi:

At some point it's gonna happen, you know?

W. Curtis Preston:

yeah.

W. Curtis Preston:

Well, apparently that some point is today.

W. Curtis Preston:

Uh, but yeah, so the plumber, who is a good guy, uh, he's been here before,

W. Curtis Preston:

um, he, um, he's talking to me about, he knows a guy that does, uh, PEX repiping

W. Curtis Preston:

so, we'll,

Prasanna Malaiyandi:

Our listeners will learn all about water piping

Prasanna Malaiyandi:

soon in the next few episodes as we go

W. Curtis Preston:

More, more than you ever wanted to know.

W. Curtis Preston:

Uh, and by the way, I did check they can go through the attic.

W. Curtis Preston:

So, um, that is a, that is a real possibility for the, um, and, um, yeah,

W. Curtis Preston:

so anyway, um, and I just realized that, uh, my son-in-law is home today.

W. Curtis Preston:

He's not normally home.

W. Curtis Preston:

I just heard him making noise.

W. Curtis Preston:

I hope he gets his bath out of the way before the plumber gets here.

W. Curtis Preston:

I didn't, I didn't warn him.

W. Curtis Preston:

Uh, I actually hear, I hear a bath going on right now, . So,

W. Curtis Preston:

so that answer's that question.

W. Curtis Preston:

Way too much information going on at depress pressing

W. Curtis Preston:

household.

W. Curtis Preston:

Well, listen.

Prasanna Malaiyandi:

okay.

W. Curtis Preston:

Yeah, we wanna bring on our guest.

W. Curtis Preston:

Uh, he is both, I would say, a friend of the pod.

W. Curtis Preston:

He's also been an enemy of the pod at once.

W. Curtis Preston:

You may recall that we had an episode where basically we just argued

W. Curtis Preston:

with Tom without his per, without him being here to defend himself.

W. Curtis Preston:

Um, that was over a blog post that he said, uh, something about, uh, backup

W. Curtis Preston:

people reporting to security people.

W. Curtis Preston:

And, uh, I

W. Curtis Preston:

think I had an issue with that or something.

W. Curtis Preston:

Tom has been in the industry about 20 years and he is an

W. Curtis Preston:

event lead over at Gestalt.

W. Curtis Preston:

It the, uh, what would you call it?

W. Curtis Preston:

The makers of the Tech Field Day series, which, uh, uh, my

W. Curtis Preston:

employer has used quite a bit.

W. Curtis Preston:

And, um, uh, we're glad to have him on the podcast.

W. Curtis Preston:

Welcome, Tom Hollingsworth.

Tom Hollingsworth:

Well, thank you for having me on Curtis.

Tom Hollingsworth:

It was, uh, it was fascinating to listen to an episode where I was, I was arguing

Tom Hollingsworth:

with somebody and it wasn't even here.

Tom Hollingsworth:

But, uh, I, I, I love, I love listening to you guys, and I've learned quite a bit.

Tom Hollingsworth:

In fact, uh, the very first time that Curtis and I ever met at Tech Field Day

Tom Hollingsworth:

back in 2011, he was teaching me about data de-duplication, and I was trying to

Tom Hollingsworth:

convince him that IP V6 was important.

Tom Hollingsworth:

And I can tell you which one of those things panned out a lot better than the.

W. Curtis Preston:

Uh, well, you know, is it, is that the thing where

W. Curtis Preston:

you do the nat behind the thing?

W. Curtis Preston:

That's what I recall really learning from you was that you gotta do

Tom Hollingsworth:

if you want me to come crashing through your roof

Tom Hollingsworth:

like the Kool-Aid man, just keep

Tom Hollingsworth:

it up my.

W. Curtis Preston:

Yeah.

W. Curtis Preston:

Yeah.

W. Curtis Preston:

I wanted to bring on somebody that actually understood networking

W. Curtis Preston:

far better than me, right?

W. Curtis Preston:

Which, which is basically many people in the world.

W. Curtis Preston:

With ransomware attacks.

W. Curtis Preston:

One of the things that we talk about is once you've, um, you

W. Curtis Preston:

know, figured out that you actually have a ransomware attack, you,

W. Curtis Preston:

you want to isolate the network.

W. Curtis Preston:

And there there's a discussion, you know, I've been talking with with CISOs lately

W. Curtis Preston:

and, and, and what, what appears to be the reality is that that few environments

W. Curtis Preston:

actually do the, the actual full.

W. Curtis Preston:

Like we just we're just shutting everything off.

W. Curtis Preston:

Right.

Prasanna Malaiyandi:

Go grab the cable, pull it out quick, quick,

W. Curtis Preston:

Speaker:

They're actually, I know.

W. Curtis Preston:

Speaker:

Tom, did you ever watch, uh, alias when it was on

Tom Hollingsworth:

I've seen a couple of

W. Curtis Preston:

and.

W. Curtis Preston:

Okay, well there's an episode in there when they were having a cyber

W. Curtis Preston:

attack and the, uh, what's his name?

W. Curtis Preston:

Um, uh, Marshall Flank man comes running into the data center and he just literally

W. Curtis Preston:

starts flipping, flipping power switches.

W. Curtis Preston:

He's like, they're downloading all the files off the server and he down.

W. Curtis Preston:

He just flips all the power switches off.

W. Curtis Preston:

And so, you know, on one end there is the complete.

W. Curtis Preston:

like networking, shutdown, like literally both internal and external, right?

W. Curtis Preston:

Um, because, you know, once the, once the ransomware is inside, it's gonna try to

W. Curtis Preston:

crawl around and, and make things worse.

W. Curtis Preston:

So that's one way.

W. Curtis Preston:

And then there are, you know, and, and then there's the, those that go, well,

W. Curtis Preston:

well, I'm just going to turn it off, or I'm gonna unplug the cable at the

W. Curtis Preston:

one server or the three servers that appear to be infected and I'm not gonna

W. Curtis Preston:

worry about the rest of the network.

W. Curtis Preston:

And somewhere in the, between those two extremes is what everybody else does.

Prasanna Malaiyandi:

And maybe we should also talk about basics

Prasanna Malaiyandi:

of networking before we jump into this to talk about the detail.

Prasanna Malaiyandi:

Because just

W. Curtis Preston:

Go ahead.

W. Curtis Preston:

Go ahead.

W. Curtis Preston:

Prasanna,

W. Curtis Preston:

what, what do you think we should be talking about first?

Prasanna Malaiyandi:

no, I think it's sort of, because what you just

Prasanna Malaiyandi:

mentioned, Curtis, like everyone might think, oh, all computers

Prasanna Malaiyandi:

are plugged into the same network.

Prasanna Malaiyandi:

Right?

Prasanna Malaiyandi:

I think it's important to talk about some of the best practices from networking,

Prasanna Malaiyandi:

Tom, if you could, about sort of network isolation, BLANs, other things like that.

Prasanna Malaiyandi:

Before we get into sort of the other side of things,

W. Curtis Preston:

Yeah, so please explain all networking technology, period before

Tom Hollingsworth:

the good news.

Tom Hollingsworth:

,you've already talked a little bit about it because it's just

Tom Hollingsworth:

a series of tubes, pipes, if you will, that we send things through.

Tom Hollingsworth:

Uh, now the, the, the important thing to realize when you're trying to think

Tom Hollingsworth:

about how ransomware propagates through a network is to realize that, um, the

Tom Hollingsworth:

way that networks have traditionally been built is we have this perimeter

Tom Hollingsworth:

on the outside, you know, it's probably bounded by firewalls and a bunch of

Tom Hollingsworth:

other stuff, and it looks really, really imposing on the castle walls,

Tom Hollingsworth:

but inside of the network, it's a whole lot easier to get around.

Tom Hollingsworth:

And that's just due to the nature of the way that that networks operate.

Tom Hollingsworth:

I mean, ethernet is effectively like trying to shout out somebody's order

Tom Hollingsworth:

number at a fast food restaurant and hoping that you get the right one.

Tom Hollingsworth:

Everybody's gonna hear the message, but if it's not meant for you,

Tom Hollingsworth:

we're just gonna ignore it.

Tom Hollingsworth:

But the problem is, is that that allows you to propagate a lot of information

Tom Hollingsworth:

very quickly, and that's what ransomware is trying to take, uh, advantage of

Tom Hollingsworth:

whenever it's, it's trying to, uh, do almost like, you know, reconnaissance

Tom Hollingsworth:

lateral movement in the network.

Tom Hollingsworth:

So I'm, I'm looking for a whole bunch of, um, , potentially vulnerable servers

Tom Hollingsworth:

going all the way back, you know, to the beginning of my professional IT career,

Tom Hollingsworth:

I was actually working on a help desk, uh, when the S SQL slammer worm came out.

Tom Hollingsworth:

And boy, you'd be surprised how many people had that port open to the internet,

Tom Hollingsworth:

uh, because everything shut down.

Tom Hollingsworth:

And it was really weird to see that.

Tom Hollingsworth:

And you're like, well, you know, at the time I'm, I'm kind of

Tom Hollingsworth:

freshly minted in my career.

Tom Hollingsworth:

And I'm like, well, how could that happen?

Tom Hollingsworth:

And, and now all these years later, I look at it and go, oh my God,

Tom Hollingsworth:

these people were stupid because you're not supposed to do that.

Tom Hollingsworth:

But that's one of the things that people want to take advantage of because the,

Tom Hollingsworth:

the systems want to talk to each other.

Tom Hollingsworth:

They want to be able to exchange information.

Tom Hollingsworth:

That's the purpose of a network.

Tom Hollingsworth:

You actually have to do extra work to prevent them from talking to each other.

W. Curtis Preston:

Right.

W. Curtis Preston:

Yeah.

W. Curtis Preston:

I think that's, you know, I, I, and the, the number of times I went in and out

W. Curtis Preston:

of data centers, uh, over the years, I remember only one, uh, where they had

W. Curtis Preston:

very solid internal firewalls, basically.

W. Curtis Preston:

Right.

W. Curtis Preston:

That, that it was very difficult to do, to traverse laterally

W. Curtis Preston:

within the organization.

W. Curtis Preston:

And that was actually Intuit, uh, right.

W. Curtis Preston:

And, and it's because of what they felt they had.

W. Curtis Preston:

Right.

W. Curtis Preston:

They had all of this very sensitive personal data, thanks to their, you know,

W. Curtis Preston:

they, they had QuickBooks, they have TurboTax, they have all of that stuff.

W. Curtis Preston:

And so they had to basically firewall off systems between each other to

W. Curtis Preston:

prevent that lateral movement that you're right by design in most networks, you

W. Curtis Preston:

buy a switch, you buy, well, a bunch of switches, you plug everything in.

W. Curtis Preston:

And everything talk, everything can talk to everything.

W. Curtis Preston:

Um, and unless you do something to prevent it, a lot of those ports that

W. Curtis Preston:

you talked about, right, just like the SQL, uh, issue, a lot of those

W. Curtis Preston:

ports are visible to the internet.

W. Curtis Preston:

Right.

W. Curtis Preston:

I, I think a, another one would be a, a vCenter Right.

W. Curtis Preston:

And Hyper V, the, that, those ports being visible to the internet, I suppose

W. Curtis Preston:

you hear about that a lot as well.

Tom Hollingsworth:

Yeah.

Tom Hollingsworth:

I usually do.

Tom Hollingsworth:

Whenever there's some kind of, uh, a vulnerability that comes out and

Tom Hollingsworth:

everyone's like, I hope you don't have these exposed to the internet, and

Tom Hollingsworth:

you can literally hear the scrabbling as people run into their keyboards

Tom Hollingsworth:

to figure out if that's the case.

Tom Hollingsworth:

But, you know, as,

Tom Hollingsworth:

as Prasanna mentioned, I mean, we have ways to kind of like segment

Tom Hollingsworth:

networks away from each other.

Tom Hollingsworth:

And it's funny that you bring up that, that Intuit had kind of a, a rigorous

Tom Hollingsworth:

internal firewall structure because in my experience, um, companies or organizations

Tom Hollingsworth:

that are very, uh, heavily regulat.

Tom Hollingsworth:

Have much more strict internal structure.

Tom Hollingsworth:

And the reason for that is because they need the ability to say

Tom Hollingsworth:

for a fact, Curtis cannot see anything on this network because he

Tom Hollingsworth:

hasn't been authorized to see it.

Tom Hollingsworth:

Now, you can do that through software constructs.

Tom Hollingsworth:

I mean, VLANs, virtual local area networks are kind of the, the most common

Tom Hollingsworth:

way to do it, where we, we effectively divide some, uh, uh, partition on the

Tom Hollingsworth:

switch and we say, this port belongs to this vlan, so it can only talk to

Tom Hollingsworth:

other ports that are on that vlan.

Tom Hollingsworth:

Uh, but that's not even good enough for some organizations.

Tom Hollingsworth:

And, and the, the one that everybody always thinks of is Mission Impossible,

Tom Hollingsworth:

the Tom Cruise movie with the, the machine that's in a vault that's

Tom Hollingsworth:

not connected to anything else.

Tom Hollingsworth:

We would call that an air gap system.

Tom Hollingsworth:

Or you can have an air gap network a lot of times things like, um, HVAC or

Tom Hollingsworth:

management systems are air gap from the rest of the network because they

Tom Hollingsworth:

have different controls and different needs, but I also don't trust those

Tom Hollingsworth:

people to, um, secure their stuff.

Tom Hollingsworth:

So I'm gonna build a wall in front of that air gap or just completely

Tom Hollingsworth:

isolate it, uh, itself so that I don't have to worry about securing it.

Tom Hollingsworth:

And if, uh, you, you say hvac, you say things like, you know, uh, um,

Tom Hollingsworth:

environmental control systems and any security people listening to this

Tom Hollingsworth:

podcast are immediately thinking, man, those are back doors that I

Tom Hollingsworth:

can use to get into the system.

Tom Hollingsworth:

Because no matter what, they're still gonna have to be connected

Tom Hollingsworth:

to the network somehow.

Tom Hollingsworth:

And that just increases your, um, you know, your threat profile.

W. Curtis Preston:

Right.

Prasanna Malaiyandi:

Yeah, it's interesting because I think most people

Prasanna Malaiyandi:

who think about home networks, right?

Prasanna Malaiyandi:

Everything's typically flat in a home, right?

Prasanna Malaiyandi:

Everything can talk to everything, every single iot device out there, right?

Prasanna Malaiyandi:

And they're not always thinking about, Hey, I got this smart light bulb.

Prasanna Malaiyandi:

Isn't it great?

Prasanna Malaiyandi:

Isn't it awesome?

Prasanna Malaiyandi:

And then realizing that's on my network, everything is now exposed and could

Prasanna Malaiyandi:

be potentially exposed if there's a security issue with that single device,

Tom Hollingsworth:

Those devices are, you know, they obviously have an IP address,

Tom Hollingsworth:

they have some kind of a control system.

Tom Hollingsworth:

You would hope that most of them have some kind of a security function that

Tom Hollingsworth:

allows them to, to securely communicate back to whatever controls them.

Tom Hollingsworth:

But multiply that by a factor of 10 for all of the devices that could be

Tom Hollingsworth:

on your average enterprise network.

Tom Hollingsworth:

And when you start saying things like, you know, access controls for those devices,

Tom Hollingsworth:

or port security like network engineering and, and operations folks, like, they

Tom Hollingsworth:

just start breaking out into hives.

Tom Hollingsworth:

because like the, the, just the amount of work that it takes to create that

Tom Hollingsworth:

level of security is its own monster.

Tom Hollingsworth:

I mean, anyone who's ever deployed a technology like 8 0 2 0.1 x, which

Tom Hollingsworth:

is effectively, I am only gonna allow authorized devices to be plugged into

Tom Hollingsworth:

this port, knows that like there's this whole enrollment process and

Tom Hollingsworth:

are you on the authorized users list?

Tom Hollingsworth:

And what happens if you're using a different device today?

Tom Hollingsworth:

And it's just, it's maddening and it, it drives people to insane to the

Tom Hollingsworth:

point where, and that's the normal people who know what they're doing.

Tom Hollingsworth:

Could you imagine an executive plugging their laptop into a network port one

Tom Hollingsworth:

day and going, this doesn't work.

Tom Hollingsworth:

And you tell 'em, oh, it's doing that on purpose because we

Tom Hollingsworth:

want to keep everything secure.

Tom Hollingsworth:

What do you think is gonna happen?

Tom Hollingsworth:

The executive's probably gonna look at you and go, I don't care.

Tom Hollingsworth:

Make

Prasanna Malaiyandi:

Turn it off.

Prasanna Malaiyandi:

Exactly.

Prasanna Malaiyandi:

. We don't need that.

Tom Hollingsworth:

It's getting

Tom Hollingsworth:

in my way.

Prasanna Malaiyandi:

Yeah,

W. Curtis Preston:

Yeah.

W. Curtis Preston:

Well, I know that when we, when we had, um, you know, we had a, a, a security

W. Curtis Preston:

person on and they had a list of things that they wanted people to do that they

W. Curtis Preston:

felt were common sense, that were, um, ways to prevent basically, sort of,

W. Curtis Preston:

I, I think the proper thing today when we talk about ransomware is to just

W. Curtis Preston:

assume something in your, in your world is going to get ransomware, right?

W. Curtis Preston:

It's just, it is, I think it's just impossible to, to,

W. Curtis Preston:

to stop it 100% of the time.

W. Curtis Preston:

So just assume that's going to happen.

W. Curtis Preston:

So then there's all about.

W. Curtis Preston:

How to prevent it from activating itself, from talking to the command and control

W. Curtis Preston:

servers and also the lateral movement.

W. Curtis Preston:

Right?

W. Curtis Preston:

So he

Prasanna Malaiyandi:

reducing the black

W. Curtis Preston:

Speaker:

lateral movement, right?

W. Curtis Preston:

Speaker:

So what's that?

Prasanna Malaiyandi:

Limiting the blast

W. Curtis Preston:

radius.

W. Curtis Preston:

So, so Tom, what, what kinds of things besides VLANs?

W. Curtis Preston:

Because even VLANs, you know, we have the, we have the VLAN

W. Curtis Preston:

for this and the VLAN for that.

W. Curtis Preston:

Still all the servers within that VLAN can talk to each other.

W. Curtis Preston:

What else can companies do, uh, with modern networking equipment to prevent

W. Curtis Preston:

lateral movement or to basically prevent it from everything and then, and then, uh,

W. Curtis Preston:

selectively allow it for certain servers.

Tom Hollingsworth:

Well, the first thing you have to do is you have to

Tom Hollingsworth:

realize that a completely flat network.

Tom Hollingsworth:

Is not a stable network.

Tom Hollingsworth:

I mean, there is a limit to the amount of chatter that a network can tolerate

Tom Hollingsworth:

before it starts running into problems.

Tom Hollingsworth:

Um, ethernet is not a, uh, a medium that allows for a large number of hosts because

Tom Hollingsworth:

eventually they're gonna, it, you know, it's like recording a podcast eventually

Tom Hollingsworth:

with too many guests on the podcast.

Tom Hollingsworth:

You're all gonna wanna talk over the top of each other,

Tom Hollingsworth:

and ethernet doesn't like that.

Tom Hollingsworth:

So once you had a certain boundary, you kind of have to divide it

Tom Hollingsworth:

up into these little domains.

Tom Hollingsworth:

Um, collision domains are what we call them, and that's one

Tom Hollingsworth:

of the things that a VLAN is.

Tom Hollingsworth:

But as we've learned over the years about what we really should be doing,

Tom Hollingsworth:

we've kind of built a super set of that.

Tom Hollingsworth:

And anyone out there who has been reading any kind of the tech press recently, or

Tom Hollingsworth:

been to any trade show in the last couple of years, probably heard of something

Tom Hollingsworth:

like Zero Trust Network Architecture or, or, you know, just Zero Trust in general.

Tom Hollingsworth:

It's a buzzword.

Tom Hollingsworth:

I'm, I'll be the first to admit it, but the principles behind it are fairly sound.

Tom Hollingsworth:

what you do is you take the tools that you've already been given, those ones

Tom Hollingsworth:

that I told you, make your network team break out in hives, and you try to

Tom Hollingsworth:

implement them in such a way as to reduce the complexity of the implementation.

Tom Hollingsworth:

And think about like, you know, think about a teenager and they

Tom Hollingsworth:

want a, a list of, uh, things that they can do when they get a car.

Tom Hollingsworth:

Are you gonna tell them you can do anything you want, but

Tom Hollingsworth:

you can't do this and you can't do that and you can't do this?

Tom Hollingsworth:

Or are you gonna be more explicit?

Tom Hollingsworth:

You can only do these things and if it's not on that list, you can't do it.

Tom Hollingsworth:

Well, most people would say, well, I'm only go, I'm gonna do the second

Tom Hollingsworth:

thing because I want to make sure that they're only going to school and to

Tom Hollingsworth:

work into this one friend's house.

Tom Hollingsworth:

But we don't build networks that way.

Tom Hollingsworth:

I mean, we, we typically allow as much as possible because of the

Tom Hollingsworth:

situations we find ourselves in where something doesn't work right.

Tom Hollingsworth:

And we don't know why.

Tom Hollingsworth:

So we will put a little catchall at the bottom of the, the access

Tom Hollingsworth:

list and go permit everything else.

Tom Hollingsworth:

and then we leave it.

Tom Hollingsworth:

And that's the worst thing that you can do.

Tom Hollingsworth:

And what Zero Trust Network architectures try to do is they try

Tom Hollingsworth:

to say, okay, that server over there is running our backup software.

Tom Hollingsworth:

What should it, what should communicate with it?

Tom Hollingsworth:

And how should it be communicated with, you know, maybe it only needs to accept

Tom Hollingsworth:

connections on these three or four ports.

Tom Hollingsworth:

Maybe it only accepts connections from these authorized users.

Tom Hollingsworth:

And you're effectively creating an isolation for that unit.

Tom Hollingsworth:

And if something needs to access it and you're having problems with it, the

Tom Hollingsworth:

software usually allows you to kind of dig into that a little bit and go, oh, it

Tom Hollingsworth:

looks like that this program did an update and it now needs to communicate over this

Tom Hollingsworth:

port, uh, and I need to allow that port.

Tom Hollingsworth:

But you're doing it in a, in a way that allows you to kind of control that access.

Tom Hollingsworth:

But more importantly, what happens is that when something tries to operate

Tom Hollingsworth:

outside of that access control, it slams it shut and hopefully will send

Tom Hollingsworth:

you some kind of a warning, you know, Hey, we just noticed that this server

Tom Hollingsworth:

over here is trying to communicate with the rest of the network on Port 4 45.

Tom Hollingsworth:

and I know it shouldn't be doing that.

Tom Hollingsworth:

You need to take a look at it.

Tom Hollingsworth:

And so limiting that blast radius, that broadcast capability tends

Tom Hollingsworth:

to prevent lateral movement.

Tom Hollingsworth:

And like you said, people who are going to attack you are, are

Tom Hollingsworth:

going to be dedicated in doing it.

Tom Hollingsworth:

Either they're gonna be dedicated to looking for a very specific exploit and

Tom Hollingsworth:

just kind of hauling in whatever they can do, or they're gonna be looking to

Tom Hollingsworth:

attack you, you specifically, however they can get to you that second kind

Tom Hollingsworth:

of attacker, very difficult to block.

Tom Hollingsworth:

It's like a door lock, a dedicated burglar is gonna get into your house.

Tom Hollingsworth:

You're looking to prevent more of the first one where it's like, oh, we were

Tom Hollingsworth:

able to get in through your HVAC system and boy, we're gonna turn this thing loose

Tom Hollingsworth:

and see what open file shares you've got out there and what we can do with them.

Tom Hollingsworth:

You, you need to create structure in the organization that does not allow

Tom Hollingsworth:

people to move laterally that that prevents them from accessing things.

Tom Hollingsworth:

Or worse yet, alerts you when things start doing a lot of scanning across

Tom Hollingsworth:

your network, looking for those kinds of things because the, the rest of the group

Tom Hollingsworth:

that's trying to get into your network doesn't know that stuff's there either.

Tom Hollingsworth:

They're gonna have to go looking and just like the burglars that are casing the

Tom Hollingsworth:

joint, you need to look for those people.

Prasanna Malaiyandi:

So multiple things popped up in my head,

Prasanna Malaiyandi:

Tom, as you were talking.

Prasanna Malaiyandi:

So the first is, as you're talking about the burglar example, I'm gonna bring this

Prasanna Malaiyandi:

up again for the second week, but Curtis had recommended reading The Cuckoo's Egg.

Prasanna Malaiyandi:

I don't know if you've read that book.

Prasanna Malaiyandi:

Tom.

Prasanna Malaiyandi:

Highly recommend you read it.

Prasanna Malaiyandi:

It's basically, 1980s, a hacker gets into a mainframe and starts moving

Prasanna Malaiyandi:

laterally across all these like military networks and science networks

Prasanna Malaiyandi:

because everything was connected.

Prasanna Malaiyandi:

And like you said, that example was go and try all the door locks and he

Prasanna Malaiyandi:

would try default passwords and some of these systems, like the mainframes,

Prasanna Malaiyandi:

people would not change the defaults.

Prasanna Malaiyandi:

And so he got in and it was just that lateral movement across

Prasanna Malaiyandi:

everything in the environment.

Prasanna Malaiyandi:

So that's like the first thing that came to mind as you were talking.

Prasanna Malaiyandi:

Um, the other thing that also came to mind is I totally get the reason to have

Prasanna Malaiyandi:

like that zero trust and only enables services that, and patterns that are known

Prasanna Malaiyandi:

to be valid and disable everything else.

Prasanna Malaiyandi:

Uh, my question.

Prasanna Malaiyandi:

As a network engineer or operations person, how do

Prasanna Malaiyandi:

you manage that at the scale?

Prasanna Malaiyandi:

Because there's so many applications, so many servers, it's hard to predict

Prasanna Malaiyandi:

what's going to talk with what, um, and coming up with, because

Prasanna Malaiyandi:

like, everything's all connected.

Prasanna Malaiyandi:

Like in my mind I think about like Facebook and graphs, right?

Prasanna Malaiyandi:

Everything is connected in the world, right?

Prasanna Malaiyandi:

And so everything in your network to some extent is probably

Prasanna Malaiyandi:

connected in some form or fashion.

Prasanna Malaiyandi:

So how do you sort of go about even coming up with, okay, these things

Prasanna Malaiyandi:

are the things that should be talking to the backup server in your example.

Tom Hollingsworth:

So it takes a lot of teamwork because as a network person,

Tom Hollingsworth:

I don't care what's running over my network, I just need to make sure that

Tom Hollingsworth:

these two things can talk to each other.

Tom Hollingsworth:

And so in a way, like if you've ever deployed a server, um, you, you have a

Tom Hollingsworth:

list, okay, it needs to communicate, uh, using this protocol over these ports or,

Tom Hollingsworth:

you know, uh, think about, uh, opening something like, I need to open HTTPS to

Tom Hollingsworth:

the server, but not http because I don't want it to ever communicate over http.

Tom Hollingsworth:

And that's actually one of the things that we've noticed a lot recently is that a

Tom Hollingsworth:

lot of protocols that used to have their own dedicated ports have now just started

Tom Hollingsworth:

writing over, uh, HTTP and https s.

Tom Hollingsworth:

Because it's just easier.

Tom Hollingsworth:

Uh, bit Torrent was actually one of the first ones to start doing this because

Tom Hollingsworth:

they're like, well, eighty's gonna be open anyway, which is the port for http.

Tom Hollingsworth:

So we'll just ride on that because most people fire, most people's firewalling

Tom Hollingsworth:

systems just allow that by default, because that's what the web uses.

Tom Hollingsworth:

And so it gets kind of insidious and you almost have to think at a higher level.

Tom Hollingsworth:

So what.

Tom Hollingsworth:

it crack open any networking textbook in the world, and they're gonna

Tom Hollingsworth:

give you this seven layer model.

Tom Hollingsworth:

It's like a seven layer dip from Taco Bell, but there's no refried

Tom Hollingsworth:

beans in the seven layer OSI model.

Tom Hollingsworth:

But we play a lot in the bottom of that, where the physical connections

Tom Hollingsworth:

happen, where the IP addresses allow systems to talk to each other.

Tom Hollingsworth:

Once we get above a certain level, that's where the applications take over.

Tom Hollingsworth:

And as networking people, we're not as concerned about that.

Tom Hollingsworth:

But boy, the server people are because, oh, you know, I need to be able to have

Tom Hollingsworth:

these two devices talking to each other.

Tom Hollingsworth:

I need to make sure this is all un impeded.

Tom Hollingsworth:

And the first thing that happens when two servers can't talk to each other is you

Tom Hollingsworth:

gotta find the network people, people.

Tom Hollingsworth:

And you're like, you need to tell me what's going on here.

Tom Hollingsworth:

And then invariably, like the security team gets drawn in because like, oh no,

Tom Hollingsworth:

we told him that he had to block that because nobody should ever be using that.

Tom Hollingsworth:

And, and you, you really do have to pull those people together.

Tom Hollingsworth:

I mean, think of, you know, think of a book like, uh, gene Kim's Phoenix project.

Tom Hollingsworth:

Like you can't work in isolation anymore.

Tom Hollingsworth:

As much as we might like to.

Tom Hollingsworth:

Because so many things are so inter interdependent now.

Tom Hollingsworth:

It's like, you know, the, the old joke is, is what does the server do?

Tom Hollingsworth:

I don't know.

Tom Hollingsworth:

Unplug the cable and we'll see who screams the loudest.

Tom Hollingsworth:

You wanna figure out what people, uh, what port is being used.

Tom Hollingsworth:

Let's block it and see who comes to yell at us.

Tom Hollingsworth:

Like, that's kind of the way you have to do some of these things.

Tom Hollingsworth:

Cuz

Tom Hollingsworth:

the other thing, and we, we all know that nobody, nobody ever

Tom Hollingsworth:

skips documentation, right?

Tom Hollingsworth:

I realized while editing this episode, that we forgot to

Tom Hollingsworth:

throw out our disclaimer.

Tom Hollingsworth:

Uh, Prasanna and I work for different companies.

Tom Hollingsworth:

He works for zoom.

Tom Hollingsworth:

I work for Druva.

Tom Hollingsworth:

This is not a podcast of either company.

Tom Hollingsworth:

It is an independent podcast.

Tom Hollingsworth:

So the opinions that you hear are ours.

Tom Hollingsworth:

Also, if you'd like to join the podcast, please reach out to me

Tom Hollingsworth:

at w Curtis Preston on, uh, at Gmail or at WC Preston on Twitter.

Tom Hollingsworth:

Or linkedin.com/i N slash Mr.

Tom Hollingsworth:

Backup.

Tom Hollingsworth:

And you'll find me, uh, we'd love to have you join and also be sure to

Tom Hollingsworth:

rate us at your favorite podcatcher.

Tom Hollingsworth:

Thanks a lot.

Tom Hollingsworth:

Now onto my silly story.

W. Curtis Preston:

You brought up an old memory of mine.

W. Curtis Preston:

Literally like my first months in being a cis admin, we were trying to decommission,

W. Curtis Preston:

um, uh, the, you know, the, the, the first computer designed to run Unix was the

W. Curtis Preston:

three BK and the at and t had a three BK I think it was like a three B 1000.

W. Curtis Preston:

And it was their attempt at a multiprocessor architecture.

W. Curtis Preston:

And we had this beast and we were trying to decommission it.

W. Curtis Preston:

And, uh, we had gotten down to, we had fi you know, and, and we had gotten down

W. Curtis Preston:

to that phase where it's like, well, we're just gonna turn it off and whoever

W. Curtis Preston:

yells will be the one that we missed.

W. Curtis Preston:

Right.

W. Curtis Preston:

But I remember the, um, We had, uh, stripped it, all of, all of

W. Curtis Preston:

its regular networking cable.

W. Curtis Preston:

I don't exactly remember exactly why, but I remember that there was one cable

W. Curtis Preston:

left and it was running across the floor and we were doing the last like

W. Curtis Preston:

download of, of whatever it was off of this server onto something else.

W. Curtis Preston:

And the manager for that cost center was in there and he

W. Curtis Preston:

kept stepping on the cable.

W. Curtis Preston:

And, um, we told him that he was slowing down the download whenever

W. Curtis Preston:

he would step on the cable.

W. Curtis Preston:

And, um, we actually caught him, we left him into data center.

W. Curtis Preston:

We actually caught him like watching the monitor and like the throughput speed

W. Curtis Preston:

and sort of stepping on and stepping up and off and off on the cable.

W. Curtis Preston:

Anyway.

W. Curtis Preston:

Yeah.

W. Curtis Preston:

Good, good stories.

W. Curtis Preston:

I, so the question I want to ask you about, all of the things you just talked

W. Curtis Preston:

about, is this something built into modern networking equipment or is this, um,

W. Curtis Preston:

you know, are these extra applications that I'm buying that then configure

W. Curtis Preston:

that networking equipment for me?

Tom Hollingsworth:

So it can be both.

Tom Hollingsworth:

The, the basics of being able to isolate hosts and configure systems

Tom Hollingsworth:

has been built in for years.

Tom Hollingsworth:

I mean, anyone can write an a c L, right?

Tom Hollingsworth:

The thing is, is that scaling that across a large organization

Tom Hollingsworth:

is where it typically falls down.

Tom Hollingsworth:

Eventually, your security team can't keep up with all the changes.

Tom Hollingsworth:

They throw their hands up in the air and it lies fallow for as long as

Tom Hollingsworth:

it takes for you to get infected.

Tom Hollingsworth:

So the additional tools that are basically being brought to market and are, are

Tom Hollingsworth:

popular now, kind of organize that system.

Tom Hollingsworth:

They put a, a, a shiny.

Tom Hollingsworth:

UI on it, if you will, to, to go in and say, okay, I, I want to enable port

Tom Hollingsworth:

security on these ports because back when I started this port security was,

Tom Hollingsworth:

if it isn't being used, shut it off.

Tom Hollingsworth:

Just like shut down the port.

Tom Hollingsworth:

And then if somebody plugs into it and it doesn't work, well then now we know

Tom Hollingsworth:

we need to enable that port and we need to know who's trying to use it.

Tom Hollingsworth:

But now you have the ability to like have somebody plug in a device,

Tom Hollingsworth:

whether it's an IOT system or what have

Tom Hollingsworth:

you, and this, the device will like register with the system.

Tom Hollingsworth:

It'll say, Hey, I need access.

Tom Hollingsworth:

And then the system can come back and say, Hey, it looks like somebody

Tom Hollingsworth:

plugged in an S thermostat over here.

Tom Hollingsworth:

Well, that's actually a bad example cause they don't use wires, but you know, a

Tom Hollingsworth:

laptop or some other kind of device, you need to go, like check it out.

Tom Hollingsworth:

Or you can even set a policy that says, I'm going to allow you for

Tom Hollingsworth:

now, but I have the ability to just cut it off if I need to.

Tom Hollingsworth:

Or if it's one of these recognized device classes or something like that.

Tom Hollingsworth:

So for smaller systems, , you know, for, for smaller organizations, if

Tom Hollingsworth:

your IT department isn't already completely overworked, you can't

Tom Hollingsworth:

implement some of this by hand.

Tom Hollingsworth:

It's just a matter of if it works really well, that means you're gonna

Tom Hollingsworth:

be spending a lot of time tuning that system to keep working effectively.

Tom Hollingsworth:

And once you get past a certain point, the, uh, the solutions

Tom Hollingsworth:

that do this are reassuringly expensive because they're worth it.

W. Curtis Preston:

Right.

W. Curtis Preston:

Oh, I understood cuz that, that they would help you save the, the, the labor.

W. Curtis Preston:

And is there a category of these types of tools that, that a

W. Curtis Preston:

category name that we give to them?

Tom Hollingsworth:

Uh, there's, there's a bunch of different ones.

Tom Hollingsworth:

Uh, access management is typically one that you, you see, um, honestly,

Tom Hollingsworth:

tools like Aruba ClearPass or uh, Cisco ice, uh, ise, uh, integrated

Tom Hollingsworth:

services engine, or integrated security engine, I forget which one it is.

Tom Hollingsworth:

But they're, they're, they're not identity and access management,

Tom Hollingsworth:

although they can be integrated that.

Tom Hollingsworth:

There are some smaller ones that, that have these capabilities.

Tom Hollingsworth:

A lot of it is, is mostly figuring out what you need because there's

Tom Hollingsworth:

different, you know, some systems are, are configured so that you're

Tom Hollingsworth:

controlling access to devices.

Tom Hollingsworth:

I only wanna authorize people to be able to log into this

Tom Hollingsworth:

device and make changes to it.

Tom Hollingsworth:

Well, that's different than I want to change the way that people

Tom Hollingsworth:

in my network are accessing data like that is a different kind of

Tom Hollingsworth:

identity and access management.

Tom Hollingsworth:

So you need to do a little bit of investigative work to make sure that you

Tom Hollingsworth:

are, uh, properly using the right tool.

Tom Hollingsworth:

Cause if you spend a lot of money on one that doesn't give you what you want or

Tom Hollingsworth:

does a, a, a terrible job of it, then not only are you gonna be upset, but the

Tom Hollingsworth:

people that are or authorizing your budget are not gonna be very happy with you.

Prasanna Malaiyandi:

Now a lot of these changes, if I think about an

Prasanna Malaiyandi:

enterprise environment, things are easier to a fair extent to control, right?

Prasanna Malaiyandi:

If you're looking at servers or virtualization, other things like that.

Prasanna Malaiyandi:

But then I go to think about other environments like a school, right, where

Prasanna Malaiyandi:

you have students coming and going, right?

Prasanna Malaiyandi:

Or a stadium or a conference center, right?

Prasanna Malaiyandi:

Does it get significantly more difficult to do what you talked

Prasanna Malaiyandi:

about, Tom, in those environments?

Prasanna Malaiyandi:

Or can the same tools apply there as well?

Tom Hollingsworth:

Yes and no.

Tom Hollingsworth:

Um, I, I, I'm, I'm the typical IT nerd.

Tom Hollingsworth:

The answer is, it depends for whatever question you ask, but I'll tell you

Tom Hollingsworth:

that in some ways, um, schools and other places where your user base

Tom Hollingsworth:

is not employed directly by you.

Tom Hollingsworth:

Can have a slightly easier time if you're willing to, um, sacrifice a little bit.

Tom Hollingsworth:

So I know that there are a lot of colleges out there that treat their student

Tom Hollingsworth:

dorm networks like the wild, wild west.

Tom Hollingsworth:

We don't care what goes on out there, but we're not gonna keep an eye on it either.

Tom Hollingsworth:

So like, if

Tom Hollingsworth:

there's a, you know, a piece of ransomware or something that's running rampant

Tom Hollingsworth:

through the system, all we did is tell you that you had to have your antivirus

Tom Hollingsworth:

up to date to be able to join our network.

Tom Hollingsworth:

So,

Tom Hollingsworth:

so what, uh, the stadiums are actually a, a really interesting, uh, problem

Tom Hollingsworth:

too, because not only do you have a a, a group of users that are outside of

Tom Hollingsworth:

your control, they're very transient, um, in a lot of those places.

Tom Hollingsworth:

Like they, they actually have, uh, wireless networks that are set up

Tom Hollingsworth:

so that, um, they can only talk.

Tom Hollingsworth:

, like they block all device to device communication, which

Tom Hollingsworth:

is something that you can do.

Tom Hollingsworth:

It's a little bit more complicated, but it effectively treats, um, the stadium

Tom Hollingsworth:

itself like a demilitarized zone in a, uh, in a, in a security structure.

Tom Hollingsworth:

So for most people that are, that are familiar with it, you know, you've got the

Tom Hollingsworth:

outside internet, which is big and scary.

Tom Hollingsworth:

You've got your inside network, which is soft and, and you know,

Tom Hollingsworth:

uh, you don't want it to get hurt.

Tom Hollingsworth:

And then in the middle you have the dmz, which is basically the moat where

Tom Hollingsworth:

you're like, I'm gonna put everything that I don't care if it gets attacked

Tom Hollingsworth:

out there so that if it breaks, it can't get back into my network.

Tom Hollingsworth:

And so, but the otherwise, the other thing there is I only allow

Tom Hollingsworth:

certain traffic to come back through.

Tom Hollingsworth:

So if something bad were to happen, I can just basically cut it off and

Tom Hollingsworth:

sink it into the moat and I'm done.

W. Curtis Preston:

Yeah, I think, uh, hotels have a similar model, right?

W. Curtis Preston:

Where the base, uh, I know having, having plugged in multiple devices that

W. Curtis Preston:

needed to talk to each other in hotel networks, they don't like that very much.

W. Curtis Preston:

Uh, and you end up having to bring basically your own router if, if

W. Curtis Preston:

that's something that you want to do.

W. Curtis Preston:

Right?

W. Curtis Preston:

Um, so the, so it sounded like, um, if I understood you correctly,

W. Curtis Preston:

the access management part is this sort of basic security thing, that

W. Curtis Preston:

there are tools that do just that, and then there's also this identity

W. Curtis Preston:

access, which is a, a bigger pain.

W. Curtis Preston:

I would, I would imagine.

W. Curtis Preston:

But those that want that, and it sounds like when we put those

W. Curtis Preston:

two together, that's what, what we call a SEIM tool, right?

W. Curtis Preston:

Is uh, uh, identity and access management.

W. Curtis Preston:

But it sounds like there's just an access management.

W. Curtis Preston:

That, for those that need just that there, there's smaller and less

W. Curtis Preston:

expensive than a full SEIM tool.

W. Curtis Preston:

Yeah.

W. Curtis Preston:

Not, not inexpensive, but just less expensive.

Tom Hollingsworth:

Well, and it, it also matters as to what you're spending

Tom Hollingsworth:

your resources on, because there are tools that will do this for free.

Tom Hollingsworth:

But they are not supported at all by anybody other than people on a forum.

Tom Hollingsworth:

And they'll be glad to tell you that you misconfigured something

Tom Hollingsworth:

and go figure it out yourself.

Tom Hollingsworth:

Like we, we've dealt with that.

Tom Hollingsworth:

And I'm not really crapping on the open source community because

Tom Hollingsworth:

they do an amazing job of this.

Tom Hollingsworth:

I'm crapping on the fact that open source communities are not as well supported

Tom Hollingsworth:

as the bigger players in these markets.

Tom Hollingsworth:

And that's honestly where the expensive part comes from.

Tom Hollingsworth:

You're not paying for the software, although you, you

Tom Hollingsworth:

kind of are in some ways.

Tom Hollingsworth:

You're paying for somebody to answer the phone when somebody is like breathing

Tom Hollingsworth:

down your neck because something won't work or something won't come online.

Tom Hollingsworth:

And so a and a and you're also trying to get to that point where

Tom Hollingsworth:

it's, it's not automated as much as it is as low friction as possible.

Tom Hollingsworth:

Because what you want in situations is people to just

Tom Hollingsworth:

be able to get on the network.

Tom Hollingsworth:

That's, that's the thing.

Tom Hollingsworth:

If you've ever tried to log into a wifi network that has a captive portal

Tom Hollingsworth:

that requires you to like accept a whole bunch of licensing agreements

Tom Hollingsworth:

and type your room number in and all the other stuff, you know that it's not

Tom Hollingsworth:

the most frustrating thing, but it's definitely not what you want to hear.

Tom Hollingsworth:

As opposed to like, oh, this device has already been pre-authorized cuz you logged

Tom Hollingsworth:

in with your active directory username.

Tom Hollingsworth:

Well, we'll just let it on the network.

Tom Hollingsworth:

That's completely frictionless.

Tom Hollingsworth:

But the amount of effort that it takes to make it frictionless is where your time

Tom Hollingsworth:

and resource invests gonna come from.

Prasanna Malaiyandi:

Tom, I know we started this all out with Curtis asking,

Prasanna Malaiyandi:

how do you prevent lateral movement in networks right from ransomware?

Prasanna Malaiyandi:

Just given the fact that ransomware does move laterally in a lot of networks?

Prasanna Malaiyandi:

Does this mean people are not using these tools or have not

Prasanna Malaiyandi:

configured the networks correctly?

Prasanna Malaiyandi:

Because it seems like if you did all the things that we just talked about, it

Prasanna Malaiyandi:

should have prevented a lot of the lateral movement that we see in ransomware today.

Tom Hollingsworth:

Well, Prasanna, I'm gonna tell you something

Tom Hollingsworth:

that my dad always tell me, and you have to understand.

Tom Hollingsworth:

My dad grew up in the country.

Tom Hollingsworth:

If a frog had wings, he wouldn't bump his ass every time he hopped.

Tom Hollingsworth:

So yes, if you turn on all of these tools, you will cut down on a lot of this stuff.

Tom Hollingsworth:

But does that mean your network's not working correctly?

Tom Hollingsworth:

No.

Tom Hollingsworth:

It just means that we didn't enable all these extra features that we have

Tom Hollingsworth:

to keep track of because I can get four, uh, four network ports on a.

Tom Hollingsworth:

and plug four devices in there and they're gonna work is the

Tom Hollingsworth:

best way for them to work.

Tom Hollingsworth:

Absolutely not, but I also don't have to do a whole lot of extra configuration.

Tom Hollingsworth:

A lot of people are looking at this from the perspective of, I need to

Tom Hollingsworth:

make sure that everything is, is able to communicate with everything else.

Tom Hollingsworth:

They're not looking at it like you, like the example you had earlier, Curtis, when

Tom Hollingsworth:

you log into the hotel wifi and I can't talk to anything else on the hotel wifi.

Tom Hollingsworth:

They're not thinking in a, in an isolation mode.

Tom Hollingsworth:

And we're, we're that ship's turning because a lot of people are now

Tom Hollingsworth:

realizing that, that that traditional idea of having a very stiff, crunchy

Tom Hollingsworth:

perimeter with a very soft internal network doesn't work so well.

Tom Hollingsworth:

Because what ends up happening is, is that once people get through

Tom Hollingsworth:

the perimeter, they have free reign to do whatever they want.

Tom Hollingsworth:

You, you do have to build these controls in place to effectively

Tom Hollingsworth:

slow them down or to herd them to places that you want them to go.

Tom Hollingsworth:

And that's what a lot of people have spent time developing and working on.

Tom Hollingsworth:

And there's varying degrees of success to make that work.

Tom Hollingsworth:

It has to shift the mindset though.

Tom Hollingsworth:

Um, you know, application people are just turn on all the ports

Tom Hollingsworth:

and I'll turn them off later.

Tom Hollingsworth:

When I tell you which ones I don't need, you won't, because you'll

Tom Hollingsworth:

get busy doing something else.

Tom Hollingsworth:

It's like developers, they're like, I'm gonna load everything I can possibly

Tom Hollingsworth:

think of in the memory so that I know the library that I need is there.

Tom Hollingsworth:

And then you wonder why your, your, uh, application is consuming

Tom Hollingsworth:

like three terabytes of ram.

Tom Hollingsworth:

It's like, uh, maybe you need to pa pair back a little bit on that.

W. Curtis Preston:

Yeah.

W. Curtis Preston:

So it, it sounds like these tools are there.

W. Curtis Preston:

Uh, I think a lot of people do use them, but you talked about, like in the very

W. Curtis Preston:

beginning, you, you said that people's heads are gonna start spinning or

W. Curtis Preston:

whatever, because there is a lot of work involved in implementing these things.

W. Curtis Preston:

And the moment you flip that switch from, you know, per, you know, from

W. Curtis Preston:

everything is permitted to only the things that are permitted or permitted, uh,

W. Curtis Preston:

you're gonna get 5,000 tickets, right?

W. Curtis Preston:

I can't do this and I can't do that.

W. Curtis Preston:

And they, they see that.

W. Curtis Preston:

They see that very real worry.

W. Curtis Preston:

Uh, and I, and I think it stops many people from implementing this

W. Curtis Preston:

because they just see it as the amount of work they're gonna have

W. Curtis Preston:

to do to initially implement it.

W. Curtis Preston:

Um, they're, and they're not seeing the risk of what's gonna happen when

W. Curtis Preston:

they get a ransomware infection, and then it just goes crazy.

Tom Hollingsworth:

Most tools that are, are set up like this.

Tom Hollingsworth:

Uh, they have a learning mode where they will, you could put 'em in place and

Tom Hollingsworth:

they just sit there and they watch for at

Tom Hollingsworth:

least the first, you know, week or two.

Tom Hollingsworth:

And they're mapping out all of these application dependencies.

Tom Hollingsworth:

So, you know, the backup system needs to receive traffic on this port for

Tom Hollingsworth:

this application from this subnet.

Tom Hollingsworth:

And then it allows you to carefully craft that rule so that only devices

Tom Hollingsworth:

from this subnet can talk to that server on these ports and nothing else.

Tom Hollingsworth:

And if you let the tool go long enough, you'll be able to like, suss

Tom Hollingsworth:

out exactly what you need to know.

Tom Hollingsworth:

But yeah, that first day you click the switch to from, you know, allow list to

Tom Hollingsworth:

deny a list is just like you're, you're staring at the ticket queue because

Tom Hollingsworth:

you're like, oh, what happens if I, if this machine hadn't been turned on for a

Tom Hollingsworth:

week or what, you know?

W. Curtis Preston:

yeah.

Tom Hollingsworth:

Yeah.

Tom Hollingsworth:

It just, it, it, it is, it's maddening because you're always gonna wonder if you

Tom Hollingsworth:

didn't get the right stuff, but like you said, would you rather be worried about

Tom Hollingsworth:

one machine that can't talk to another?

Tom Hollingsworth:

Or would you be worrying about the fact that you're getting a phone call from

Tom Hollingsworth:

the CIO saying, uh, yeah, the database has just got encrypted by this new flavor

Tom Hollingsworth:

of malware that we haven't seen yet.

Tom Hollingsworth:

Uh, why did that?

W. Curtis Preston:

Yeah.

W. Curtis Preston:

Another thing I want to ask you, I wanna sort of move forward into

W. Curtis Preston:

the, the ransomware part here.

W. Curtis Preston:

Although Prasanna, I'm so glad you basically told us to go backwards.

W. Curtis Preston:

You always, you're really good at that, you know, you're really good at

W. Curtis Preston:

making me go backwards.

W. Curtis Preston:

Uh, anyway, uh, I wanted, so one of the things, so we talked about

W. Curtis Preston:

trying to limit lateral movement.

W. Curtis Preston:

Another thing that was suggested was to not permit, uh, new, new either

W. Curtis Preston:

new domains, like domains that just recently were created, or domains

W. Curtis Preston:

that w got recently active, right.

W. Curtis Preston:

From a DNS perspective, is that, is that still fall under the networking purview?

W. Curtis Preston:

Um, or is that like, is that another world?

Tom Hollingsworth:

It, it tend, anything that involves names and not

Tom Hollingsworth:

numbers tends to float up towards the application team or the security

Tom Hollingsworth:

team.

Tom Hollingsworth:

Uh, and the reason for that is because, like you said, like one of the things

Tom Hollingsworth:

that, that we see a lot in security now is it's this idea that you wanna black hole

Tom Hollingsworth:

things that are, that are relatively new.

Tom Hollingsworth:

Like why is this machine suddenly starting to communicate over a d n

Tom Hollingsworth:

s name that I've never seen before?

Tom Hollingsworth:

But it also

Tom Hollingsworth:

requires that your devices have the intelligence to be able to resolve that

Tom Hollingsworth:

because, you know, application layer firewalls will see, oh, you are trying to

Tom Hollingsworth:

access this service that I don't recognize on a domain that I've never seen before.

Tom Hollingsworth:

Whereas a, a lower level, almost like a packet filtering firewall will say,

Tom Hollingsworth:

oh, well that's an IP address connection on this port from here to there.

Tom Hollingsworth:

Uh, I don't see a reason why I shouldn't be using that.

Tom Hollingsworth:

And so, You, You, kind of have to integrate those two things together

Tom Hollingsworth:

because like you said, you know, something doesn't look right here

Tom Hollingsworth:

because why would it be contacting a brand new DNS name that it should, it

Tom Hollingsworth:

has no reason to contact or worse yet?

Tom Hollingsworth:

Uh uh.

Tom Hollingsworth:

You can ask the people over at SolarWinds.

Tom Hollingsworth:

Why is this DLL suddenly talking to .ru addresses?

W. Curtis Preston:

right?

W. Curtis Preston:

Yeah.

W. Curtis Preston:

Well, when he says new domain names, he actually means domain names that were

W. Curtis Preston:

like recently registered, not just domain names that are new to your network.

W. Curtis Preston:

And then also ones that, that were, they were registered but they had, they hadn't

W. Curtis Preston:

been active or something like that.

W. Curtis Preston:

So that sounds like that's a d n s uh, you know, there's a d I world, right?

W. Curtis Preston:

Um, we had, we had somebody on from that.

W. Curtis Preston:

I think we need to have some, because this is, I think that's, , if you can

W. Curtis Preston:

reasonably do that, where you could basically push a button, just sort of like

W. Curtis Preston:

the, the, the deny the allowed deny thing.

W. Curtis Preston:

If you can reasonably say, I, I don't want, I don't want anybody

W. Curtis Preston:

talking to domain names that were registered 24 hours ago.

W. Curtis Preston:

Right.

W. Curtis Preston:

I, I If you could, if you could do something like that, it will of course

W. Curtis Preston:

also create some trouble, uh, tickets.

W. Curtis Preston:

But I'm thinking far less.

W. Curtis Preston:

And if you could do that, it stops to command and control, uh, you

W. Curtis Preston:

know, the, the ransomware from reaching out at command and control,

W. Curtis Preston:

um,

Tom Hollingsworth:

slows the

Tom Hollingsworth:

process down.

Tom Hollingsworth:

But the one thing I will say there though, is that you need to make sure

Tom Hollingsworth:

that your users are expecting that change.

Tom Hollingsworth:

Because if it requires you to go out and check a list or, uh, get some kind of

Tom Hollingsworth:

una authorization to go to this domain name, even if it adds one second to the

Tom Hollingsworth:

resolution time, that's one extra second that people are going to complain about

Tom Hollingsworth:

and you know who they're gonna complain.

Tom Hollingsworth:

the networking team, because the network isn't working.

Tom Hollingsworth:

Not the d n s block list checker or the application that has this built into it.

Tom Hollingsworth:

Oh, no, no.

Tom Hollingsworth:

It's the network's fault because the packets aren't

Tom Hollingsworth:

going where they're supposed to.

W. Curtis Preston:

As we used to say back, back when I was, you know,

W. Curtis Preston:

when I first said that we, we would say the problem's under the floor.

W. Curtis Preston:

Right?

W. Curtis Preston:

Uh, meaning, meaning it was a networking problem.

W. Curtis Preston:

Um, go ahead, Prasanna.

Prasanna Malaiyandi:

So moving on.

Prasanna Malaiyandi:

So we talked about how to prevent lateral movement, how to detect these, uh,

Prasanna Malaiyandi:

rogue, uh, servers that are coming up.

Prasanna Malaiyandi:

One thing I wanted to ask is, so say you do get hit by ransomware, right?

Prasanna Malaiyandi:

They're able to move laterally.

Prasanna Malaiyandi:

What happens next from a networking perspective?

Prasanna Malaiyandi:

Well, I guess two questions.

Prasanna Malaiyandi:

One is how do you, how would you go about bringing down your network or sort

Prasanna Malaiyandi:

of isolating what needs to be isolated?

Prasanna Malaiyandi:

Like how do you actually figure out what's going on in your network?

Prasanna Malaiyandi:

And then the second question is, okay, now that you've sort of

Prasanna Malaiyandi:

identified that, how do you slowly recover from those situations?

Tom Hollingsworth:

Incident response is never fun because

Tom Hollingsworth:

it's a whole lot of cleanup.

Tom Hollingsworth:

And, uh, and, and the first thing you have to do is you have to,

Tom Hollingsworth:

you have to get people out of your network because there's, you know,

Tom Hollingsworth:

there's obviously, there's the tools that kind of run on their own.

Tom Hollingsworth:

And there are tools that kind of have to be piloted by people.

Tom Hollingsworth:

So you have to create, uh, limits on the, on the system to be able to stop that.

Tom Hollingsworth:

And fingers crossed that you're not in a situation where your entire

Tom Hollingsworth:

network has been taken down by whatever is causing the problem.

Tom Hollingsworth:

Because I've seen that before too, where not only does it try to laterally move to

Tom Hollingsworth:

infect systems, it also throws up enough extra garbage that you are, it's Inca,

Tom Hollingsworth:

you're capable of logging into any of your

Tom Hollingsworth:

management networks.

Tom Hollingsworth:

So we're lesson number one.

Tom Hollingsworth:

Make sure all your management networks are kind of isolated so that you

Tom Hollingsworth:

always have the ability to use those.

Tom Hollingsworth:

But the first thing that I would.

Tom Hollingsworth:

As I would cut off outside access immediately, I would

Tom Hollingsworth:

lock the firewall in place.

Tom Hollingsworth:

I mean, you don't have to like run through the data center screaming with

Tom Hollingsworth:

your hair on fire and start yanking cables out like the alias episode.

Tom Hollingsworth:

But you need to be able to lock all of those connections down.

Tom Hollingsworth:

And specifically you need to look for ones that, you know, could be like, you know,

Tom Hollingsworth:

from really weird external addresses, or worse yet ones that are coming in.

Tom Hollingsworth:

Once you've blocked that external access in and out, you gotta do it

Tom Hollingsworth:

in both directions because obviously you don't want anything getting out

Tom Hollingsworth:

because the two things that I can think of are command and control traffic.

Tom Hollingsworth:

If some kind of tool that's being, uh, um, orchestrated or data exfiltration

Tom Hollingsworth:

and, and you're like, oh, well I can stop those file transfers.

Tom Hollingsworth:

Yeah, look up oil rig.

Tom Hollingsworth:

It was, uh, it was able to exfiltrate data through DNS queries.

Tom Hollingsworth:

Like that's the kind of crap you have to worry about.

Tom Hollingsworth:

So you've gotta lock it down.

Tom Hollingsworth:

Then you have to isolate because that's

Tom Hollingsworth:

the other thing too.

Prasanna Malaiyandi:

But, but before you move on,

W. Curtis Preston:

I stop you there?

W. Curtis Preston:

Uh, so how, how do you do that, right?

W. Curtis Preston:

Is this, is this something where you have to create.

W. Curtis Preston:

A button to press up, you know, because this sounds like a lot of little steps

W. Curtis Preston:

you probably need to do to do this manually, or is there something I can

W. Curtis Preston:

do upfront that says, in the event of a ransomware attack, push this button.

W. Curtis Preston:

Hey, gum.

W. Curtis Preston:

Shut up.

W. Curtis Preston:

Anyway, uh, in the event of a ransomware attack, press this button and it

W. Curtis Preston:

does the 10 things I need to do.

W. Curtis Preston:

Uh, what, what do you think

Tom Hollingsworth:

Some of them do have a big red button press here to, to like

Tom Hollingsworth:

terminate all firewall connections.

Tom Hollingsworth:

But most of the time you're gonna have to create like a checklist or, or have

Tom Hollingsworth:

a system of like, okay, I'm gonna go into these rules and I'm gonna uncheck

Tom Hollingsworth:

these five boxes and then I'm gonna hit the terminate connections button to make

Tom Hollingsworth:

sure that no new connections can be made.

Tom Hollingsworth:

Also, if you have a rule at the bottom of your firewall list that

Tom Hollingsworth:

says Permit ip, any, any, take it out

Tom Hollingsworth:

now because it's not doing you any good.

Tom Hollingsworth:

But, but more importantly, you, you have to, you know, uh, all

Tom Hollingsworth:

kill switches have to be wired.

Tom Hollingsworth:

, there's no such thing as a magical switch that you can just hit, even if it's one

Tom Hollingsworth:

that the, that the provider has given you investigate what it actually does.

Tom Hollingsworth:

Does it dump the rules completely?

Tom Hollingsworth:

Does it just like suspend the rules until you go in and manually add them?

Tom Hollingsworth:

Remember that that could also cut off your connection to the firewall, so

Tom Hollingsworth:

you need to have another way to get into it just in case that happens.

Tom Hollingsworth:

Another reason for an isolated management network, but the, the idea is, is that

Tom Hollingsworth:

you, you, you need to investigate what your options are because God help you

Tom Hollingsworth:

if you really do have to run down to the data center and yank the cables

Tom Hollingsworth:

out, and if that is a case and, and hey, it's just as valid as anything else.

Tom Hollingsworth:

Can you make sure that you have the right keys, that you know which

Tom Hollingsworth:

firewall you're yanking out of?

Tom Hollingsworth:

Are there any other exits off of your network?

Tom Hollingsworth:

Because that's another problem that you may run into.

Tom Hollingsworth:

What happens if someone has created another exit off of your network,

Tom Hollingsworth:

either accidentally or on purpose?

Tom Hollingsworth:

And what happens then?

Tom Hollingsworth:

Because you know it's just as easy for me to plug something into your network.

Tom Hollingsworth:

And if there's another way off of it, I'm gonna find it.

Prasanna Malaiyandi:

Yeah.

Prasanna Malaiyandi:

The one other thing though, I know you talked about, and it totally

Prasanna Malaiyandi:

makes sense to kill all incoming and outcoming traffic, but just thinking a

Prasanna Malaiyandi:

step forward, like when you're dealing with incident response, like doesn't

Prasanna Malaiyandi:

that also take out like your chat channels, your slack channels, your

Prasanna Malaiyandi:

video conferencing, everything else, like what do you do at that point?

Prasanna Malaiyandi:

Is it just hope you have everyone's cell phone numbers?

Tom Hollingsworth:

you need to have a plan for out of band incident

Tom Hollingsworth:

response because y it's, it, it's just like any crime scene.

Tom Hollingsworth:

I need to figure out what's, what's been hit and I need to figure out

Tom Hollingsworth:

how much of it is going to spread.

Tom Hollingsworth:

And you're thinking to yourself like, I can't shut my network down

Tom Hollingsworth:

permanently because you know it's gonna cost me X amount of dollars.

Tom Hollingsworth:

Yes, but it's also gonna cost you x plus whatever amount of

Tom Hollingsworth:

dollars when the next system gets

Tom Hollingsworth:

hit, when it uncovers a device that no, nobody's patched it in years.

Tom Hollingsworth:

Um, I'm not gonna lie.

Tom Hollingsworth:

Incident response can work over iMessage text threads for a good couple of

Tom Hollingsworth:

hours while you try to figure that out.

Tom Hollingsworth:

Or, you know, buy your incident response team like those little, you

Tom Hollingsworth:

know, hotspots or enable the data plans on their phone so that they can join

Tom Hollingsworth:

their laptop there and join a Slack instance outside of your network.

Tom Hollingsworth:

because that way nothing is working internal to your network.

Tom Hollingsworth:

Because that's the other thing too.

Tom Hollingsworth:

If you, if this is something that's particularly insidious on a window

Tom Hollingsworth:

system and your incident responders are using Windows systems and they join

Tom Hollingsworth:

the network to be able to do incident response and their laptops get compromised

Tom Hollingsworth:

because they join the network again, you're gonna feel really, really dumb.

Tom Hollingsworth:

It's like, uh, the professional, when they blew up the bomb squad truck, it's like,

Tom Hollingsworth:

come on guys, what were you expecting?

W. Curtis Preston:

You just reminded me of the, there's a, there's a series

W. Curtis Preston:

of commercials and there's one where the commercial is like, it's like a

W. Curtis Preston:

horror movie and the, there's a bunch of kid, it's like the, you know, I got

W. Curtis Preston:

the guy with the, the, the ax murderers looking for the group of kids, and

W. Curtis Preston:

they're like, why don't we go hang out?

W. Curtis Preston:

Why don't we go hide in that shed over there with all the, uh, with all

W. Curtis Preston:

the, uh, machetes or something like

Tom Hollingsworth:

Yeah.

W. Curtis Preston:

Um, so, so we talked about blocking external traffic.

W. Curtis Preston:

What about blocking internal traffic?

W. Curtis Preston:

You know, uh, basically the lateral traffic, uh, be due to the, we

W. Curtis Preston:

know we have ransomware and we know it's gonna try to crawl.

W. Curtis Preston:

What about blocking that, uh, access?

Tom Hollingsworth:

So that's where you hope that your management

Tom Hollingsworth:

networks are, um, isolated because the first thing I would do going

Tom Hollingsworth:

into a router is shut down the route.

Tom Hollingsworth:

Tables prevent, um, traffic from being passed across network boundaries.

Tom Hollingsworth:

Um, what you're effectively doing in there is you are

Tom Hollingsworth:

containing the damage to one area.

Tom Hollingsworth:

Now, yeah, you're gonna take things down, but if you can isolate that network as

Tom Hollingsworth:

the location for wherever the problem is, you can then bring other networks

Tom Hollingsworth:

back online and be relatively certain that they're not gonna be infected.

Tom Hollingsworth:

I really hope that you're not using like, just regular routing, that you

Tom Hollingsworth:

have some kind of a security boundary there, because that makes it a whole lot.

Tom Hollingsworth:

But you, you've got to think in, in phases.

Tom Hollingsworth:

Obviously, you know, using the kill switch is gonna take everything

Tom Hollingsworth:

down, but then you have to start, you know, can I bring this back online?

Tom Hollingsworth:

Is this going to be infected?

Tom Hollingsworth:

What would I be looking for?

Tom Hollingsworth:

Um, so I actually have a, a story about this, uh, this happened

Tom Hollingsworth:

last year to my children.

Tom Hollingsworth:

Uh, one of 'em goes to the public high school here, uh, and I got a rocket

Tom Hollingsworth:

text message from their IT department saying, please turn off all public school

Tom Hollingsworth:

issue devices until further notice.

Tom Hollingsworth:

And I'm like, uhoh, somebody got hit with something fun.

Tom Hollingsworth:

And this was like the last day before Christmas break or something.

Tom Hollingsworth:

So we went in and we turned off my kid's MacBook, right?

Tom Hollingsworth:

So now, immediately I, because I know what the, the thing was, I don't

Tom Hollingsworth:

want anybody to like phone home and get infected and then like infect

Tom Hollingsworth:

the parents networks or whatever.

Tom Hollingsworth:

Okay, no problem.

Tom Hollingsworth:

We just shut it off.

Tom Hollingsworth:

But then I'm like, I wonder what it could.

Tom Hollingsworth:

like I, I'm kind of curious and, and they've, to this day, they've never

Tom Hollingsworth:

disclosed what it was, but you would get an email like the next week,

Tom Hollingsworth:

oh, if you're using like a, a, a, a corporate phone or if you're using

Tom Hollingsworth:

a MacBook, you can turn it back on.

Tom Hollingsworth:

Well, that automatically kind of lowers the horizon of, it has to

Tom Hollingsworth:

be something that's focused on Windows or something like that.

Tom Hollingsworth:

So then you start running through your head of what it could possibly be.

Tom Hollingsworth:

Well, an incident response, you have to do the same thing.

Tom Hollingsworth:

What server got hit?

Tom Hollingsworth:

Oh, well, it was the database server and it was running this version of,

Tom Hollingsworth:

uh, you know, windows or SQL server.

Tom Hollingsworth:

Okay.

Tom Hollingsworth:

Does that mean that Max can get on the network?

Tom Hollingsworth:

Do I want them on the network?

Tom Hollingsworth:

Is it a situation where even though they can't be infected, they could

Tom Hollingsworth:

propagate something to another location?

Tom Hollingsworth:

Like there's a lot that you have to go into because obviously the executives are

Tom Hollingsworth:

gonna be like, when can we do back up and.

Tom Hollingsworth:

and if you're a publicly traded company, oh God, the stockholders are like outdoors

Tom Hollingsworth:

with pitchforks and torches and they wanna know when they can get their dividends.

Tom Hollingsworth:

And you're like, uh, when I figure out how much of this data got encrypted

Tom Hollingsworth:

or stolen, and you're always gonna be fighting that tension and you can't

Tom Hollingsworth:

just shut everything off forever.

Tom Hollingsworth:

So that's part of incident response is you've got one team working on figuring

Tom Hollingsworth:

out how to stop whatever infected you, but you've got another team figuring

Tom Hollingsworth:

out how to bring things back online.

Tom Hollingsworth:

That's why we call it business continuity now.

Tom Hollingsworth:

Right.

Prasanna Malaiyandi:

It is interesting about the incident response.

Prasanna Malaiyandi:

How have you seen cases?

Prasanna Malaiyandi:

Like how do you actually, well, two questions I have.

Prasanna Malaiyandi:

How do you figure out like that, this segment, going back to what

Prasanna Malaiyandi:

you said, you kill all the routes.

Prasanna Malaiyandi:

How do you figure out that this segment is safe or not?

Prasanna Malaiyandi:

And then I guess that, yeah, that's actually only one question.

Tom Hollingsworth:

Well, so typically what, and, and you're, you're

Tom Hollingsworth:

effectively, when you create these boundaries, it's, it's like looking

Tom Hollingsworth:

for the hot potato effectively, because unless you, like in the alias episode,

Tom Hollingsworth:

just go click all the switches off.

Tom Hollingsworth:

Those devices can still communicate to each other at layer two.

Tom Hollingsworth:

Now, where you don't wanna have a problem is, is that it's in the data.

Tom Hollingsworth:

because if you isolate the layer two data center, now you've got a real problem.

Tom Hollingsworth:

Because if those servers, if if it's looking for servers, those

Tom Hollingsworth:

servers can still get infected.

Tom Hollingsworth:

That's why it's actually better to have like a, you know, a host route or

Tom Hollingsworth:

something like that, or something that, that kind of isolates that per unit thing.

Tom Hollingsworth:

I mean, honestly, like a V switch is perfect for this because like,

Tom Hollingsworth:

if it's not bound for that host, I'm not gonna let it go any further.

Tom Hollingsworth:

But effectively what you have to do is you have to look for chatter

Tom Hollingsworth:

that's still going on in the network.

Tom Hollingsworth:

Like you, you, I've shut all this down.

Tom Hollingsworth:

and I told my users to like disable their machines or, or turn them off or

Tom Hollingsworth:

whatever, what's still trying to talk.

Tom Hollingsworth:

And then you go take that on a case by case basis.

Tom Hollingsworth:

Oh, this device is still sending traffic that it's, but

Tom Hollingsworth:

it's looking for this server.

Tom Hollingsworth:

Okay, well I'm, I, I can shut it off because I know that it's probably safe.

Tom Hollingsworth:

But then you run into something like, oh, this thing is chattering

Tom Hollingsworth:

an awful lot and it's chattering on a way that it shouldn't be chattering.

Tom Hollingsworth:

Like that's how I've gone and found hosts that have been infected, but not

Tom Hollingsworth:

by ransomware, but by early malware because they just kept hammering the

Tom Hollingsworth:

firewall with these outbound requests.

Tom Hollingsworth:

And I'm like, you shouldn't

Tom Hollingsworth:

be doing that.

Tom Hollingsworth:

So it's, it's almost like a little bit of detective work.

Tom Hollingsworth:

The good news is, is that even though the network devices are kind of like

Tom Hollingsworth:

dumb from the perspective of I don't care what application is trying to talk,

Tom Hollingsworth:

where they're really good at telling you that things are still generating traffic.

Tom Hollingsworth:

It's like, oh, this port is still sending a ton of packets f bound

Tom Hollingsworth:

for this address on this location.

Tom Hollingsworth:

And so then you're like, oh, I think something might be up here.

Prasanna Malaiyandi:

Do you ever see cases where people.

Prasanna Malaiyandi:

, almost do a, like, create a black hole on the device itself to sort

Prasanna Malaiyandi:

of sync the packets there so it doesn't go out, rather than having

Prasanna Malaiyandi:

to necessarily do it on the switch.

Tom Hollingsworth:

Um, you can, uh, that's actually a really great way to

Tom Hollingsworth:

determine what it's trying to contact is to create like a null route on the system.

Tom Hollingsworth:

Uh, uh, going all the way back like three or four years.

Tom Hollingsworth:

Like Mark Marcus Hutchins, that's how he actually stopped a major outbreak of

Tom Hollingsworth:

malware, uh, for all the good it did, and he got arrested by the FBI later.

Tom Hollingsworth:

But he basically black hole the dns.

Tom Hollingsworth:

He bought the domain black hole it because if that domain name was

Tom Hollingsworth:

active, then it would stop propagating.

Tom Hollingsworth:

And so he figured that out by saying, oh, I wonder where this is

Tom Hollingsworth:

going and I wonder what it's doing.

Tom Hollingsworth:

You can do that.

Tom Hollingsworth:

And it's actually the next step in incident response, which you've isolated

Tom Hollingsworth:

the system, is I wanna see how it behaves and what it's trying to do.

Tom Hollingsworth:

Cuz that could give me a clue as to what I got hit with and

Tom Hollingsworth:

what they could be looking for.

Tom Hollingsworth:

And that gives you, you know, a, a little bit of opportunity, but that's

Tom Hollingsworth:

a little bit more of an advanced tool that you would, you would want to use.

Tom Hollingsworth:

Uh, just because black holding traffic on a, on a device takes

Tom Hollingsworth:

a little bit of setup, especially if you're fighting against people

Tom Hollingsworth:

who don't want you to do that.

Prasanna Malaiyandi:

Yeah.

W. Curtis Preston:

Speaker:

Yeah, so it sounds like.

W. Curtis Preston:

Speaker:

A, a lot of the things that you talked about in the last couple of minutes, they

W. Curtis Preston:

Speaker:

would be a lot easier to do again, if we segmented the network in the first place,

W. Curtis Preston:

Speaker:

right?

W. Curtis Preston:

Speaker:

We put people with Windows laptops on one network.

W. Curtis Preston:

Speaker:

We put people with Mac laptops on a network, another network.

W. Curtis Preston:

Speaker:

We put the, the, the phones right?

W. Curtis Preston:

Speaker:

That are doing the wifi.

W. Curtis Preston:

Speaker:

We put them on another network.

W. Curtis Preston:

Speaker:

Um, and we put servers on a different network.

W. Curtis Preston:

Speaker:

We put, maybe we put servers of a different type on, on a different network.

W. Curtis Preston:

Speaker:

So that way you could basically say you don't have to tell the,

W. Curtis Preston:

Speaker:

the, the users to not do anything.

W. Curtis Preston:

Speaker:

You can just say shut off the, the laptop, uh, network.

W. Curtis Preston:

Speaker:

Right?

W. Curtis Preston:

Speaker:

Um, and you, you shut off the laptop network and so on.

W. Curtis Preston:

Speaker:

And, and all the networks that where we don't currently,

W. Curtis Preston:

Speaker:

what we're not looking at.

W. Curtis Preston:

Speaker:

And then, okay, who's trying to talk?

W. Curtis Preston:

Speaker:

Who's trying to talk?

W. Curtis Preston:

Speaker:

Why is this server surfing?

W. Curtis Preston:

Speaker:

The web

Tom Hollingsworth:

Yeah.

W. Curtis Preston:

Speaker:

There's nobody over there.

W. Curtis Preston:

Speaker:

Why is this server going over report 80?

Tom Hollingsworth:

Well, a lot of places already kind of have this by

Tom Hollingsworth:

default, even if they didn't realize they were doing it because you have

Tom Hollingsworth:

different classes of devices that you wanna treat them differently.

Tom Hollingsworth:

Like for example, the uh, um, the server network, we want to have a

Tom Hollingsworth:

little bit more security in there.

Tom Hollingsworth:

Maybe a little less host to host East to west traffic kind of thing.

Tom Hollingsworth:

The wireless network where all the laptops and the devices connect.

Tom Hollingsworth:

I'm a little less careful about that because I actually have identity

Tom Hollingsworth:

management in place that validates the users when they try to log in.

Tom Hollingsworth:

Maybe I have a guest wireless network for my, for people that come into the lobby.

Tom Hollingsworth:

That one's wide open to the internet outbound only.

Tom Hollingsworth:

So I don't need to worry about that quite as much.

Tom Hollingsworth:

And then, you know, like phones and printers and things like that, that

Tom Hollingsworth:

have very specific things like, you know, I wouldn't enable Bonura in my

Tom Hollingsworth:

internal network, but maybe for the printer vlan I would, because I want

Tom Hollingsworth:

people to be able to find a printer.

Tom Hollingsworth:

Open up their laptop.

Tom Hollingsworth:

So they've already created these segments.

Tom Hollingsworth:

You just have to know where the buttons are to shut them off.

Tom Hollingsworth:

So maybe the example is I wanna isolate the servers from the rest

Tom Hollingsworth:

of the network, cuz I think there's something in there, but I can still

Tom Hollingsworth:

leave the wireless network up.

Tom Hollingsworth:

Maybe have everybody join the guest access network and force them all

Tom Hollingsworth:

out to the internet to do, you know, incident response or chat channels

Tom Hollingsworth:

or something like that where I'm, you know, but I'm creating these bounds so

Tom Hollingsworth:

that traffic flows one direction only, or it prevents certain things inside

Tom Hollingsworth:

of other areas because, you know, there's nothing to say like the, you

Tom Hollingsworth:

know, the, the, uh, s IDs that are on printers that are like, you know, set up,

Tom Hollingsworth:

uh, set me up or something like that can't be compromised.

Tom Hollingsworth:

And then if they can get into your printer network, it's like,

Tom Hollingsworth:

oh crap, where can they go from?

W. Curtis Preston:

Yeah.

W. Curtis Preston:

And, and Bonjour of course would be the, um, I, I don't know how would

W. Curtis Preston:

I define

Prasanna Malaiyandi:

file sharing.

Tom Hollingsworth:

It, it is, it's almost like an auto configuration announcement,

Tom Hollingsworth:

uh, setting where, uh, it, it, and you can thank Steve Jobs for this.

Tom Hollingsworth:

He's like, I hate setting up printers.

Tom Hollingsworth:

And so basically what he did is he set up a system so that the printers

Tom Hollingsworth:

can announce that they exist.

Tom Hollingsworth:

And your laptop is constantly listening for these.

Tom Hollingsworth:

Bonura is another one of those protocols that is extra chatty and you kinda

Tom Hollingsworth:

wanna put bounds on it so that like you don't have the Apple TV four hallways

Tom Hollingsworth:

down announcing itself to the people in accounting because one, it's annoying.

Tom Hollingsworth:

And two, you never know when you're gonna do something you're not supposed to.

Prasanna Malaiyandi:

Interesting.

Prasanna Malaiyandi:

So yeah, I guess a lot of these are really around setting up

Prasanna Malaiyandi:

that initial network properly.

Prasanna Malaiyandi:

So then when you do have these issues, you can recover quickly and

Prasanna Malaiyandi:

identify and then recover quickly.

Prasanna Malaiyandi:

Right?

Prasanna Malaiyandi:

But if you don't have that initial setup done, then you're

Prasanna Malaiyandi:

in for a world of hurt, I guess.

Tom Hollingsworth:

and not just initial setup.

Tom Hollingsworth:

You actually do have to treat the network like a living, breathing organism.

Tom Hollingsworth:

I can't think of a single server admin out there that installs,

Tom Hollingsworth:

you know, windows server.

Tom Hollingsworth:

What are we up now?

Tom Hollingsworth:

20 20, 20 23 Windows, server X, I don't know, installs it

Tom Hollingsworth:

and then never patches it.

Tom Hollingsworth:

Never

Tom Hollingsworth:

touches it again.

Tom Hollingsworth:

Like, like you people are probably just shaking, even thinking.

Tom Hollingsworth:

, you cannot configure a network and then just leave it alone.

Tom Hollingsworth:

You do have to go in and, and tweak things and move things and change things.

Tom Hollingsworth:

And, you know, not just when you're trying to fix a broken thing,

Tom Hollingsworth:

either, you have to like, okay, is this subnet big enough for the

Tom Hollingsworth:

number of hosts that are in it?

Tom Hollingsworth:

Should I create routes over here?

Tom Hollingsworth:

It looks like there's a lot of extra traffic going on over this direction.

Tom Hollingsworth:

Maybe I need to disallow that because it looks like it's something

Tom Hollingsworth:

that shouldn't be happening.

Tom Hollingsworth:

Like, if you're not constantly pruning back what you are working on then,

Tom Hollingsworth:

and that's the problem that a lot of the, the, uh, ransomware writers have

Tom Hollingsworth:

figured out, like a lot of, a lot of their secrets, if you wanna call them,

Tom Hollingsworth:

that are just inadequate it support.

Tom Hollingsworth:

Like, we're gonna hope that you had left this on by default and we're

Tom Hollingsworth:

gonna take advantage of it and use it.

Tom Hollingsworth:

And if you did, I'm sorry, but like, you know, if any best practices

Tom Hollingsworth:

guide out there says, shut that off, and you didn't shut it off,

Tom Hollingsworth:

are you in that big of a hurry?

W. Curtis Preston:

Yeah, well we're, we're living in a world

W. Curtis Preston:

where, uh, you know, people don't even change their default password.

W. Curtis Preston:

So, um, listen, here's the thing, Tom, my plumber's here, so, uh, I, I, you

W. Curtis Preston:

know, I got a tradesman that actually showed up at two o'clock when he said

W. Curtis Preston:

he was gonna be here at two o'clock.

W. Curtis Preston:

So I gotta , we gotta shut this baby down.

W. Curtis Preston:

Uh, Tom, this has been, this has been a great conversation.

W. Curtis Preston:

Um, so thanks, thanks a lot.

Tom Hollingsworth:

Well, thanks for having me.

Tom Hollingsworth:

It's, it's been fun to talk about networking with, uh, with some folks

Tom Hollingsworth:

that coming at it from a slightly different perspective and understanding,

Tom Hollingsworth:

you know, what are we trying to accomplish with it, and in some

Tom Hollingsworth:

cases, what are we trying to disallow?

Prasanna Malaiyandi:

Hmm,

W. Curtis Preston:

Absolutely.

W. Curtis Preston:

Thanks again, Prasanna, once again, making me go backwards,

Prasanna Malaiyandi:

I, you know me, I try, you take one step back, two steps

Prasanna Malaiyandi:

forward or something like that, right?

W. Curtis Preston:

something like that.

W. Curtis Preston:

I

W. Curtis Preston:

like that.

W. Curtis Preston:

All right.

W. Curtis Preston:

And thanks again to our listeners.

W. Curtis Preston:

Remember to subscribe so that you can restore it all.