Last year LastPass suffered two hacks that left their customer’s data exposed. What can you learn from this event, even if you’re not a LastPass customer? We use this hack as an example of what your company should do (or not do) if it ever suffers such a hack. We also talk about password managers, and what this hack means to those who use them. You do use one, right? This is a great episode, chock full of information. We hope you enjoy it.
Transcript
[00:00:40] W. Curtis Preston: Hi, and welcome to Backup Central’s. Restore it All. Podcast. I’m your host, w Curtis Preston, a a Mr. Backup. And I have with me my peripheral, uh, uh, my, my, my peripheral proliferation consultant. Prasanna Malaiyandi. How’s it going? Prasanna,
[00:01:00] Prasanna Malaiyandi: Curtis. It’s one of those things that I don’t think anyone’s gonna ever solve, and everyone talks about everything going wireless, but at the same time, like there’s certain things, you just have to plug it.
[00:01:13] W. Curtis Preston: I have a, I have a wireless mouse, but it’s plugged, but it has a dangle that has to be plugged in, which makes it, which makes it, puts it on my list of, right. Um, and, uh, I, I, the thing is, I ha you know, it’s the whole USB-C thing. USB-C, right? That’s
[00:01:30] Prasanna Malaiyandi: yep.
[00:01:31] W. Curtis Preston: Um,
[00:01:32] Prasanna Malaiyandi: a new standard.
[00:01:32] W. Curtis Preston: because I have a, I have a MacBook Pro that has four USB-C ports, which are completely worthless to 95% of
[00:01:41] Prasanna Malaiyandi: Of the devices. Yep.
[00:01:43] W. Curtis Preston: You gotta Right, because we’ve had so many years. I mean, even new stuff that you buy now, it generally comes with, you know, unless it needs a u USB-C
[00:01:52] Prasanna Malaiyandi: You know what you should do?
[00:01:54] W. Curtis Preston: what?
[00:01:55] Prasanna Malaiyandi: They sell dongs that convert U S B A to u s bbc.
[00:02:00] W. Curtis Preston: Well, that still wouldn’t solve my problem. Right? Because I have, again, it’s the, I have too many things to plug in. That’s the problem. Right. Um, so, um, and they don’t like, well anyway, yeah, but this is, this is what you and I spend time talking about.
[00:02:17] Prasanna Malaiyandi: Yeah.
[00:02:19] W. Curtis Preston: Uh, and it’s just, you know, I mean, I’m, I’m ashamed of this cable.
[00:02:25] Prasanna Malaiyandi: I hate
[00:02:25] W. Curtis Preston: For the,
[00:02:26] Prasanna Malaiyandi: Yeah.
[00:02:27] W. Curtis Preston: for, for those of you that are, are, you know, blessed enough to be watching this on Backup Central, like this, this is my, this is my,
[00:02:39] Prasanna Malaiyandi: Oh, Curtis.
[00:02:41] W. Curtis Preston: it’s just, it, its just a mess. Right? Um, and, you know, and I, I mean, I have a cabled, you know, if you were paying attention, you would’ve seen a cabled Naked Hard Drive, which I’m currently using to download all of my, um, my io, my iCloud photos, which came up in a previous podcast.
Uh, I re I finally figured out how to get my iCloud photos out of iCloud, uh, in their full native, naked format. Although it’s H E I C format, of course,
[00:03:15] Prasanna Malaiyandi: But I think that’s what your iPhone takes anyway.
[00:03:20] W. Curtis Preston: It, it, it does. Right. Um, and, and I, I don’t know what H E IIC brings that, let’s say p and g or JPEG brings, maybe it, it’s got that auto scaling. I don’t know. I don’t know what it does, but all I know is if I want to use it in any normal program, I have to convert it to p and g or J. But I will have it at least in its full native format because I can’t support, uh, my phone doesn’t have enough storage to store the 11,000 photos that I had in.
I, I had no idea. I had 11,000
[00:03:52] Prasanna Malaiyandi: so, and I know this has always been the biggest challenge, is like once you take those pictures, it’s like, I know you and I chat about this the other day. It’s like probably like 30, 40% of that is garbage, right?
[00:04:04] W. Curtis Preston: I think it’s, I think it’s a higher percentage. I think, I think you’re being too kind, because I am the opposite of you, where I take pictures of everything. I’m like, Hey, Prasanna, look at this cool thing I just saw. Right? Like the, the one that comes to mind is when I was driving down the freeway and there was this person that had damaged their back right bumper.
And what they bought was this gigantic bandaid. Like literally, it’s just a, it’s just, it’s literally, it, it was made to look like a bandaid on their bumper. And I thought that was hilarious. And I sent, I sent that to you, but in order to do that, that had to be in my camera roll. Right. And, um, did I then immediately delete that?
No, I didn’t. Right.
[00:04:41] Prasanna Malaiyandi: Whoever deletes
[00:04:42] W. Curtis Preston: months later and I’m looking and I’m like, oh my God, I have four, four photos of this, of this car. Yeah. This is, this is the problem. This is why you and I do what we do, because there’s so much data. Most of it worthless,
but in the middle of it is stuff that really matters.
[00:05:02] Prasanna Malaiyandi: and the problem is the cost to go and prune the garbage data is so high that you’re never
[00:05:09] W. Curtis Preston: is, it is. Yeah. So I, so I started, I started going through my, my, uh, camera roll starting with like the very beginning of my camera roll. And I’m literally, I’ve, I’ve learned the, the keyboard is your friend here, right? You wanna do it on your screen because you can, I can hover my finger hovers over the command key and then I’m, I’m going back and forth between the right arrow and the delete key, right?
Cuz you have to do command delete to delete something. Then you do right arrow to go to the next picture. So I’m like, I’m like right arrow, command, delete command, delete command, delete command, delete, right arrow. Right. So it’s doable, but it is time. Right? It is time consuming. But at some point I will run out, I will hit the wall of the
[00:05:58] Prasanna Malaiyandi: you’ll give up.
[00:05:59] W. Curtis Preston: of storage that, um,
[00:06:01] Prasanna Malaiyandi: Apple
[00:06:01] W. Curtis Preston: I gotta, so it’s, it’s time, time to prune now,
[00:06:04] Prasanna Malaiyandi: Yep. And that’s the thing. So I know we talk about backing up data often. I know we talk about archiving data. I don’t think we’ve, maybe we need an episode on pruning data. You.
[00:06:19] W. Curtis Preston: It would be the most boring
[00:06:21] Prasanna Malaiyandi: I, I don’t, but I don’t. Sorry. Maybe it’s not an episode, but I think it’s something we should be talking about.
[00:06:27] W. Curtis Preston: I agree. I agree. Uh, what, what, let me, let me file that, let me file that. If there’s anybody out there that’s a pruner are, are there professional data pruners? Is there software that will somehow help me figure this out? Um, Probably not, but, um, well, let’s get to the, let’s get to the, the actual, uh, subject of the day.
And, and it’s one that we’re revisiting something that we’ve talked about before, but I think it’s, I, I mentioned that I wanted to come back and do a full summary episode on what we were going to call the LastPass Hack. Um,
LastPass
[00:07:05] Prasanna Malaiyandi: hack the conclusion.
[00:07:07] W. Curtis Preston: LastPass hack, TL tdl, T T Ld R the Tld, R uh, T L D r. And so I’m gonna listen to a, you know, 40 minute podcast. I didn’t have time to read the thousands of articles that are written about it. So I’m gonna, I’m gonna listen to Curtis and Prasanna blather on about it. Um, There’s, those are our people.
We’re, we’re glad you’re here. Uh, by the way, before we get started, I’ll throw out our, I’ll throw out our, um, our disclaimer Prasanna and I work for different companies. It’s why, it’s why we still get along, probably. Um, and, uh, I work for Druva. He works for Zoom. Uh, this is not a podcast of either company. In the opinions that you hear are ours, sometimes not even ours.
Sometimes we’re regurgitating other people’s opinions,
[00:07:58] Prasanna Malaiyandi: And sometimes we have disagreeing opinions too.
[00:08:00] W. Curtis Preston: and sometimes I have disagreeing opinions. Uh, And, um, but, um, and then also be sure to rate us on your favorite podcaster and also, uh, and follow us the, the more people that follow us, it helps. Why do we ask you to do that? The more you rate us, the more you follow us, the more that helps us, uh, especially in Apple, uh, podcast.
It helps us bubble up to the popularity, which helps other people find us, right? So that’s why we ask you to follow us. Uh, if, if you, if you’re a regular listener, then uh, it helps you find us, but it also helps other
[00:08:36] Prasanna Malaiyandi: Yeah. And, and if you have friends who you think would enjoy this podcast, please share with them as well, right? Because we try to keep this fun, entertaining, but still technical to some degree, right? About things that matter in data protection and all the other fields that we talk about. So,
[00:08:55] W. Curtis Preston: Yeah. And, um, yeah, and, and tweet us, right? Tweet, you know, uh, tag me. I should, you know, I should really get it. I need to get a Twitter handle and a email address just for the show. Uh, but then I gotta build a whole new following for that. Anyway, it was, it’s a whole thing. So I’m at WC Preston on Twitter, uh, and Prasannas, pmalaiyandi, or, uh, good luck spelling that.
And, um, you can
[00:09:21] Prasanna Malaiyandi: Tag us. Yeah, tag us. We’d love to interact.
[00:09:24] W. Curtis Preston: yeah, yeah, yeah, yeah. Um, and, uh, if, if, if Prasanna, uh, responds to your tweet, it will increase his tweet, his Twitter activity by a hundred percent.
Anyway, so let’s talk about this LastPass hack.
So this is, I mean,
[00:09:42] Prasanna Malaiyandi: Maybe before the hack, what is LastPass? Maybe that’s probably a good
[00:09:46] W. Curtis Preston: yeah, well, well yeah, so well what is LastPass and, and why are we talking about it? Well, LastPass is, was one of the leading password managers. I do not think it is anymore. I would be surprised if they managed to stay in business after this. Um, the. Um, and
[00:10:06] Prasanna Malaiyandi: it’s one of those cloud based.
[00:10:07] W. Curtis Preston: LastPass, i’d, I’d love to.
Yeah. And it is a, it is a cloud-based app, right? And we are a fan of password managers as much as we’ve had episodes where we talk about what’s wrong with password managers, um, they’re still better than not having a password manager. And not, and not everybody agrees. Well, the, the occasionally you may hear, um, somebody who says, well, I don’t wanna put all my passwords in one place because if that place is hacked, then my passwords are gone.
And we’re gonna talk about that, like how this, how this, um, figures into that. Um, but do you wanna sort of give, give a, a summary of what you know, I, I don’t know if, do we wanna summarize it as it happened or how we found out about it?
[00:10:54] Prasanna Malaiyandi: I, mm-hmm. I, good
[00:10:57] W. Curtis Preston: I think we should do as it happened first, because it’s easier. Um, right. Cuz there was, there were essentially two events. Right. You wanna summarize the first one and then I’ll summarize the second
[00:11:06] Prasanna Malaiyandi: So what happened in August, right? That first incident is there was a threat actor who had gained access to an engineer’s device, right? And so once they had access, they were able to then get into the LastPass development environment, right?
Because that’s what the engineer had. And they were able to sort of root around in there, find details, create sort of environments, and, but they were contained within the development environment. Now they were able to get access to the engineer’s laptop . So we don’t know how they got access to the engineer’s laptop, but they were able to, once they got on the laptop, they were able to access all of the internal LastPass systems from that perspective. Now, they were also pretty sophisticated.
Um, it seems that the laptop was properly configured with EDR tools, right? That kind of monitor to make sure, and those were tampered with and didn’t trigger, right? And they were also able to actually do legitimate authentication into LastPass environment fully with MFA as well, right? Which is like, it looked like the legit person was logging in, right?
And so they were doing everything they could to sort of hide what they were doing on this developer’s laptop. But it is important to note that no customer or vault data was accessed in this first incident.
[00:12:34] W. Curtis Preston: Yeah. What they did right? And what they did wrong. Right. So they did contact the security firm, right. Mandian, but they didn’t really, I don’t think they adequately communicated what happened. Right. Um, but that was just one event. And then time passed and there was the second event, which was the much, I, I think the, in terms of much more damage.
Yeah. Much scarier. So it’s, What’s interesting is that the, the, um, the, what do you call it? The tactics, techniques and procedures. The TTPs that this hacker used in the second attack made it actually seem like it was a new hacker.
Like they, they totally changed their, their, their, you know, their method of attack. And so they actually didn’t think that, um, you know, and their, their IOCs, their indicators of compromise were also different. So it,
[00:13:29] Prasanna Malaiyandi: It’s like, how would you even know? Yeah. You wouldn’t connect the two at all. Right? It’s like, oh, they’re doing something completely different and can’t be the same guys.
[00:13:37] W. Curtis Preston: But they learned at some point that it was actually the same person and this was worse. So, so first off, it was, uh, the, the threat actor got access to the employee’s home computer, right? That’s where they used this, uh, a vulnerable. Media software package. Now we don’t know if that’s one that that employee was supposed to install or if it’s something, you know, is he like me and he makes videos at home for, you know, for podcasts or something?
Uh, and, and not part of his job. And then here’s the thing is the, the hacker was able to install a key logger and that key logger watched the, um, the DevOps. It was a DevOps employee. Watch the DevOps employee type his master password. Now those of you that use password managers, you just lost your breath, right?
Cuz you’re like, holy cow, they have the master password. Um, and then, and then this is after the employee authenticated with mfa. So the, the problem is that that then gives them access to the, um, To the password vault, right to the, to the LastPass corporate vault. So this is, so lemme explain what that would mean.
So if you’re using a password manager for corporate reasons, you’ve got passwords to your servers, passwords to your services, passwords to your cloud services in this case, uh, to a cloud service that they were using to as a, as a backup. And, um, and even though, you know, all the stuff, you know, they had an EDP as well, alerting and logging was enabled.
But the, but because they used genuine authentication credentials, nothing looked malicious. Right. And then the, the result was that at some point the, the threat actor got access to the cloud-based. Storage that they were using for backup because they had this homegrown backup system that we don’t fully understand, but we know that at some point they would, they would put a copy of a bunch of things, which included the customer vault, uh, the encrypted customer vault, but put the customer vault in this, basically in, in an object up in some sort of cloud-based storage.
But this, the hacker got access to that and was able to download all of, all of this data. Right now it is important to, to state that the bulk of the data they accessed was encrypted. Unfortunately, some of the customer data was unencrypted from what we, um, you know, sussed out is passwords and things like that.
Were definitely encrypted. But other things may, and again, this is, this is us, um, you know, theorizing. So maybe they left the email address unencrypted, maybe they left things like the IP address, uh, or the website that the, that the password was unencrypted, right? So they were saying that parts of the customer data was unencrypted.
Um, and the, and, and so that’s, that’s really bad,
[00:16:53] Prasanna Malaiyandi: Yeah. That’s really, really bad.
[00:16:55] W. Curtis Preston: so, so it’s interesting, those are the two events. And if, and if we had heard about it, just the way you described and, and I described that would be one thing, but the thing that for me makes this stand out, especially the second attack, is that we found out about it in pieces.
Do you remember that?
[00:17:17] Prasanna Malaiyandi: Yeah. In December, they were like, oh, nothing happened. And then it was like, oh, by the way, someone got access to the customer vaults, but don’t worry. Everything is good. And then it was slowly like this.
It was slow releast.
[00:17:28] W. Curtis Preston: said, oh, by the way, yeah, by the third piece, the, the, the third shoe, which is weird cuz that’s a three footed person. The third shoe to drop was, that was what we just described, basically how this happened, that it was a DevOps engineer because that is, that is scary.
Why don’t you, I know you explained it on a previous podcast, but why don’t you explain what that, why, why is that a big
[00:17:49] Prasanna Malaiyandi: yeah. So I, for a lot of people who aren’t familiar, so typically how an organization like an a cloud-based company, right, would be structured is you have your developers writing code, but they’re not typically the ones who are deploying the code into a production environment, right?
Managing those, all the rest, typically that’s done by a, a DevOps person, right? And so they’re the ones
[00:18:12] W. Curtis Preston: short for,
[00:18:13] Prasanna Malaiyandi: developer and operations,
[00:18:16] W. Curtis Preston: right.
[00:18:16] Prasanna Malaiyandi: right? And so they’re the ones who basically have access to all the customer data, right? So anything that, like if I was using LastPass right, and I had my password vault stored in the cloud, the DevOps person could potentially have access to it because they have credentials and authentication and authorization to access those resources because they need to as part of their job. Now in a lot of companies, you limit who are DevOps folks, right? You make sure that everything is super secure. LastPass did the right thing in terms of having mfa, right? And a password vault and everything else, right? So it’s not like they did anything wrong from a DevOps perspective. It’s just the hacker was smart and knew exactly who to target, right?
Because they’re the ones who have the keys to the kingdom and they’re the ones who can get into everything.
[00:19:03] W. Curtis Preston: Yeah. Yeah. The, I, again, this is one of those where I, I wish I knew just a little bit more. So, so for example, when I, when I hear that they were able to, that basically, once he got the master password, Or, or she, right. We don’t know. Uh, the, the, the threat actor, once they got the master password, they were then able to access the corporate vault as the person, because it looked like the person, and they said, they mentioned, they made the mention of that was after the person, um, uh, authenticated with mfa.
Right? So I’ll just speak the, the way my password manager works. So my, I have to authenticate with MFA on each new thing that accesses my password vault. Meaning I, I use multiple profiles in, um, Chrome, right? Uh, so I use, you know, I have a, the, I’m on, I’m on my independent profile that accesses all my independent stuff that I do for, um, you know, for like the podcast, right?
Um, and my, my W Curtis Preston Gmail account. So that, that is that. Uh, profile. And then I have another profile that is my work profile. And if, and, and then I have a third, which is a, a podcast manager, right? So I have all these different profiles and each time I go between them, if it’s been a while, uh, I have to reauthenticate myself.
Right?
[00:20:34] Prasanna Malaiyandi: But, but if, but if you were stuck in the same profile and you were accessing multiple things within that profile and your password manager, it wouldn’t ask you for mfa.
[00:20:46] W. Curtis Preston: No, it wouldn’t. but let me, let me just, what I’m, what I’m trying to say is, That in order, assuming the, the threat actor had the master password, he would need to use it within the context
[00:21:02] Prasanna Malaiyandi: Of that
[00:21:02] W. Curtis Preston: that it was originally being used.
[00:21:06] Prasanna Malaiyandi: So if they had,
[00:21:08] W. Curtis Preston: understand how they would do
[00:21:10] Prasanna Malaiyandi: so if they had access to the employee’s laptop, right? They hacked it. They compromised it. If they were using that as a proxy to access everything internally in last path.
[00:21:20] W. Curtis Preston: right. But that would mean that, would they, are they literally driving Chrome, for example, just to just you, you see what I’m saying? Because for it to work in Chrome, they would have, for it to work in my environment, they would have to literally drive Chrome.
[00:21:33] Prasanna Malaiyandi: And that, I don’t know, but that’s kind of what I’m imagining is
[00:21:38] W. Curtis Preston: But then again, you and I aren’t, aren’t ethical hacker people, so, so basically he.
He or she would, was able to gain the master password and then use that master password in the same environment that it was initially typed after MFA had already been uh, entered. And so really they just look like the user using their master
[00:22:07] Prasanna Malaiyandi: Exactly. Which is why it didn’t flag in any of the alerts and auditing.
[00:22:11] W. Curtis Preston: yeah.
[00:22:12] Prasanna Malaiyandi: The one thing I would say though is they should have, so even though we talked about DevOps engineers being powerful, right? And having access to everything, they should have had alerting and monitoring turned on on those accesses, right?
Because it’s probably not likely that the DevOps person would have gone and accessed the backups in, or it should have at least flagged and alert and be like, Hey, is this normal behavior or the fact that they’re copying out a bunch of password vaults, right? That does not seem like normal behavior for a, DevOps person.
[00:22:47] W. Curtis Preston: yeah. Agreed. The thing that killed me was like, as we were talking about is that it just came out over time. It’s like a little bit here, a little bit there, a little bit here, a little bit there. And it was like, when are, and that’s why when we saw the latest one, that’s when I said, okay, I’m gonna wait a while to see if any other shoes drop.
Uh, and then, then it’s two people see, cuz it’s, it’s a three footed
[00:23:11] Prasanna Malaiyandi: Three person becomes a
[00:23:12] W. Curtis Preston: so, so, um, uh, and I, and I think that’s happened now, right? Uh, it’s been a full month. And so I think let us summarize, um, you know, things that we can learn from what happened. And the, the, the first thing I want to talk about, so this is things that we can learn from what happens if you as a company are subject to.
Some type of attack like this. And then also what can you do as a consumer of password managers to protect yourself from your password manager being, uh, uh, yeah. So the first that I want to talk about, and I really want to put it up, up in front, and that is communication. Um, uh, I want to, I want to emphasize that I’m not an official incident response person.
I’m not an official corporate communications specialist. Right. Although I do communicate quite a bit for the corporation for which I work. Right. Um, and I’m not doing that at the moment,
[00:24:23] Prasanna Malaiyandi: Yep.
[00:24:23] W. Curtis Preston: but I do sometimes speak officially on behalf of, of the company that I work for. I think that they did poorly in this case in terms of letting us know what had happened.
I think they were trying to do. It’s that classic thing of trying to do damage control, right? Um, and, and focusing more. And by damage control, I don’t mean trying to limit what was happening inside, trying to limit how bad it looked on the outside. Um, and, um, yeah, make the story sound not so bad. And I, and, and again, this is me theory theorizing.
This is just, I think it’s possible that the only way we found out about the DevOps thing is that it somehow leaked to some press. And so then they, they decided to add that in at, at, at some point. Um,
[00:25:22] Prasanna Malaiyandi: I also, so, so I go back and think about like some of the past breaches that we’ve talked about, right?
[00:25:28] W. Curtis Preston: mm-hmm.
[00:25:29] Prasanna Malaiyandi: You’re in most of the situations, you’re right, it was poor transparency by the company who had been attacked. And so everyone was in sort of a frenzy trying to be like, what’s going on? And are things safe?
It’s kind of like, um, when, right. This breach happened, right? Everyone. Across the news everywhere was like drop last, pass drop, LastPass. Because no one knew the severity of what had happened and everyone was freaked out,
[00:25:58] W. Curtis Preston: Yeah. And, and by the way, I, I don’t think we, I, I think that recommendation would’ve stayed the same whether we found out later or sooner. But I, I, I can’t think of a good example. I can
think of one company. Um, I can think of one company that what they did was they had a page, right?
Here’s our page regarding the, and I don’t want to name the company because I’m going to both say good things and bad things. They had a page where they posted stuff in terms of what we’re doing, and then all you had to do was follow that page.
[00:26:30] Prasanna Malaiyandi: yep.
[00:26:31] W. Curtis Preston: They, they started posting and then they, they did continue posting.
There were some, some long delays. The problem with that was that they didn’t really have a plan and that they, and, and what, what we saw was, oh, we’re, we’re testing the recovery environment. We’ve tested the recovery. We’re good. We’re, you know, and they were communicating, so they were doing the communication part that I think is important.
It’s just what we saw through that was that they didn’t have a plan going in, and so they developed their recovery plan in the midst of the recovery.
[00:27:04] Prasanna Malaiyandi: I, I think the one that I kind of was impressed about, if I recall correctly, was the Okta hack, right? I think Okta had done a decent job of communicating what had happened, being transparent, um, as they had information and being upfront,
[00:27:23] W. Curtis Preston: Yeah.
[00:27:24] Prasanna Malaiyandi: uh, at least that’s what I recall. Now, this was a while ago.
I wanna say it was like 16 months ago. Something
[00:27:30] W. Curtis Preston: Well, I know that no further shoes dropped, right? It was like, here, here’s what happened, da da, da. Um, and sometimes you get the sense when you see the no customer data was accessed, you’re like,
[00:27:42] Prasanna Malaiyandi: It’s really,
[00:27:43] W. Curtis Preston: it right? Like in this case, uh, you know, that that’s not what they said, but I’m talking about in LastPass.
But
[00:27:50] Prasanna Malaiyandi: do wonder if people were aware that LastPass does not encrypt everything.
[00:27:57] W. Curtis Preston: I think that was definitely news, right? Um, so, uh, by the way, I’m gonna add that on my list, uh, of stuff that I want to just talk about. So communications I think is just so important when you have a public breach like this. Uh, and then second is the homegrown backup thing. How are you?
I think they were like 200 million company. How are you? A 200 million company and you don’t have a corporate backup system? I, I just, I can’t, I cannot. And again, yeah, I know that like it’s important to me and not as important to other people. But you’re, you’re a company that this is, I, I don’t know how to
[00:28:42] Prasanna Malaiyandi: it’s it’s like a
[00:28:44] W. Curtis Preston: you, you are holding the crown jewels of other people. It’s, it’s, it’s like a bank that just has their friends guarding the vault. Right. That they don’t hire a security firm to guard their vault. Um, they just, they wrote essentially a script. That. And so it was, it was two problems. One is that the script contained hardcoded, uh, credentials to the cloud account where they, where they, uh, where they stored the data.
And then the second was that that script was then, uh, it was in plain text and it, and it was crawled by the threat actor. The, apparently during the, during one one of the attacks, I don’t know if it was the first attack or the second attack, we don’t know. Uh, the idea that a, that a company of that size would’ve homegrown backups was just, I dunno.
Do you wanna defend them at all on that or?
[00:29:35] Prasanna Malaiyandi: So, so I would say that it might be difficult depending on what they’re doing to find something that works. Right? And I can’t blame them for having something homegrown. Now. There’s a lot of mistakes they made, like hard coding credentials, right? Not locking it down. Right. Keeping it completely separate.
All those other aspects, which I will fault them for, but like if you had to try to back up S3 today, right? What are you gonna do? Aws s3, right? There aren’t really many great systems for doing that. So if you are talking like databases or virtual machines and things like that, a hundred percent agree with you.
[00:30:12] W. Curtis Preston: well, they, well, they would have s well, we don’t know,
we don’t know
[00:30:16] Prasanna Malaiyandi: don’t know
[00:30:17] W. Curtis Preston: infrastructure is. Right. But, um, I mean, unless literally it wasn’t possible.
[00:30:25] Prasanna Malaiyandi: because my guess is they’re, they’re storing all their password vaults in object store, even if it’s on-premises. Right. Or in their data center. In their data center, yeah.
[00:30:36] W. Curtis Preston: Yeah. I don’t know. We don’t know. Right. Um, we don’t know what they’re stored. Yeah. That would be the only reason.
[00:30:44] Prasanna Malaiyandi: Act actually you could probably it in a database.
[00:30:48] W. Curtis Preston: So I’ll, I’ll just, so I’ll make this recommendation even bigger then. Don’t create an infrastructure that you can’t back up. How about that? Don’t create an infrastructure that you can’t back up via tools that are readily available out there.
[00:31:02] Prasanna Malaiyandi: still disagree with that.
[00:31:04] W. Curtis Preston: Okay. Well, you’re wrong. It’s okay.
[00:31:07] Prasanna Malaiyandi: I, I think if I look at like, what are you gonna do around, IM ml like chat G p T, right? No one has tools for that. You’re, if you’re on the bleeding edge of something, there’s not gonna be a tool to help you back that up. Or if you have such large amounts of data, like look at iot workloads,
right?
[00:31:24] W. Curtis Preston: Well,
[00:31:25] Prasanna Malaiyandi: Some of those workloads I think are just difficult.
There’s a
[00:31:29] W. Curtis Preston: right. So, so can we, can we agree that if you’re designing a homegrown backup system, first off, you shouldn’t do it. If there’s any way to not do please don’t do it. Number two, if you are forced to do a homegrown backup system, and you’re doing it in such a way that you have hard code, Credentials in a plain text, visible script.
You are doing it wrong. And you, you are, you need to be fired.
[00:31:54] Prasanna Malaiyandi: A hundred percent. And,
[00:31:55] W. Curtis Preston: we on the same?
[00:31:56] Prasanna Malaiyandi: I, I wanna add a third.
[00:31:58] W. Curtis Preston: yeah.
[00:31:59] Prasanna Malaiyandi: be investing in a person or part of a person who will continue managing and monitoring that service. Right. Your backup service if you’re not using commercial product
[00:32:11] W. Curtis Preston: you do need that maintenance, right? All right. So the next lesson is, and maybe this will be easier to talk about, is the, is that EDR systems can be compromised? Who knew? Right? Uh, we don’t know the details, but somehow they compromised, uh, the EDR system. Uh, be fascinated to know about
[00:32:32] Prasanna Malaiyandi: yeah, and I think it is, don’t put all your eggs in one basket. Right? They always talk about like security and layers. Right. Don’t always. And I think it just sort of goes back to the old data center model, right? Whereas like you just protect the perimeter and that’s it. Because if you don’t allow anyone in, you’re all good.
But now with like cloud and connectivity and everything else, right? Just look at what happened. You compromise a employee’s laptop when they’re at home and you’re screwed, right? So I think you need the layers of defense. And what do we always talk about? Uh, least privilege.
[00:33:07] W. Curtis Preston: least privilege. Yep. Yep. least privilege and, and se and, and, um, separation of, of roles, right? Um, all of those things. And the, the more you can, the, the more sensitive something is, the more you want to, number one, limit the number of people that are able to do that thing. And number two, if at all possible, divide it into multiple levels of authentication, right?
Things like downloading your corporate customer fault.
[00:33:36] Prasanna Malaiyandi: Yeah. Well, and that should be, and I know we talk about like multifactor authentication. I think there now needs to be sort of that push towards two person. Yeah.
[00:33:46] W. Curtis Preston: Yeah. Two person authentication. Yeah. Ex actually, and the, the thing I was about to talk about is just that MFA is not perfect, right? MFA is very helpful, uh, but it is not perfect. And this if, if someone gains control of somebody’s laptop,
[00:34:00] Prasanna Malaiyandi: Yep.
[00:34:00] W. Curtis Preston: uh,
MFA’s not very helpful.
[00:34:02] Prasanna Malaiyandi: or the other, thing I was thinking is even when the Octa hack right, they basically did MFA fatigue, right? Which is, uh, which is a real problem, right? You just keep pounding. Uh, a threat actor just keeps pounding a victim until they just give up and they’re like, okay, fine, that’s me.
[00:34:16] W. Curtis Preston: which for the record, I think that person needs to be fired.
[00:34:20] Prasanna Malaiyandi: Yep.
[00:34:21] W. Curtis Preston: If, if you get 59 MFA requests and you accept one of them, in my opinion, you, uh, I don’t know, maybe you don’t agree with that, but I’m just like, you, you like, if I got 59 MFA requests, I would be calling my MFA person. I would be contacting it going, I don’t know what’s going on, but I’m getting 59 MFA requests.
I would not go, ah, fine. Leave me alone. Right. It’s three in the morning. Dammit. I’m trying to sleep. please go hack my company. Another thing that I wasn’t that impressed with is that this person had third party software, vulnerable third party software installed on her laptop.
And that was, Somehow allowed. Right? It’s very common, at least everywhere I, everywhere I’ve worked, um, since like the concept of administrator. Well, I guess that’s always been a thing since the concept of laptops been around. Mind you, when I, when I first
[00:35:18] Prasanna Malaiyandi: Back in the day, there were no
[00:35:20] W. Curtis Preston: everybody didn’t have laptops and the lap, a laptop was a computer that was small enough that you could fit on your lap.
That was a big computer. But since laptops have been common, we’ve had this problem of, you want to say, I’m gonna take administrator away. From the, the person, but when you do that, in many environments, the person can’t
[00:35:42] Prasanna Malaiyandi: do stuff.
[00:35:43] W. Curtis Preston: Right. Uh, I I would prefer that you take administrator away. Uh, if, but it, it, it, like in my case, there are many things I do that where I end up needing administrator a access.
And so if you’re going to give an administrator or, or a root to a person, right? Then you need to, you need to monitor what they’re putting on your laptop.
[00:36:05] Prasanna Malaiyandi: exactly. And, and yeah, I remember actually recently reading, uh, or seeing tweets about someone who, you know, o b s software that’s used for like, uh, open broadcast systems Right. For doing a lot of like the screens and a lot of like media creators use it. Someone went to go install it and so they entered into Google and then they clicked the first link turned out to be malware.
[00:36:28] W. Curtis Preston: Oh yeah. So you could be installing what you think is real software and it’s not. Right. Yeah. Which is which for those of you that don’t under, that’s the whole point behind the, the feature in Mac os it says you are, uh, trying to install software
[00:36:45] Prasanna Malaiyandi: an unknown to,
[00:36:46] W. Curtis Preston: unidentified developer. Right. Um, and, and, uh, yeah,
[00:36:52] Prasanna Malaiyandi: And we could probably take this for a different episode, but I’m just wondering if, do you feel that that problem gets reduced by like the Apple App store?
[00:37:03] W. Curtis Preston: Yes. I do think that problem gets reduced Again, like anything else, it doesn’t completely eliminate it. It’s not, it’s like mfa, it doesn’t solve everything because bad stuff does get into the app store. Um, but I do think it, it’s reduced right? Um, so, all right, so let’s switch to the customer, uh, focus here.
So the, the first thing I wanna talk about is, one of the things that came out in this, which you, you, you’ve alluded to already, is that we, we discovered through this story that LastPass doesn’t encrypt all customer data,
[00:37:35] Prasanna Malaiyandi: Yep. That’s just like, that’s just like sending off. Yeah. That’s like sending off fire alarms in my
[00:37:41] W. Curtis Preston: whoop,
[00:37:42] Prasanna Malaiyandi: Yeah. I was like, I was like, how can they, like people assume that a password manager is my data, I’m using it for a purpose. I assume that everything would be encrypted.
[00:37:55] W. Curtis Preston: and I’m paying you for the service, right? This is not one of these things where you should also be able to, uh, utilize my data to maybe make some money in some other way.
[00:38:08] Prasanna Malaiyandi: Well, it depends. I will say if they did explicitly cover it out in their terms of service, then that could be, uh, Potential use of that data. However, I don’t see why they would use like email address as a field that gets unencrypted when they can get that, especially in your vault, when you can get that information from, say, your profile or your account information.
Right, which will have your email address.
[00:38:36] W. Curtis Preston: yeah. We, we don’t know what they had encrypted or unencrypted. We know definitely that the passwords were encrypted. Um, and then, but we don’t, we don’t know. So here’s my, here’s my recommendation, and that is talk to your password vendor, password manager, vendor and say, Hey, are you storing any of my data unencrypted?
And if so, what, what is it? Um, and, uh, and do that in writing. Right. Um, and, and hopefully their answer is, uh, no, of course not. Well, we store like your account information, maybe you’re
[00:39:10] Prasanna Malaiyandi: Yeah, and and that’s typically what I would expect is your account information, your profile information. Right. Whatever it is that they need to be able to bill you to communicate with you. Right. That’s their
[00:39:20] W. Curtis Preston: and that, and that would, and even then, I would expect that to be encrypted. Just encrypted
for them, not encrypted for us, right? Yeah. Yeah. Um, yeah. Uh, so that’s my first recommendation. And then the next one, and this is,
[00:39:35] Prasanna Malaiyandi: Strong passwords.
[00:39:37] W. Curtis Preston: yeah, so, so, so me ask you a question Prasanna, uh, I mean, I already know the answer, but I’ll ask you a question.
This idea that they were hacked and, and they, and they got access to, to the vault. This is the thing that everybody’s worried about with using a password management. They got access to the vault. Why does that not just complet. Say, well, you shouldn’t use password managers because you put all your passwords in one place where somebody could hack
[00:40:04] Prasanna Malaiyandi: Well, and I think we talked about this in some of the prior episodes around password managers, right? But the biggest thing is if I’m a user, and I have to remember passwords for 50 different websites or a hundred different websites, they’re all probably gonna be similar-ish. You know? And so if a password is able to be guessed for just even one of the sites gets compromised, then a attacker can guess the other ones, you know?
[00:40:27] W. Curtis Preston: Y yeah. Yeah. And that, and that’s, yeah, that’s the whole point of the password manager is that you can get a super long, um, and by super long, I mean, you know, 15 to 20 characters, uh, which is where I think you, I think you need to be at least 15 characters at this point with your passwords based on how long it takes.
So let’s just talk about that because they sent out, um, some recommendations on what customers should do.
[00:40:51] Prasanna Malaiyandi: Mm-hmm.
[00:40:52] W. Curtis Preston: And they, it was very interesting. Basically they mentioned that they had changed their method of encryption at, at some point, right? Basically the way they salt and encrypt their, their passwords, they had changed it over time.
And what they said was, if your pass, what I recall them saying was, cuz I’m just going off a memory, was like, if your password is of a short, of a certain length, number one, and or if your
password was created before this date, then we recommend you change your password now. Right? And we’re not talking about the master password, we’re talking about the, the individual passwords.
And so that I think is, um, there, I’d say two recommendations. Where, where are you in terms of your password length?
[00:41:41] Prasanna Malaiyandi: for, I would say 24 characters minimum.
[00:41:46] W. Curtis Preston: Yeah, I’m, I’m, yeah. So, but I agree with you except get, uh, accept what I was doing last night. You know, what I was doing last night?
[00:41:54] Prasanna Malaiyandi: Huh.
[00:41:55] W. Curtis Preston: I, I was, I was reen enabling, I, I, so I bought a new, uh, uh, I bought an Apple TV and um, I went, I went to go, I remember it was because I’ve, I’ve made my passwords longer over time, and I went to go, uh, authenticate discovery plus, and it was one that had a, a, a 20 character password and b mixed case.
[00:42:20] Prasanna Malaiyandi: You know what you could do for
[00:42:21] W. Curtis Preston: case. No, I know, I know I figured it out, but I’m just saying I, I know the, I know the solution. Right? The Apple key keyboard solution that, by the way, that’s amazing. Um, but at the time I was, I was just cursing myself for having. Right. I was like, why can’t, because I just, I just let Dash pass create, Hey, create me a 20 character password, and they do the uppercase m lowercase Q seven squirrely, you know?
[00:42:48] Prasanna Malaiyandi: They handle all of
[00:42:49] W. Curtis Preston: hardest thing to type if you, yeah. Um,
[00:42:52] Prasanna Malaiyandi: You know what? You know what’s actually the hardest.
[00:42:55] W. Curtis Preston: what,
[00:42:56] Prasanna Malaiyandi: it when they change, like, you know, the apostrophe versus the, the other one. Right. And it’s always so difficult to tell which one it is.
[00:43:08] W. Curtis Preston: Well, my, my problem is o or a zero, like, is it, is it an O or a zero? Um,
[00:43:14] Prasanna Malaiyandi: I know the one I
[00:43:15] W. Curtis Preston: that an L or a pipe sign?
[00:43:17] Prasanna Malaiyandi: Yeah, I know The one I use uses different colors for numbers versus letters.
[00:43:23] W. Curtis Preston: Oh, it does, it does. When you’re looking at it in the, in the, yeah. The U Dashlane does the same thing when you’re looking at it in the ui. Um, but anyway, the point is you, I I would say the two things to do is, is to make sure you have really long passwords. And I, you know, I’d say minimum of 16, that if, if, if you’re, if, if the difference between a 16 character password and a 24 character password is just the number of character, number of characters, and you can just set that number and you can enter that.
There is nothing wrong with having, having a 24 character password. Right? Um, and for those few passwords that you do occasionally need to type manually, uh, I’m talking to you, iTunes, um, the app, the Apple password, right? You’re, every once in a while you’re asked for your Apple password. That one, what I would use is I would use.
A password phrase, you know, the, the battery horse staple thing. Um, if you don’t know what I’m talking about, Google Battery horse staple. And, um, it, it make it really long and make it unique and then, and then store that, um, in, in the password manager. Any additional recommendations on that
[00:44:37] Prasanna Malaiyandi: No, I agree on that. I think just in general, I know that with LastPass, right, they said before this date, if your character password was this long, you should go reset it. I think, and I’m guilty of this as well, I think people should, even if they have a password manager, they should still go back in periodically change their passwords, right? Because like I just started going through and for like the critical websites I’ve gone through and started changing my password because I realize I haven’t changed it in a long time because
you don’t always go back in. At least the one I use, it’s not an easy mechanism to go change it, and so like it’s not automated, which I know some password managers support, but for me it’s like painful.
And so I’ve now just like you’re going through all your pictures and going through and looking at them, I’m doing that same effort with all my passwords where I’m just going back through and being like, okay, do I need this? Do I not? Let me go change it.
[00:45:32] W. Curtis Preston: I completely agree with your recommendation, was actually gonna be, be my my second recommendation.
What you should probably do, don’t think of this as a giant project you need to do tomorrow. This is a, this is a housekeeping thing.
[00:45:45] Prasanna Malaiyandi: Yep.
[00:45:46] W. Curtis Preston: Um, figure out a way, um, you know, in, let me look in my, I’m just gonna pull up
[00:45:54] Prasanna Malaiyandi: Do you have 670 passwords? I believe.
[00:45:56] W. Curtis Preston: No, I know that, I know that, that’s not what I’m thinking. I’m gonna add it to my recommendations for password managers. So I’m looking at my vault. It shows that there’s a last used column. I think there should be a last change.
[00:46:11] Prasanna Malaiyandi: yep.
[00:46:12] W. Curtis Preston: To make it easy for me because this means that cuz yeah, this, I, I, I want to know passwords that I haven’t changed for five years.
Cuz what I was gonna recommend, and it, it means that you’re gonna have to like figure this out manually is just every day go and change a few passwords, right? If, if you’re like me and you have, you know, close to a thousand passwords and a password manager, and I’m sure when some people say that, they’re like, what the, what the what?
And I’m like, welcome to SaaS land, man. I, I will say, you know what I’ve started, what I’ve started to do is I’ve started as much as possible is to use Google Authenticator, right? To use my, instead of if
[00:46:49] Prasanna Malaiyandi: You sign in with Google.
[00:46:51] W. Curtis Preston: sign in with Google. Uh, that way I don’t have, I don’t have a new password, uh, to manage.
But, um, the, um, it, it would be really nice, uh, this is, this is for the password manager folks. It would be really nice if there was a last change.
[00:47:07] Prasanna Malaiyandi: Or to notify me, right? Just have a notification that says, okay, notify me after 365 days
[00:47:16] W. Curtis Preston: Right. Yeah. So in my case, in my case, based on the number of passwords I, I have, I would have to change two of them a day to change all of my passwords within a year. And again, this is, this is just housekeeping. It’s not, you know, you don’t have to, cuz the idea of me sitting down and changing 700 passwords right now is that that is, that is daunting.
But, but Dashlane doesn’t help me in that. It doesn’t tell me which passwords I’ve changed recently.
[00:47:40] Prasanna Malaiyandi: Yep. You just have to remember,
[00:47:43] W. Curtis Preston: yeah.
[00:47:43] Prasanna Malaiyandi: the A’s. One day, the B’s, another day, the C’s, another day.
[00:47:47] W. Curtis Preston: Believe in that. That’s only 26, you know, and at some point you’re gonna get to a letter that has a hundred, a hundred websites. Uh, maybe do it, do it by, by month. Like, yeah, maybe.
Maybe that’s, maybe that’s a doable thing. Um, only, only, um, Only do it on days of the week that end in a Y. Um, uh, okay, so we already talked. Um, yeah, so reach out to your password manager and what, and find out Right. And then, and, and while you’re reaching out to your password manager, use that last episode if you haven’t listened to the previous episode where we had the, the guy that, that yeah. See that, that, that built this, that it was, it was a university and they went and did, um, uh, this investigation and find out which of these your password manager has addressed and which they, and which they are saying.
So in the case of Dash Lane, for example, the one that I know they haven’t addressed is the, the password or the, the email. The known email issue. And that is honestly a problem with a lot of places, not just Dash lane. Um, and I wish they would do away with it, but I don’t know. I’m not in the meetings. Um, any final thoughts?
Prasanna?
[00:49:12] Prasanna Malaiyandi: so I would say two. One is if you’re using LastPass, you should probably consider moving to something else.
[00:49:19] W. Curtis Preston: Um, I would be stronger with that.
[00:49:21] Prasanna Malaiyandi: yeah, that would be one. And the second is everyone should be using a password manager. Doesn’t matter what type it is, if it’s a cloud base, if it’s free, if it’s the web browser base, right?
Whatever it is, use a password manager.
[00:49:36] W. Curtis Preston: Yeah. The evidence is not in LastPass’ favor. Um, and I, I think it’s, I think there are so many people, the only people, the only people that are left as LastPass customers are customers who are just flat out not paying attention. Right. They, they probably didn’t get the emails from LastPass. They probably don’t follow any industry news and, um, at, at some point.
[00:50:03] Prasanna Malaiyandi: well, I, I think, I think what it’s actually gonna be also is if you’re a large corporate environment, right? You’re probably looking at how to migrate away, but you might have a contract. So on your renewal, you’re probably not gonna sign up again, but you’re dealing with it as it is while you’re migrating to a different solution.
That could be my
[00:50:25] W. Curtis Preston: good point. That’s a good point cuz they do use it, they people do use it as a corporate password manager. Yeah. Um, I would think if I, if I were LastPass, clearly LastPass does not care what I think cuz they’re not gonna enjoy this, this particular episode at all.
But if I were LastPass, I would be proactively, because even though I can’t see the encrypt, I can’t see the unencrypted password, I can see the password length based on the length of what’s stored. And I most likely have a last modified date on that record. Right. And I, and if I, assuming they have that, but I think that’s a pretty standard field to have.
If, if I have that I would be proactively like, you know, like my daughter, uh, has a car. She says, she says that, um, She’s taking care of it, but Toyota thinks that she’s still subject to the airbag recall. I get a, I get a placard, not a placard, but you know, one of those like poster card things in the mail every month for the last multiple years
because they’re like dude, please fix this.
And I think that’s what you LastPass should be doing for any of their existing customers.
[00:51:41] Prasanna Malaiyandi: Yeah, that makes sense.
[00:51:43] W. Curtis Preston: yeah. All right. Well, hopefully, uh, you found some use in this, uh, not a, not a particularly enjoyable episode, but hopefully we can learn some things from
[00:51:53] Prasanna Malaiyandi: Exactly. Yep.
[00:51:54] W. Curtis Preston: and with that, I want to thank you for your attention and thank you for listening.
And be sure to subscribe so that you can restore it all.
