Nothing but the best for our listeners! Curtis and Prasanna review their favorite episodes from 2022, from the time Curtis had to restore Backup Central to the shocking outage at Rackspace ( which is still ongoing!). This is your chance to catch up on what you missed last year as we prepare for a new year! Here are links to all the episodes discussed. Happy New Year!
Mr. Backup Forced to Restore Backup Central
Restore Test Failed to Bad Documentation
Security expert rips Okta for their response to hack
How to prevent ransomware, slow its spread, and respond if you get it
How to prepare for an emergency (at home and work)
Backup practitioner tells stories from the trenches
Top 5 security mistakes you’re making in the cloud
International lawyer discusses e-discovery in US & other countries
Shocking RackSpace Hosted Exchange Ransomware Attack
Transcript
as 2022 comes to an end. This is our year-end review episode, where we talk about, I think about a dozen episodes that really hit home for us during the year. I hope you enjoy it. And I hope you have a great 2023. Hi, and welcome to Backup Central’s Restore it All podcast. I’m your host, w Curtis Preston, aka Mr. Backup. And I have with me the guy that can probably explain why there’s a lot more echo in this room than there normally is. Prasanna Malaiyandi
[00:00:51] Prasanna Malaiyandi: Echo. Echo, How’s it
[00:00:53] W. Curtis Preston: hear it on your end? Cuz I really hear it on my end.
[00:00:56] Prasanna Malaiyandi: I can hear it. It’s not as bad, but you could definitely tell it’s different than our normal podcast recordings.
[00:01:03] W. Curtis Preston: Yeah, although I’m in the same room, although for those of you that are used to watching the video, these things are normally way far back. It’s because my desk is literally in the middle of the room because, um, I have prepared for the flooring, uh, the final room, the flooring of the final room, and the
[00:01:21] Prasanna Malaiyandi: Woohoo.
[00:01:22] W. Curtis Preston: Multi-month flooring project and I decided to just leave my desk set up and I just have to move it around as the flooring progresses.
[00:01:31] Prasanna Malaiyandi: Which isn’t bad. At least it’s all set up and
[00:01:34] W. Curtis Preston: yeah. And everything else is outta here. Um, and then, and the doors are off the closet and, uh, you know, all
[00:01:41] Prasanna Malaiyandi: it’s crazy how much an empty room like, or how much the stuff in a room helps with echoes. Right? Because once you remove it all, it’s like, wow,
[00:01:51] W. Curtis Preston: Yeah. Well, and also the, I think the big thing is that it was carpet, right? It was previously carpet and now I’m just looking at concrete, which
[00:01:58] Prasanna Malaiyandi: So now when you put in flooring, are you gonna put in sound
[00:02:03] W. Curtis Preston: Well, we’ll see how, we’ll see how it goes, right? I think that well, we’ll see. We’ll see. Who knows? Who knows?
[00:02:09] Prasanna Malaiyandi: Does wallpaper help? You can put wallpaper up.
[00:02:12] W. Curtis Preston: wallpaper, do I look like a wallpaper kind of guy to you? Um, we’ll see. We’ll see what happens once the flooring gets in. Um, see if it
[00:02:22] Prasanna Malaiyandi: but it is a final room, which is good.
[00:02:25] W. Curtis Preston: It is the final room. And, uh, yeah. So I’m excited about that.
And I’m sure that the three, the three listeners, uh, that care about that project will also be excited that I finished this. And then the other, all the other listeners are gonna be like, finally he is gonna stop talking about this flooring project
[00:02:43] Prasanna Malaiyandi: And then it’ll just be the next project. Don’t worry listeners. We got you. We’ll keep things interesting.
[00:02:48] W. Curtis Preston: Well, speaking of look, so interesting.
So, so the, the true, the full story is that that project literally started in the January of last year in that, that’s when I started. Ordering the, um, the flooring from Costco. And it took months due to, I don’t know if you’ve heard, but there was a big supply chain problem.
[00:03:12] Prasanna Malaiyandi: Was there I didn’t,
[00:03:14] W. Curtis Preston: but yeah, so, so January, so I thought I was gonna start this project a lot earlier, but I ordered it in January and I got it like in, I don’t know, April I think.
Um, and then I started it kind of in. June, something like that. Um, but anyway, anyway, but we are looking back on 2022, so it’s natural that I look back on what I did the entire year of 2022, which was try to put down flooring
[00:03:44] Prasanna Malaiyandi: It’s okay. It’s, it was a sizeable project, Curtis, you
[00:03:48] W. Curtis Preston: It was, it was, well it still is. It still is. Um,
[00:03:52] Prasanna Malaiyandi: the question. Now that you’re probably 90% of the way through,
[00:03:56] W. Curtis Preston: Mm-hmm.
[00:03:58] Prasanna Malaiyandi: Would you have hired this out to someone to get it done in a much shorter timeframe or.
[00:04:06] W. Curtis Preston: I, I will say this is not on the same level as the solar project, like the solar project. I regret doing it the way I did. I don’t regret this. It’s just, it’s just, it’s a lot. And I saved, like, I don’t know, I think based on the pricing I saw, I saved like four grand in labor. So, um, Yeah. Um, although I still, I have to pay for some labor because I do have to pay somebody to put down the, uh, the baseboard.
But cuz I’m not doing that cuz that’s just, that’s detail work that it’s just better left as people that know what they’re doing. Um, but, uh, yeah, so I, I’m not regretting it. Um, I, I will say this, I’m not gonna do it again and, um, I’m not gonna do the second floor
[00:04:54] Prasanna Malaiyandi: Wait, I was gonna ask, what about the stairs?
[00:04:56] W. Curtis Preston: I am doing the stairs.
I’m gonna do the stairs. But by the way, when you put vinyl flooring on stairs, you have to put non tread or, yeah. Non, non, non-skid. You have to put tread on it because, uh, people come down and stocking feet and they just come flying down the stairs. I don’t, I don’t want to, I, you know, I don’t wanna see my family go down the stairs like that.
Um,
[00:05:20] Prasanna Malaiyandi: the stairs with like a slide, you know? That way it’s obvious.
[00:05:25] W. Curtis Preston: Um, so we’re looking back on 2022, right? And you know, I, I thought, you know, we did, we did roughly 50 episodes. Um, 50 plus. I don’t know, could we took a week or two off there, here and there for holidays? Um, one of which was last week, right?
We’re recording this on December 29th. Uh, when I’m looking back on 2022, I one, the first thing that pops out to me was, do you remember when I had to restore backup central?
[00:06:00] Prasanna Malaiyandi: Oh, I do remember that. Yeah,
[00:06:02] W. Curtis Preston: That that was an episode, Mr. Backup forced to restore backup central.com. Um, I don’t remember why. Do you remember why ? I, I don’t remember what happened.
[00:06:14] Prasanna Malaiyandi: it was, it was. I think it was your forums got corrupted and you had an issue, so you had to restore your message
[00:06:25] W. Curtis Preston: Yeah, there was.
[00:06:26] Prasanna Malaiyandi: lose all of the back history.
[00:06:29] W. Curtis Preston: Yeah. Yeah. I think you, I think the table, like the tables got corrupted in the, in the MySQL database. Yeah. Yeah. And so, and I, I think I didn’t notice it right away, like it was corrupted for a few days, and so I had to, I had to go back to my object storage version. Uh, so that, and, and most importantly it worked.
Um, so I , so I liked that, that episode. Right. Uh, that was back in February.
[00:06:56] Prasanna Malaiyandi: I know you said that it was successful and you were kind of a little surprised , right? Um, but had you actually done a restore test at any point in time or is it one of those things where I was
like, uh, I, hope this works.
[00:07:13] W. Curtis Preston: It’s not the first restore that I’ve done at Backup Central. Um, and it wasn’t the first restore it, it was maybe the first time. No, it’s not even the first time. No, no, it’s just, um, I think it had been a while since I’d done it. That’s probably what it was. It had been a
while since I’d done it,
but, um, yeah.
But hey, it
worked. So
[00:07:38] Prasanna Malaiyandi: It worked.
[00:07:39] W. Curtis Preston: so that was a, that was a
[00:07:40] Prasanna Malaiyandi: That’s all that matters in the end.
[00:07:41] W. Curtis Preston: Yeah. But speak, yeah. Speaking of which, maybe you can talk about this, uh, the Restore test failed, uh, podcast that we had, um, due to bad documentation.
[00:07:54] Prasanna Malaiyandi: Oh my gosh, this happens all the time. I.
[00:07:57] W. Curtis Preston: Gary Williams.
Yeah.
[00:08:00] Prasanna Malaiyandi: this happens all the time where it’s like, okay, I know that we have all this documentation. I think in that episode, correct me if I’m wrong, but I think it was that like they’d forgotten to include like an active directory server or some other piece of their infrastructure.
As part of the backups that had happened. And so when they went to restore it and do their DR test, it wasn’t like they didn’t have the bits to be able to first authenticate before they can actually do the restores and bring everything up. And so their DR test failed. But luckily they found it out while they were doing the actual test and not when they actually had to do a failover to recover from an. And that just goes back, I know Curtis, you always talk about this too. It’s like make sure you test your dr. Make sure you have everything done right.
[00:08:48] W. Curtis Preston: And make sure that it’s not the same person doing it, which is kind of what happened there. I think he, wasn’t he a new person? He was a
new person, or it was something, he was new and he tested it and, and it didn’t work well.
Uh, but
[00:09:01] Prasanna Malaiyandi: that’s right. Yeah. It was
[00:09:02] W. Curtis Preston: they tested it because they figured it out and Yeah.
[00:09:04] Prasanna Malaiyandi: yep.
[00:09:05] W. Curtis Preston: Yeah. I
like, I liked that episode
[00:09:07] Prasanna Malaiyandi: He was new and it’s like you always say it’s like a poor job. Yep.
[00:09:13] W. Curtis Preston: Yeah. Yeah. And we, and then we had, we had a couple of great episodes with a guy that we refer to as Snorkel 42, because that’s his Reddit handle. Right. Um, he, he’s not, it’s funny, he, you know, he goes on video, uh, we know who he is, but he just prefers to go by that by the handle, and that’s fine.
Um, but we, he, he, there was the big Okta hack.
which you may recall
from, uh, when was that? I’m looking
[00:09:41] Prasanna Malaiyandi: that was like, I wanna say
[00:09:42] W. Curtis Preston: half look, looks like it was in March.
Yeah. Yeah. Um, and yeah, he ripped a, a new one. That’s what, what I recall.
[00:09:51] Prasanna Malaiyandi: and, and, and I think he, he was unhappy with the fact that they weren’t transparent about what happened and it took so long before they shared any information. Right. I don’t think he was blaming necessarily what had happened, that they had maybe had a fault in security. I think it was they could have done a better job of messaging, informing people what had gone on.
But cuz I think for weeks people were in the. They didn’t know what happened and
[00:10:18] W. Curtis Preston: you,
yeah, you, you, you fast forward up to another interesting episode that we did just like last week. The one about the, um, the. Darn it. Um, what’s it called? Rackspace? The Rackspace
outage. They, that, that’s still going on, right? It started on December 2nd. It’s still ongoing. And, uh, they have been as transparent as milk.
Um,
[00:10:47] Prasanna Malaiyandi: I was gonna say like a black hole, but
[00:10:50] W. Curtis Preston: Yeah. Yeah. Um, they, you know, Yeah, that’s just a med. And, and, and, and we, we talked about the same thing with the Uber, the Uber story. That was
another great story where we talked about lessons that we can learn. And again, it’s like, Transparency matters. Um, now unfortunately, there’s a LastPass story.
We we’re doing a review of 2022, but also in 2022 was the LastPass hack, which we’re gonna do an episode on, I think next week. But, um, that, so on the one hand they’re being pretty transparent. On the other hand, it’s gonna really hurt them as a company. But, um, you know, there are people who are saying You need to drop last
pass. Um,
but, um, so let’s see what else, what else
[00:11:40] Prasanna Malaiyandi: So, so the other thing, the other thing also with snorkel, so I know we did a couple episodes, right? I know we talked about the Octa.
The other thing though that I think he did a really good job with is the basics that you should be thinking about from a security perspective to try to prevent the damage from ransomware, right?
Or bad actors,
[00:12:01] W. Curtis Preston: exactly. Exactly. Just. Just assume you’re going to get ransomware and then do these things in your environment so that when you get ransomware, it will be able to do little to no damage.
[00:12:15] Prasanna Malaiyandi: Yeah, and it was a lot of common sense things that you didn’t have to s shell out a lot of money for, right? But it was things that you should be doing from a best practices perspective, and I really like that episode as well.
[00:12:28] W. Curtis Preston: Absolutely. That was, I, I, yeah. And you know, it’s funny, uh, just in terms of like fun episodes, the one right after that one, the one with Virginia Nichols, the one where you, and I said like three words, um, that was about preparing for an emergency, not so much a disaster, but that, that was just like, in terms of when I look back on episodes, it just, you know, made me, I don’t know, made me laugh.
I guess it was a
fun episode because that woman. A powerhouse. Um, you know, and literally we were just like, how do you prepare for an emergency? And then she talked for 45 minutes. Right. Um, ,
that was, that
[00:13:06] Prasanna Malaiyandi: And, and it’s things that you don’t even think about, right? It’s like an emergency in your community, right? Not necessarily like a
computer outage, right? It was, Hey, you have a fire, you have a tornado that goes through your neighborhood. What are the things you need to do? How do you prepare? How do you make sure you have a communications plan?
So all the things that people should
[00:13:25] W. Curtis Preston: That was, yeah. And, and, and she approaches it both from a personal standpoint, uh, as well as a, um, a business
standpoint. Right.
[00:13:36] Prasanna Malaiyandi: yeah, the risk, right?
[00:13:38] W. Curtis Preston: yeah.
Yeah. And then there was an episode where we had, what I’m gonna refer to as the most mentioned person this podcast. Do you know who I’m talking about?
[00:13:55] Prasanna Malaiyandi: Mr. Stuart Little, I love Stuart. He always comes up
[00:14:02] W. Curtis Preston: comes up just a lot. Yeah, he just, yeah, he just comes up. Uh, and so basically we finally got him on, he’s retired now. Um, and, uh, you know, and we talked about his, you know, so his backup practitioner tells stories from the trenches. That was a lot of fun. Uh, cuz I’ve known Stuart for a long time.
He has, he has more than a few Curtis stories right? That he likes
to tell. Where I don’t, I don’t necessarily fare very well in those stories, but,
[00:14:34] Prasanna Malaiyandi: it’s okay. It’s a learning experience for all of us. Yeah. And then well, and then speaking of that, right after that, we sort of started talking about various technologies that are out there and how sort of.
Disc isn’t great for long-term archiving, right. Or keeping data around for a long time. Tape is great, but there are downsides with tape as well.
And so there was, uh, this awesome new techno, or not new, but this technology I never heard about, right? But that was introduced by another guest of the podcast, Daniel Rose Hill. Hey Daniel, the backup anorak right uh, talking. Talking about M disk, right? And we had the inventor, right, Barry Lunt on the podcast talking about that technology.
[00:15:25] W. Curtis Preston: Yeah. That was pretty cool cuz it like, you know, he was there at the very beginning and he was able to, Uh, he, he, he set straight what I would think of as a few, a few misunderstandings of mine. Right. Um, remember I thought that like, uh, cuz he’s LDS and I thought that LDS was like part of
the foundation of it, right.
Um, and he’s, you know, he is like, oh, we
would’ve loved to have him as a customer, but that, but that isn’t, that isn’t how it happened. Right. Um, and, uh, but yeah, that idea of, I, I, I. I hope that technology continues to do well. It’s one of these things where it’s easy to explain and, um, uh, it’s also to easy.
It’s easy to, I don’t know, confuse it, you know what I mean?
Um, there are some sort of knockoffs out there. Yeah.
Um, and then this next one that, that I’m looking at here, it was Paul Hadgy from, um,
he’s the, the, CEO of Harangi. So security, right.
I don’t think enough people talk about security in the cloud.
And he, he, he has a company that basically this is what they do is they help people do better security in the cloud.
Right. Um, and, uh, I think that was a, that was a very interesting.
[00:16:47] Prasanna Malaiyandi: Speaking of security in the cloud, I think that there’s like a misunderstanding from some customers or some end users that when you move to the cloud, everything is handled for you. You don’t have to worry about security, right? You don’t have to. Everything is secure.
You don’t have to worry about backups. Yeah. , and I think that’s a fallacy because it makes the cloud is so powerful and so flexible that you can easily shoot yourself in the foot if you’re not careful. Right. Even look at what Amazon used to do years ago, right? And they fixed it finally, right? Anytime you created an S3 bucket, it was publicly available to everyone.
And how many security issues did that cause? And yet, even though they’ve changed it, where the default is no longer enabled by default, right? Public access enabled by default. Still people got it wrong and people left buckets open, which led to breaches and other
[00:17:41] W. Curtis Preston: right.
[00:17:43] Prasanna Malaiyandi: It is a hard thing, especially with all these complex services and things that you normally may not consider when dealing with an on-premises infrastructure that you can sort of put a wall around and be like, yep, this is good for me.
When it’s in the cloud, you no longer have that ability to necessarily put the wall in place unless you do things explicitly and different.
[00:18:05] W. Curtis Preston: Exactly. Um, and, and then the, the next thing that you know, this is one of those where it’s a topic that comes up a lot, I think, or at least I bring it up a lot on this podcast. And it was interesting to talk to somebody who actually does it. And that is this idea of the, the actual legal. Side of things, right?
We, we, we had a, a person who specializes in international law, Joseph Dayner, and, uh, he’s an international lawyer and talking about e-discovery and, you know, the way it works in the US and the way it works in other countries. It was interesting. I, I knew that we were, we were different when it comes to other countries, but it was interesting how he said that, you know, there are some countries that find the way we.
Discovery, like hilarious. The idea that you are required, if you’ve got the evidence in your backyard, you’re required to give it to the person who’s suing you.
Right. And, And, there are countries where, um, I think he said Germany was one of them, where, where they’re just like,
Uh, no, you’re not
required to do that, but, but in the US you are.
And so the, the thing that I’m always warning people about is, you know, backup and recovery and using your backups as archives and storing your email for a really long period of time. And then if you’re in the US and many other countries that if you’re sued, you then have to provide that.
And, um,
[00:19:31] Prasanna Malaiyandi: have an example of that, right, Curtis? You’ve actually experienced that
[00:19:36] W. Curtis Preston: Yeah, I have. And yeah, and it, and it’s, and it’s, um, you know, I saw it cost a company millions of dollars. Uh, I’ve also seen it, I’ve also meaning that it cost them millions of dollars to get the data back. I’ve, I’ve seen it cost a company billions of dollars where they lost the lawsuit because they were so bad at retrieving their email.
Um, that they, um, the, the judge issued, and we talked about it in the recording, the adverse inference. Uh, which is basically the judge is inferring something from your behavior that is adverse to your case. And
essentially what this judge said was nobody could be this bad at restoring their stuff. so they clearly, they’re doing it on
purpose. And so they lost the lawsuit as a result, and it was a
multi-billion dollar lawsuit. Uh, so let’s talk about, what I think is our final normal recording of 2022, which is the, the Rackspace, um, you know, recording. Yeah. If you haven’t heard that one, that’s, I think the one, the episode immediately preceding this one, we just cover what we knew.
A couple weeks ago. We know more now. Um,
[00:20:52] Prasanna Malaiyandi: but it’s not any better news though for customers.
[00:20:56] W. Curtis Preston: w well, no. Well, no, it’s better. Well, I, well, so two weeks ago we were thinking they had no backups. ,
right? It looks like now they do have some backups. They, again, just were really bad at restoring them. Um, and that, uh, and so, and so they really left some people out in the, you know, uh, so it’s, let’s,
so they restored people’s email services, but by, by moving them over to 365, okay?
Now you can send and receive. But for weeks of time, many weeks of time, they didn’t allow these people access to their previous email because they didn’t, because they didn’t have a good backup of it. And by good backup, I mean one that you could bring up in, I don’t know, a few
days, right? Well,
we don’t yet.
No. And this is because of the opacity that they have approached this, what we don’t yet know. the degree to which ransomware like affected their ability to
restore. We don’t know the degree to we, you know, we don’t know. We don’t know because they haven’t
really said anything. Um,
[00:22:14] Prasanna Malaiyandi: the challenge, right, is they haven’t said anything. A lot of people are making some assumptions, right, about what could go on just based on limited information out there or speculation, but it’s one of those things. Chris, do you think we will ever really know what happened?
[00:22:32] W. Curtis Preston: We’re just left to just theorize right.
Is the reason they were so slow at restoring that they were too busy trying to. ,
were they still putting out the ransomware fire?
Right. Um, were, because it looks like they had backups, um, for, for most of the customers, they had backups.
And interestingly enough, let’s just bring this up because I bring this up a lot. I say that SaaS services, and this is a SaaS service. Hosted exchange, they are not responsible for your backups. Right.
And, and and what I normally say is that if they’re responsible for your backups, it will be in your contract, right?
It will be in the terms of service.
Well, guess what It was in their terms of service.
[00:23:27] Prasanna Malaiyandi: Oh no,
[00:23:29] W. Curtis Preston: that’s what we know now that we didn’t know then. Okay.
Um, I wanna, I wanna do a whole lot other separate episode on this, but yeah, it, it, it, that’s, that’s new information. It was in
there, but what’s interesting, what’s interesting, you know what I, you know what
[00:23:45] Prasanna Malaiyandi: We’ll do it in the next
[00:23:46] W. Curtis Preston: episode. Uh, we’ll do it well.
We’ll, well, right now we gotta cover last pass. The next episode will be last pass, and hopefully
after that we’ll get some, but yeah, it’s in their, it’s in their sla,
[00:23:57] Prasanna Malaiyandi: Yeah. Oof.
Just based on everything we’ve talked about though, it. Mainly has been around sort of how do you recover from bad events that happen. It might be natural, it could be bad actors. How do you protect yourself or minimize the risk? And then how do you recover in situations? Right. Just sort of recapping the, a lot of the cases that we’ve talked about.
Right. And I don’t think it’s going away either. I think 2023 is gonna be more of the.
[00:24:27] W. Curtis Preston: no, it’s not going away. Join 23. More the same. More the same. Same, but better, different, worse. I don’t know that , but I will
say
Before 2023 gets here, my flooring will be done It will, it will be done. It will
be done today or tomorrow? Probably Today. Yeah. Um,
well, hey, it’s been fun. We’ve done another year of this. Uh, we’ve been doing how long? We’ve been doing this now since 2019, June of 2019.
[00:25:06] Prasanna Malaiyandi: years. Yep.
[00:25:08] W. Curtis Preston: Wow, dude. And, and, and, and the, the, um, let’s see the episode. This episode will be 184. I think that’s 184 episodes in three and a half years.
Crazy
that that we’ve figured out how to talk about this for such a long time.
[00:25:33] Prasanna Malaiyandi: I thank you to the listeners also for listening to us because without you we wouldn’t be here. Right? Doing this week in,
[00:25:40] W. Curtis Preston: Yeah. Yeah, absolutely. Absolutely. And we love the ones that reach out. We love the ones that, uh, um, by the way, we, we didn’t
even do our usual disclaimer.
This is, this is an independent podcast. This is just me and Prasanna that happened to know each other cuz we used to work at the same company. He abandoned us for a different company, So I work at Druva, he works at Zoom. We’re not speaking for either company. And, uh, this is an independent podcast, a couple of couple of indies. And, um, yeah. And so we, we, we want you to rate us, rate the podcast, scroll down to where you’re, you’re at. Uh, does that make sure to follow, you know, subscribe to the episode or subscribe to the podcast.
Give us some stars, give us some comments, man. We love the comments and we also would love to have you join the conversation. Whether you wanna do it privately or publicly, send me an email, uh, w Curtis Preston gmail or, uh, at WC Preston on
Twitter.
That
[00:26:40] Prasanna Malaiyandi: much longer. Yeah.
[00:26:42] W. Curtis Preston: Yeah. Um, I don’t know. My recommendation right now is, uh, on Tesla stock is buy, um, because it’s down quite a bit.
[00:26:54] Prasanna Malaiyandi: We are not financial advisors.
[00:26:56] W. Curtis Preston: We are not
financial advisors.
[00:27:00] Prasanna Malaiyandi: Anything you hear mentioned on the podcast is our opinion. Please speak with your own financial advisor for any advice
[00:27:07] W. Curtis Preston: not financial advisors. We’re, we’re not, uh, we’re not
legal advisors. We’re not doctors. Although we occasionally give opinions regarding all three of those. They are worth exactly what you paid for ’em. Um, so anyway, well, thanks again for another fun year
[00:27:29] Prasanna Malaiyandi: Likewise Curtis, and here’s a better 2023.
[00:27:34] W. Curtis Preston: Good luck with that.
Again, thanks for listening and be sure to subscribe so that you can restore it all.
