Tomorrow marks two weeks from when the RackSpace outage started on Dec 2, 2022. They confirmed it was via a ransomware attack and it is not. going. well. We’re going to do a deeper dive into this once it is all over, but this is a first-blush look at what is happening and RackSpace’s reaction to it. When we recorded this episode, their reaction was not looking good. I’m sad to say it’s gotten even worse. Check it out!
Transcript (Auto-generated)
[00:00:00] CP: Hi, and welcome to Backup Centrals Restore All podcast. I’m your host, W. Curtis Preston, a k a, Mr. Backup, and I have with me my medical non consultant, uh, Prasanna Malaiyandi How’s it going?.
[00:00:54] PM: Good. Curtis, how are you? , how are you feeling?
[00:00:58] CP: a as you know, it’s been a rough week or two. Um, you know, and, and you know, and, and I’m, I’m now down to three, technically down to two medications.
[00:01:11] PM: Okay. That’s an
[00:01:12] CP: one I can take, one I can take on demand. Uh, the other, um, or so, so two that I’m supposed to continue taking until they’re gone. Um, it’s been, it’s been.
Oh, you know, I don’t
[00:01:28] PM: You, you, I do have to say though, you sound a lot better than like three days ago when we talked so.
[00:01:36] CP: Yeah. Yeah. Well in the worry, and we’ll see if it happens on the podcast is if, is if I get to actually, um, you know, as you know, I like to laugh.
[00:01:45] PM: Yeah.
[00:01:46] CP: But if I, if I start to get an actual deep laugh, I will cough.
[00:01:50] PM: Mm.
[00:01:51] CP: There’s this cough that, and apparently I did a little research that it is a common side effect of a leftover viral infection.
Yeah. So, um, I had a, I had a, um, I got a really bad sinus infection from a tooth extraction, and then I got the flu. Um, and um, apparently we, you know, we went to reinvent, right? Um,
[00:02:17] PM: Couple of weeks ago. No, last
[00:02:19] CP: yeah. And it was last week. Yeah. And apparently almost the whole crew got sick when they got home. Um, either flu or covid.
Um, so, you know, I don’t know. It, it may cause us to, I wouldn’t be surprised as if, if it causes us to have some sort of new procedure or policy or something, you know, because I was supposed to go to another trade show right after,
[00:02:42] PM: yeah, and you.
[00:02:43] CP: that didn’t, that
[00:02:45] PM: Well, I know in the news recently, and once again, this is not medical advice, but just kinda keeping people up to date, right. In my, the county that I live in, they just actually were flagging it and saying, yeah, the number of cases are currently on the rise, and especially with the holidays and all the rest, they’re like, be safe, be vigilant.
Mask up if you can stay home if you’re sick.
[00:03:05] CP: And what sucks is, you know, like I’m, you know, I, I, I respect those who want to continue wearing a mask. I have no issue if you wanna wear a mask. I Prasannally, and I wore them when it was, you know, when I thought, When it a, when it was required, and I didn’t complain. I’m currently in the face of like, I am so done with the mask
And so I wear a mask when I, you know, when I kind of have to, uh, but like, I didn’t want to be the guy walking or I didn’t want to be, like, out of all of the attendees of the 60,000 attendees
right there there were like 20 of them were wearing masks and I didn’t want to be the 21st, you
know?
[00:03:45] PM: And I think that’s the hard part, right? It’s like peer pressure, societal pressure, right? It’s like, Hey, I shouldn’t be any different than everyone else, and. . I think it’s one of those
[00:03:55] CP: And also
[00:03:56] PM: has to sort of judge and figure out their own risk and figure out what they want to do.
[00:04:00] CP: yeah. Um, the, um, the mask thing came up on, um, Sebastian Maniscalco, which is a comedian that I love, and his most recent thing. Um, He was talking about. He said that he got the vaccine, not, he didn’t get it for him. He’s like, I didn’t get it for you. He says, I got it for the same reason.
Uh uh. All the Italians. No, the Italians. He said the Italians did it because he said we found out that we couldn’t taste food if we got Covid. So he is like, so that’s why he got the vaccine? Uh,
[00:04:36] PM: But I’m glad that you’re feeling better.
[00:04:38] CP: I should throw out our disclaimer that, uh, this is not a Druva podcast, not a Zoom podcast either. Where, where, uh, Prasanna happens to work. Uh, these opinions are ours, and, uh, none of this is medical or legal advice. God forbid, or, you know, red, I’ll tell you what, we, we give official device advice, but, uh, you know, gadget, gadget,
[00:05:04] PM: and put the holidays around the corner. By the way, FYI, for those, I don’t know when this podcast is going out, but Best Buy is running a sale on those Ember Smart Mugs. If you are looking for one, now is the time to buy one.
[00:05:16] CP: Yeah. I think it’s gonna be too late by the time this podcast goes out, but maybe it’ll be in time for you to buy, buy me a birthday, a gift. It’ll be coming up on my, my 57th birthday.
[00:05:28] PM: Dang. Curtis.
[00:05:31] CP: Getting, getting old up there. Not as old as Stewart though. Hey, Stewart
[00:05:38] PM: uh, Stewart, I think we’re gonna need to have Stewart come back on the podcast to have a conversation with you.
[00:05:45] CP: Yeah, yeah. Um, so I, I wanted to talk about this. You know, the story that we’re gonna talk about today and, um, actually two stories, but the, the main story, it’s one of these where you’re like, it falls into the, are you kidding me? Category, right? And it also falls into the, um, the story helps prove a couple of points of mine, right?
And you, you will see me. You will hear me talking about those points here on this podcast today. So we’re talking of course. What are we talking about? Prasanna?
[00:06:26] PM: Uh, what are we talking about? Oh, yeah, so, uh, the recent, yeah, the recent outage that happened at Rackspace, I want to say it was December 2nd,
[00:06:37] CP: Yeah, that was the beginning of it.
[00:06:38] PM: Yeah. and it’s now December 8th and people are still not, or the service is still not up and running.
[00:06:47] CP: and Rackspace as of now, their official line is, we don’t know when this is gonna end. We don’t know when or if we’re gonna be able to restore data.
[00:06:58] PM: So, so, so maybe just a quick background on Rackspace for the listeners who may not be familiar. Right. So Rackspace is a, what would you call ’em? They’re kind of.
[00:07:08] CP: where they have a bunch of racks,
[00:07:09] PM: Well, it’s kind of like Amazon before, or AWS before aws, right. In a sense, they were kind
[00:07:15] CP: Yeah. They’re, they’re, yeah. Yeah. They’re, they’re, they’re closer to a colo than a, than a cloud facility, I would
[00:07:24] PM: Yeah. But they do offer managed services, including what got hit, which is their hosted exchange.
It. Environment, so it’s not Microsoft 365 that they’re just sort of proxying through and buying Microsoft 365 licenses for this is, they’re running exchange servers in their environment, right? And basically giving you customers a similar sort of experience that you would get with a SaaS service, right?
So they’re managing all of the infrastructure, the email servers, provisioning accounts, everything else. And you as a customer, you’re just. email service provided by Rackspace.
[00:08:02] CP: Let me ask you a question.
[00:08:04] PM: Mm-hmm.
[00:08:06] CP: I understand why somebody would, I mean, I disagree, but I understand , why somebody would use on-premises exchange over Microsoft 365. What does a company gain by using hosted, uh, Microsoft Exchange? That I don’t, I don’t
[00:08:34] PM: So I, I think it comes back to a couple things, right? One is, are they using Rackspace for other services already? And this is just yet another thing that they’re just using Rackspace for, right? That could be one. The second is maybe they have certain compliance regulations or other things which they feel cannot be fulfilled by.
I, I, I know it’s, but it’s a very
[00:09:00] CP: that
[00:09:00] PM: it’s a very niche.
[00:09:02] CP: no, no, but that’s why I ask that question because if you, if you’re not compliant sitting on Microsoft Services, why are you compliant sitting on somebody else’s services?
[00:09:11] PM: be, well, it might be that Rackspace has found a differentiator, right? Or provide the value add, right?
[00:09:17] CP: They must, they must. Maybe it’s uptime
[00:09:19] PM: Yeah. Maybe, well, maybe it’s uptime, maybe it’s replication. Maybe they’re offering backup. We’ll get to that later, right? But there are all these other things potentially you could be adding as a value add. right? In addition to just what Microsoft provides, right? Or maybe it’s like an e-discovery compliance style thing that they’re also providing in. It’s hard to tell cuz I don’t know what they offer fully for the managed services. Or it could be maybe there are certain data residency requirements that aren’t met by Microsoft today because of how they operate.
[00:09:52] CP: Speaking of which I’d like to off, I’d like to announce another sale.
[00:09:57] PM: Okay. What’s your next.
[00:09:59] CP: Uh, right now Rackspace stock is on sale. Um,
[00:10:05] PM: Oh, Curtis.
[00:10:08] CP: it went from, of a high of five this week down to three something. Um, yeah, it’s been, it’s been taking a hit. And, and, and also, um, There there’s been the announcement of at least one class action lawsuit, um, on business wire. Um, Cole and Van note announces filing of Rackspace ransomware data breach class action.
So let’s talk about what you know, cuz there’s like a half a dozen, I don’t know, plus stories or you know, various stories out there. What do we know about the outage? So,
[00:10:47] PM: So, so far, we know that on December 2nd they brought down their services. They said they noticed a security incident that mainly,
[00:10:57] CP: it took, it took them a day, as far as I recall. It took them a day to say it was a security incident.
[00:11:03] PM: Well, they noticed some. Issue and they brought down their environment.
[00:11:07] CP: right,
[00:11:08] PM: Um, I believe they only brought down their exchange hosted environment, not all the other services.
[00:11:15] CP: In fact, I think. Right. Yes. Yes.
[00:11:19] PM: it was limited to just that. And so they brought everything down and they kept everything down. Like you said, they investigated it. They then published to people saying we had a security incident, and I think that was December 3rd. and then they’ve been doing periodic updates, I would say, of where they’re at.
But it’s just more of the, we’re still investigating. We’re still investigating. We don’t have a time yet, but yeah.
[00:11:46] CP: the comment in the lawsuit, it referred to him as a numerous, uh, very opaque, um, you know, uh, announcements, right? That, that, that did, they didn’t offer. We’re looking into it, you know, that was basically what they offered.
[00:12:03] PM: and for companies, right? This is their email, right? . This is, uh, really critical for a lot of companies in order to do business and being down for six, seven days with no e t A on when they’re going to be fixed. Or what the process is cuz they haven’t even talked about what the recovery mechanisms are either.
And I know we’ll talk a lit bit later, Curtis, about sort of what they’re offering for a Band-Aid solution I guess, right now or, but yeah. So as of right now though, all those customers are a little hosed.
[00:12:35] CP: Yeah, I mean it’s been, we’re now on December 8th. This thing is still ongoing. I’m hoping that by the time this episode publishes the,
[00:12:44] PM: They’ll be back up and run.
[00:12:46] CP: one way or the other.
[00:12:47] PM: Yeah, but could you just imagine as a, like when you had your company, Curtis, if your email went down for a week, what would the impact be for you?
[00:12:57] CP: I mean, it just, it is just ridiculous, right? I, I, I suppose, well, we’d be, you know, really unable to communicate with outsiders, which is kind of the, the point of a company, right? Uh, maybe you’d be able to talk to a few people via chat and phone calls and whatnot, but email. Is such a critical part of, of a typical company that the idea of email going down for at this point almost a week or more, uh, is just unthinkable.
I, I, I can’t imagine the, the, the cost that they have, uh, that their, that their clients have incurred. They’re going to incur costs, they’re gonna inc. Reputation costs are gonna incur in financial costs. Uh, I think when this all comes out, this is gonna be, I think this is gonna be very bad for the likes of Rackspace.
Now let’s talk
[00:13:55] PM: oh, before you get, uh, it’s also going to be interesting. So, two things I wanna throw out there. One is, it’ll be interesting once this is all done, if they continue to stay in the hosting. email hosting business. Right. That’ll be one interesting thing. I did see a number that said that it’s supposed to be a 30 million a year business for
[00:14:15] CP: Yeah.
[00:14:16] PM: Right. Which isn’t peanuts. But at the same time, if you don’t have customers who are coming, right, I’m sure there are a lot of customers who are like, I don’t think I’m gonna stick around with Rackspace for my.
[00:14:30] CP: yeah, exactly. Um, so there was this, uh, or is this guy, his name’s Kevin Beaumont. So he is a security researcher and he was affected by the hosted exchange
[00:14:42] PM: Hmm.
[00:14:43] CP: and, um, or he noticed it or something. I don’t know. Um, I don’t know if he, um,
[00:14:50] PM: If he’s a customer,
[00:14:51] CP: I don’t know if he was effective or, yeah. But basically he, he just started poking around and he was looking at their.
Their email services and, um, he noticed the version that the, the, the version of Microsoft Exchange that they were running is apparently very old and is PR is, and it’s before there’s, there’s something called, uh, proxy Nutshell and that they were the patches that were available. Um, according to this, the version that they’re running is from August.
[00:15:25] PM: and it was patched in September.
[00:15:27] CP: yeah, which is passed in
[00:15:28] PM: I, I think I read something that said one server was unpatched, but I believe many of their other servers were patched.
[00:15:37] CP: Well, what I, what I, what he said later on in the article was, the way this works is all it takes is
[00:15:45] PM: One. Yep.
[00:15:47] CP: In fact, all it takes is one account, one compromised account on one. Compromised system, uh, and then you’re in. Right. Um,
[00:15:57] PM: which is I think how they moved horizontally across the entire environment, right?
[00:16:02] CP: exactly right. Uh, and, and he, and he goes on to, to, to basically point out to people who are, who actually have managed, um, Exchange to, or you know, whether they’re managing it themselves or somebody else is managing it to basically say, listen, you, you need to be running this past version. Right. Um, and in fact there have been two, there have been two versions of the software since that August 9th version that they have not patched,
[00:16:35] PM: and it’s kind of scary, like you think about patch management, especially for these critical vulnerabilities, right? It’s how do you schedule downtime to be able to apply the patches in the right order at the time, right? Maybe they applied it for all the other systems. This one maybe fell through the cracks, or maybe they had a problem trying to update this one, so they’re like, ah, we’ll just get to it on the next patch cycle.
You.
[00:16:57] CP: Yeah,
[00:16:58] PM: I, I don’t know what actually happened, by the way, this is all hypothetical, right? Or
[00:17:02] CP: Yeah, it’s all hypothetical. Should, yeah, that, that this is all, this is what appears to happen based on the information we have available. And we, we also don’t know. There’s no evidence. And again, Rackspace isn’t helping with its o o opacity. Would that be the, would that be the right word?
Opaqueness. O I think opacity seems like the right word anyway. Uh, they’re not really saying much. Um, but we don’t know. Also, we don’t know that,
like, It it appears that they had a hu that, that they had the, this unpatched or yeah, this unpatched server. But we don’t know that that’s what caused the outage.
But we do know well. Based on the information we have, we do know that they weren’t up on their patches, which is, you know, this is one of the reasons why you go with a hosted provider, right? Is that they’re, is that they’re gonna handle all of these problems that you don’t, you know, you don’t want to handle yourself.
[00:18:04] PM: Yeah, you’re like, I wanna focus on my business and not set it up and managing email infrastructure.
[00:18:12] CP: It’s very to, to, to put it mildly. It’s very disappointing on the part of Rackspace. Um, I, you know, I actually did some, um, some consulting work for Rackspace back a hundred years ago. Uh, and they seemed like a, you know, a smart group of people. I was in there to put in a, what would now be a competitor, a Druva.
Um, they were, They were, it was, it was a rocky install. That’s what I remember, . It was a rocky install. Um, but it’s interesting. So, so, so we have this thing with the patching that we don’t have it, it appears that they were behind on their patches. Why, why did they patch most of them, but, but not one of them.
Why? I don’t know. It means that they’ve, that would, that would suggest that they don’t have a, a strong patch management. Process. Right. Um, and then the other question is, why is it a week in and they haven’t been able to restore their services and there’s really only one valid answer to that question.
And by valid,
[00:19:21] PM: I think there are two. Oh,
[00:19:24] CP: okay.
[00:19:25] PM: No, actually you’re right. No, there are no valid answers. Really.
[00:19:30] CP: We, there are no valid answers, but there are, there are phrases that can answer the question that I posed. They, I don’t think they’re valid. I, I, I would say one, you know, they, they don’t want to pay their ransom. Right. Um, but the, and I hope for their sake from a company, History standpoint, I hope for their sake that this isn’t an extortion ransomware situation.
Right?
[00:20:04] PM: Yep. Where someone has exfiltrated some data and now they’re like, Hey, pay up. Or
[00:20:10] CP: cuz if you’re hosted exchange, you’ve got dozens, hundreds of companies in there.
[00:20:16] PM: Yeah.
[00:20:17] CP: By the way, they’re saying that this is, um, a portion of. Hosted exchange environment, by the way, but apparently the outage is affecting all of
[00:20:27] PM: Yeah. I think another,
[00:20:29] CP: yeah. Go ahead.
[00:20:30] PM: I think another valid reason that it could be right is, do you remember when we had Tony Mendoza from Spectralogic on when he was talking about how they recovered from ransomware? , right. I think a lot of it is, do you need to bring in those experts? Hopefully they had cyber insurance.
[00:20:50] CP: They, they according well, yeah. What we do know is that they have a quote leading cybersecurity firm, , and they’re helping them out. Yeah.
[00:21:00] PM: So hopefully they’re able, I thought it was Mandiant or one of those. I may have stumbled across that in some article
[00:21:06] CP: Yeah. I just saw the phrase, leading, leading company.
[00:21:09] PM: So I think. one is it takes time to bring them up to do the investigation, to figure out what they need to recover. So I’m thinking that doing that and then also making sure you have, because that’s the one thing that stood out for me from talking to Tony Mendoza, was you needed to have sort of clean machines that you could start using for restores.
Otherwise it’s just gonna continue propagating, right? And so finding the hardware, right, because this is a managed company, right? So they probably have servers in rack, so procuring the
[00:21:41] CP: You would think if they had, if, if, if a company would have. , you know, I, I’m, I’m guessing they have an entire space just for racks. Uh, uh, uh,
[00:21:52] PM: so, but, but finding the equipment right and then identifying the points in time, which are valid. Right. Standing up the infrastructure, the networking, right, making sure that there are no further security issues. Granted, seven days seems kind of long, but I thought that’s kind of what Tony said. It took ’em before they started getting up
[00:22:10] CP: two weeks.
[00:22:11] PM: two weeks before, well before they were fully up and running, but I think it was about a week before they could finally start doing restores and bringing up
[00:22:19] CP: I, I, I don’t remember by, by the way, that was a really good episode. Uh, it was ransomware victim tells their story. Um, the, um, I, it’s funny, you, you, you, you actually gave them a, that’s an actual valid
[00:22:35] PM: that that is a valid, that’s why I said yeah,
[00:22:37] CP: Yeah. Yeah. That’s pro. Possibly the only valid answer is, hey, This is hard.
[00:22:45] PM: Yeah. But they should be more transparent, you know?
[00:22:48] CP: they should be more transparent.
Why don’t companies understand that? I don’t get that. Um, but the worry is because due to the lack of transparency, is that, is that they don’t have backups. They’re so, so what, what have they done in the meantime for customers who just can’t do.
[00:23:12] PM: Yeah. So this is the thing that I saw in, that they were actually recommending is they’re like, Hey, we went and procured Microsoft 365 licenses for you, so you can stand up your email and continue operating. And they’re gonna help them set up, uh, what is it, forwarding rules. So
[00:23:31] CP: Yeah, so that incoming email will go to
[00:23:33] PM: And they have like the Microsoft, uh, team on standby to help customers, right.
[00:23:39] CP: Meanwhile, there’s, there’s all these, there’s all these email servers out there that have been trying to send mail to cus to, to, uh, Microsoft, uh, to Rackspace customers. And they’re we’re, we’re trying to send the mail. We’re trying to send the mail. Nobody’s taking it.
[00:23:53] PM: yeah. Well, and the thing is, even with this solution, right, it’s only going forward. You’re receiving new emails, right? All your old stuff is, oh, who knows what happened to that, right?
[00:24:04] CP: they did say that they provided for some customers an archive of, of emails.
[00:24:11] PM: I thought they said they were trying to get to that, but they don’t yet.
[00:24:16] CP: well, I thought that they gave it for some, but not all. Um, by the way, I was, I was, that, that’s a clue for me, the fact that they said they had archives of the email, but not backups,
[00:24:30] PM: I was looking at the article or from
[00:24:32] CP: Uhhuh,
[00:24:33] PM: we are working to provide customers with archive of inboxes where available to eventually import over to Microsoft 365.
[00:24:42] CP: Right.
[00:24:44] PM: So it may
[00:24:45] CP: the word archive is a bit,
[00:24:47] PM: concerning.
[00:24:48] CP: you know? Yeah. Uh, archive is different than backup. You know, we have an episode on that. Please go listen to that if you have it. Um, the, um, Yeah, I don’t know. I’m just not, I’m not, I don’t have a good feeling here on,
[00:25:05] PM: a little queasy in your tummy and it’s not from being sick.
[00:25:09] CP: Yeah. Yeah. Um, I, I don’t, I don’t know what they’re doing over there.
Um, like I said, it, it all, it all starts with why didn’t they have the service patch in the first place? That’s, that’s the first concerning thing. Um, the fact that we’re a weekend and they’re not saying that, you know, if they had said, l listen, we’re we are, we have to do a server by server scrub to, to verify that the, you know, we’ve identified the malware.
We now have to do a server by server scrub to make sure the malware isn’t anywhere else and we need to wipe the servers. You know, but they’re not saying anything. They’re just saying We’re, we’re working on
[00:25:51] PM: yeah. The other thing I wanna know,
[00:25:53] CP: any, yeah. Go.
[00:25:54] PM: the other thing I’m wondering though is I know you’re talking about backup, right? And restoring data. Do they not have a DR environment? You know, I know we, you know, we talk about this all day. Or maybe their DR. Environment was compromised as well. Right, and so we always talk about, right, you need a DR environment for situations like this where you can quickly spin up and continue operating rather than trying to go back and restore your data, keep those backups just in case so you can restore them.
But you really should have a DR environment so you’re not spending six plus days trying to bring up your environment.
[00:26:35] CP: But if you have a DR environment and you’re doing host exchange, you’re gonna have to be doing some sort of real time replication in order to, to have that DR environment be, uh, you know, effective.
[00:26:48] PM: Yep. And maybe they’re doing
[00:26:50] CP: not, I’m not saying I, I can’t.
[00:26:52] PM: Yeah. And, but even that replication, right? If it, so assuming the ransomware did not go horizontally into that DR environment, which is a big assumption. If you were doing exchange level replication at the application level, hopefully your DR site shouldn’t have been compromised
[00:27:08] CP: It has a delayed, there’s a delayed replication
[00:27:12] PM: And you’re also doing it at the application level, right? Rather
[00:27:15] CP: Right, right,
[00:27:16] PM: at the database object level. Right. So
[00:27:20] CP: Yeah. Yeah, that makes sense. It’s interesting, you know, it’s like, well, did it, did it attack exchange, or did it attack windows? We don’t know. We don’t know anything. We don’t know anything. Prasanna, why don’t we know anything?
[00:27:33] PM: That’s how these things go. Well, hopefully they publish more information. I’m not holding my breath for that though, but I think it could be a good learning opportunity because I wanna say that the US government, right, had a big push for patch exchange servers because of these specific issues. Like a couple months ago, I think there was like a cisa.
[00:27:55] CP: well that would’ve been helpful a couple of months
[00:27:57] PM: I thought so. I could be wrong, but I thought there was something
[00:28:02] CP: So, uh, I want to tack on. , um, basically put the, put, you know, on, on a related note to this, cause I’m tired of talking about Rackspace. It’s too depressing. Uh, I wanna talk about a company I’d never heard of before. Um, they’re referred to as, uh, south Korea’s, um, Google, and that is the name of the, I don’t know if, I don’t know if I’m pronouncing it right, but it looks like Ka.
um, like, it’s like, it’s like the way cocoa is spelled properly, but except with Case That’s the, so I’m sort of cacao, that’s how I’m pronouncing it. Like, like, you know, like the, the Bean for, for chocolate. Um, so they,
[00:28:52] PM: This article we found on the register,
[00:28:56] CP: yeah. On the register. Um, so, oh, it’s from si Simon. I know Simon. Hi Simon. I don’t know if he
[00:29:03] PM: what was it, title of the article just for
[00:29:06] CP: Uh, well there’s two articles. There’s Data Center Fire Takes Out South Korea’s Top Two Web Giants, and then a follow up article that was back in October. A follow up article is it’s 2022 and a Korean web Giant only now decided to write a Dr Plan So, uh, the first part is reminiscent of, um, the O V H fire, and that is that these guys apparently, What I’m, again, what I’m deducing there, there were two web giants, Neve and Cacao.
They both experienced service interruptions after a data center that hosts much of their infrastructure was shut down by a Sunday fire. So they, they are Google-like, but they’re, they’re, uh, in that they have like Facebook and messaging and a lot of different stuff like that. I don’t think they’re search engine type stuff, but they.
They and this other web giant, um, were taken down by this data center fire, which as we, again, we, we can only infer from things they, they’re, they’re hosted in somebody else’s data center. It’s not their data center. And, and it’s not a cloud data center
[00:30:23] PM: it’s like a cola.
[00:30:24] CP: a data center. Yeah, it’s like a colo and then the, the, the.
Sort of outcome from that is that this company has decided to build their own data center and they think now that maybe they should get a DR. Plan.
[00:30:41] PM: it’s, it’s surprising how often sort of backup and, and Chris, I know we’ve talked about it so many times, right? And, and we’ve had folks on talking about, okay, this is why it’s important to have a plan and to have stakeholders aligned. But no one ever thinks about DR and backup until it’s.
[00:31:03] CP: Yeah. Uh, so I guess what happened here is that the Korean government, south Korean government sternly criticized cacao and its c e o resigned. So they, um, they’ve unveiled a strategy to create tech teams, develop a business continuity plan, and built disaster recovery facilities. Um, you know, and the, the, the line in here, it’s a little odd in this day and age that a company of their size doesn’t have these things in place already.
But this idea that the blaze, you know, the, the, so the impact of this on everyday Koreans is that they all rely on this service. This is what, this is why I saw it as very much this related. They all rely on these services of like, , uh, communi for communications, right? Um, it says they’ve assumed the status of de facto telecommunications infrastructure, right?
[00:32:02] PM: And so when you go down, that affects so many people day in and day out,
[00:32:07] CP: yeah.
[00:32:07] PM: yep.
[00:32:08] CP: Yeah. So, um, so you have two companies where they were, they were hosting things, they were providing services. Many other people. In one case it’s thousands of companies. In another case it’s millions of, of everyday citizens that used them. And then both of them lost data due to some sort of thing. Uh, one was an attack, one was a fire, and neither of them had a plan to, to work around that.
[00:32:39] PM: And, and, and
[00:32:40] CP: What am I doing here?
[00:32:42] PM: they should listen to the podcast. That’s what they should be doing. But,
[00:32:46] CP: what it is.
[00:32:47] PM: but I think. I want to give maybe the benefit of the doubt that maybe they had planned for some types of outages, but they may not have planned for these types, these specific ones. Although in the case of
[00:33:04] CP: mean like the loss of an entire data center,
[00:33:08] PM: Right. May,
[00:33:09] CP: what a Dr. Plan is,
[00:33:10] PM: well, maybe they were relying on the other, like the host. Data center infrastructure deal with that. Great. Hey, should have asked a question. Yep.
[00:33:23] CP: Right? Like, you know, we talk about this all the time. Right? And, and, and by the way, you know, so I, I mentioned in the ear in the early part of the podcast that this was gonna touch on a couple, couple of hobby horses for me. One is that, You should not trust your supplier of services to be the backer of services, right?
[00:33:48] PM: Unless it’s written in contract or even then
[00:33:51] CP: What? No, no, no. I don’t think so. I, I think that, I think there’s a, um,
[00:33:55] PM: separation
[00:33:56] CP: there’s an, yeah. Yeah. That, that backup. Backup should be a backup, right? Um, that it should be done by somebody else. This is a, this is. Sort of a separation of powers, um, separation
[00:34:14] PM: Or shared responsibility model.
[00:34:16] CP: Shared responsibility, right?
I, I think it’s just best practice to have your primary stuff by one vendor and your backup stuff done by another vendor. That’s the way it’s always been done. , and then suddenly we start throwing that up in the air as to maybe that’s, you know, maybe we can save money if we use the services, right? I think back to O V H and what happened there where we, where they lost the data center, it turns out their backup stunk because they were just in the corner.
Right?
[00:34:49] PM: But I, I wanna challenge you that, on that challenge. You on that though, Curtis, right? I
[00:34:53] CP: right.
[00:34:56] PM: I think that, yes, 90% of the time don’t trust the vendor, right? The single vendor to do the right thing.
[00:35:05] CP: Yeah.
[00:35:06] PM: I feel if you can ask the right question. , do the tests, prove it out, get it in contract,
[00:35:14] CP: Yeah.
[00:35:14] PM: the full testing, which is effort, right?
It might just be easier to go do it yourself, right? But I think if you can get it in terms and verify it and prove it out, I Prasannally think it’s okay to go with, like to trust that single person
[00:35:32] CP: Do you have an example of that? Because I don’t,
[00:35:37] PM: AWS backup.
[00:35:41] CP: um, yeah, so, uh, Well, but, but that’s not the same though. That’s AWS Backup is a feature that you use,
[00:35:52] PM: Yes.
[00:35:53] CP: There’s no, there’s no AWS backup service that you rely on, and by service, I mean, you, you understand what I’m saying? It’s not like
[00:36:03] PM: it’s not a managed service.
[00:36:06] CP: it is, it is a feature that you use and you, you are going
to use. The snapshots and you’re responsible for managing it. What I’m talking about is relying, you know, putting all of the burden on the vendor. I don’t know of a service where,
[00:36:23] PM: But don’t they like, like if you, there are companies that do manage services, right? Where they take over your infrastructure and they operate it
[00:36:32] CP: yep.
[00:36:33] PM: case. So, but they’ll bring in multiple vendors. Right? Sometimes.
[00:36:38] CP: That that’s a different, that’s a different thing. I, I, let’s just say, you know, um, I’m, I’m just, I’m just saying if, if you’re asking me my druthers,
[00:36:48] PM: Yeah. No, no, it’s no.
[00:36:51] CP: phrase, I would rather you have one vendor, dut and another vendor do B,
[00:36:56] PM: A hundred percent agree a hundred percent.
[00:36:58] CP: Yeah. Um, I, I can’t think of a situation. So for example, I was going to perhaps, Think of, so it’s one reason, like, I don’t like whatever Microsoft is currently supplying regarding Microsoft 365.
First off, they don’t have a backup service for Microsoft 365. They do have an archive service. They have an e-discovery service, but they don’t have a backup service, so,
[00:37:24] PM: but if I took, uh, who is, who is, uh, Eric? Firstly, what was the company he worked for? Right? Would you say that they are a service, a managed service provider?
[00:37:39] CP: Yeah, they are. Uh, well they were, yeah, they were. Yes. They’re, cuz they do, they do private cloud, essentially infrastructure. Right?
[00:37:50] PM: but then they’re bringing in other vendors and they’re offering the entire package, right?
[00:37:56] CP: yeah. Good. Uh, good, better, best. Right. Again, if you’re asking me if. Again, my choice would be to have a different vendor do backup and recovery. Right? Because if you find out your vendor is an idiot,
[00:38:11] PM: You’re protected.
[00:38:12] CP: It’s, you’re protected. Well, the chances of you having two idiot vendors is less, right? Uh, so that’s, that’s one hobby horse.
And then the other hobby horse is, I can’t believe, um, you know, that just this idea that. People relied entirely on their, people just relied entirely under their backup. Clearly, nobody pushed them on it, right? Nobody pushed them on, you know, what’s your DR plan? All these people paying ’em all this money, and they’re not pushing them on, what’s your DR plan?
[00:38:49] PM: Yep.
[00:38:50] CP: You know? You know, I don’t, I don’t know. I don’t know what to say about.
[00:38:56] PM: Dr. Is Asmar.
[00:38:59] CP: I don’t wanna be victim blaming at the same time. You know, this is clearly Rackspace and ca cow’s responsibility. Uh, but if you are sitting there and you are using vendors and you are , you’re using vendors for stuff, you should be inquiring as to their DR plans and their ransomware readiness plans and
[00:39:23] PM: Was this covered in your book when we talked about talking about stakeholders?
[00:39:29] CP: I can’t, I can’t remember. I can’t remember if it wasn’t or not, or if it was or not. Um, mainly the book was about how to make sure you’re ready. Right? But I don’t know. So been, it’s a depressing, it’s a depressing episode. We should talk about puppies for.
[00:39:47] PM: Well, it’s not a depressing episode. I think this is one of those things, right? The silver lining is, this is lessons that other companies can look at, and hopefully they can be like, Hey, we really should be thinking about what we’re doing, and do we have Dr, do we have backup? Do we have a
[00:40:03] CP: Take this opportunity.
[00:40:04] PM: Yeah. Or the other thing is if you are using a SaaS service or who, or a managed service provider, ask them, what are your plans? Take this opportunity because everyone’s gonna be asking, Hey, what is going on?
[00:40:19] CP: Yeah. If, if, yeah, if you’re one of these people that I don’t understand, by the way, that’s using hosted exchange in somebody else’s data center instead of Microsoft 365, uh, by the way, feel free to contact us. Uh, you know, I didn’t mention, you know, that, that, you know, you can reach out to us at WC presson on Twitter or w Curtis Preston at gmail as long as Twitter stays up and, um, You know, we’d, I’d love to, if, if you’ve got an answer as to why you would use hosted exchange over Microsoft 365, uh, I would love to know it.
But if you have that, yeah. It’s time to reach out to that vendor or hosted anything,
[00:40:57] PM: anything. Exactly. It’s not just email.
[00:40:59] CP: Yeah, I mean, I, you know, this, this podcast is hosted, backup Central is hosted, but um, I know what the backup setup is. Hello? Kopi
Coffee always wants to get on the podcast. Um, yeah. So I know what the backups are on that because I, I make them happen. Right. But, um,
[00:41:24] PM: And then for your Backup Central’s website too, right? You have a hosted WordPress site, but you’re doing the backups yourself.
[00:41:32] CP: Yeah. Well, it’s part of the Sea Sea Panel Sea panel. It’s, it’s run by C panel. And then C panel allows me to configure a backup, which that backup, daily backup is then copied to, uh, S3 actually, right? And then, and it’s automatically deleted after a certain number of. Days. Right. Um, so it’s not, it’s not, the backup is not stored.
So that’s another example. Kind of like what you were talking about, like aw s backup. I don’t rely on C panel or my hosting provider to do the backup, by the way, they do offer a backup service. See, I do eat my
[00:42:11] PM: That. That’s why I was telling, that’s why I brought that up, that example, right.
[00:42:15] CP: They do offer a hosted backup service. LiquidWeb is the name of my hosting provider, by the way, love.
Been with them now for a really long time and I, you know, they are a actual managed server service provider for hosting and, um, the, they do offer a backup service and instead I use the features of C Panel to create a backup, which is then, um, you know, sent over to s3. And, uh, I think I keep 90 days even though I think that’s ridiculous.
And my Amazon S3. Is, uh, like $2 and 83 cents, something like that.
[00:42:51] PM: Maybe you should be looking at S three ia. Your cost might drop
[00:42:57] CP: Yeah. Yeah. Cuz I, you know, I’ve, I’ve used my backups two or three times, but, um, yeah, I don’t know if it’s, I don’t know if it’s possible, but
[00:43:08] PM: the saving of 35 cents.
[00:43:11] CP: Yeah. But, uh, yeah. Anyway, just l just investigate your vendors. Will you check to see if they’re doing, you know, I, I wish the best for Rackspace. I wish the best for their customers.
I hope that by the time you hear this, this has all been sorted out. If it hasn’t, dear Lord. Anyway, well, alright, uh, hope you enjoyed this episode. Remember to subscribe so that you can restore it all.
