Ok, so maybe not the most interesting topic. 😉 But we promise you, this episode has a great story that involves Mr. Backup being kidnapped by a client, basically because he had a backup level issue. Learn about full backups, incrementals, cumulative incrementals, differentials, numbered levels, tower of hanoi backups, and why all this matters. It turns out it matters a lot more these days for structured backups than filesystem backups, which have typically gone to an incremental forever setup.
Hi, and welcome to Backup Central’s Restored All podcast. I’m your host, Debbie Curtis Preston, a k a, Mr. Backup, and I with me, a guy that I’m hoping knows a little something about plumbing. Prasanna. Malaiyandi, how’s it going?
I am good, Curtis.
how’s that? YouTube? How’s that? YouTube, the, the YouTube knowledge that you have, does it cover plumbing
kind of, sort of. Unfortunately it’s not something I’m as familiar with, but depends on what you’re looking to do.
Yeah. I gotta replace tonight. I gotta replace a couple of valves. Um, Shut off valves. Um, one of them, I, one of them is on the, the, it’s the one that goes behind my fridge and in order to like, it’s like I, I can’t easily, it’s a, it’s a, what do you call it? It’s the, the threaded, no, it’s not a compression fit.
It’s the, it’s a threaded pipe. Like, uh, I really gotta get in there and get a hold of the, the upper part of the pipe, uh, in order to hold that pipe still while I’m undoing it. Cuz it, my first attempts at it did not, I don’t want to be cracking off a pipe up in the wall.
Yeah, I was just gonna say that’s, yeah, the last thing you want to do. I was thinking for the shutoff though, I’m assuming you took the handle off
The handle
to give you more space
it? No, there’s no handle. It’s just, I mean, the handle’s like integrated. It’s really small. It’s not a space problem. It’s just, uh, it’s um, you know,
Or they also say heat.
Yeah, give it
So when I saw my, oh, so when we had our water heater replaced, I was watching cuz I’m always curious to see what people do. And our pipes were really, really bad at old. because the last time our water heater was replaced was like 15 years ago. And so the guy’s like, you just heat it up really, really hot and then you tighten it first to break the stuff off and then you loosen it.
Oh, interesting. Yeah. Maybe I’ll see. I’ll take a look at that.
Yeah.
Uh,
took like in a settling torch or a butane torch and then heated up nice and hot, and then he tightened it just a little.
what, I’m not gonna be able to do that. You know why? There’s this
Oh, the drywall. Oh,
Plastic, worse than dry.
yeah.
Um, so hey, we’re gonna talk about, uh, we’re, you know, in our continuing, um, backup to basic series, you know, we talked about, like recently we talked about a difference between backup and archive.
, in a previous episode we talked about protecting your backup data. This one, um, you know, this one is, um, it’s about backup levels, right? And, you know, it’s, honestly, I’ll just say this isn’t the most exciting topic, Um, and maybe, maybe, we’ll, maybe we’ll make it a short episode if it’s not interesting. You know, we’re not gonna make, we’re not gonna make a long episode cuz this isn’t gonna be one that’s gonna be, uh, littered with stuff about cyber attacks and things like that. Backup levels
really straightforward.
really straightforward.
Yeah. Uh, I do want to help for people that sometimes struggle with the different types of backups. Um, I want them to help them understand. I, I’d say that, you know, if you just go and I’m, uh, throughout our usual disclaimer Prasanna, and I work for different companies. I work for Druva, he works for Zoom, and the, this is an independent podcast. It’s not theirs. And it’s, uh, the opinions that you hear are ours. Uh, be sure to rate us, uh, go to your favorite, um, pod catcher and uh, scroll down to wherever they got the stars and give us all the stars.
And the best thing is if you give us a comment, those are always nice, especially on Apple Podcasts, which is. the primary way that people listen to us over here. And, um, also if you wanna join the conversation, I am w Curtis Preston gmail and also WC Preston for now on Twitter. I dunno what’s
do you have a master on?
Twitter man, I don’t have a Macedon account. Uh, I am interested in that. We’ll see if, I don’t know if the average person can, can
I don’t think so. From what everything I’ve heard. Yeah.
It took me a minute, right? Twitter, you know, you sign up for account, you’re done right here. It’s like, well, what community do you want it attach to? I’m like, oh, it’s so complicated.
I don’t wanna know what community, you know? Um, we’ll see. But, uh, anyway, at WC Preston on Twitter, maybe I should sign it for my, at WC Preston on Mastodon. Make sure I get that.
I think you were throwing out the disclaimer though, because for this chapter, if people want to actually read the chapter and understand all these terms, right. Druva is offering a free ebook version.
I, you know, that wasn’t why I was saying it, but, we’ll, we’ll do that. So we are discussing my book, modern Data Protection. There’s a picture of it with the little armadillo for those of you watching on the video version on backup central.com. And uh, you can get a free ebook copy of it by going to druva.com/ebook.
That’s d r uva.com/ebook. And. while supplies last, or while the contract lasts. So, you know, this, this, this episode will live for a long time. If, if you go there and it’s not there, it’s not my fault. So, uh, first off, you know, again, this is pretty basic, but. You know, you should know what a full backup is.
Obviously. That’s it. It backs up everything, right? Typically. Well, not typically. You always have to do a full backup before you can do any, any kind of incremental backup. That’s the, that’s the next type. The, there are amazingly, a handful of different types of the traditional incremental
but before you get to incremental, so.
Yeah.
Even for fulls, are there different types of fulls?
Well, there’s not different levels for fulls, but there are different ways that fulls get done. Right. Like you, like we could be doing a file system backup or we could be doing a block level backup, but either way you’re still backing up all the things, all the bites. Yeah. Um, the, uh, and it’s, it’s the full is the thing that puts the most amount of, of stress on the thing being backed up.
Right. The, this is why we try to limit them whenever. , and this is why, this is why backups broke when VMware took off because everybody was just doing full and incremental backups and they weren’t coordinating them. And you know, it was just you,
lot of resources being used.
Yeah, exactly right. And um, if you do, if you do a nightly full backup, or let’s say the weekend full backup, and you do ’em all on Friday, And you’ve got 20 different physical servers, no big deal.
But if you do 20 full backups on 20 VMs that are all on one physical server, this is a problem. Right. Um, so I’ve got a couple different types here. There’s the, the typical incremental backup, which is, it basically, it backs up everything since the last backup of whatever kind. And typically what what you do, and this is with backups that have levels, you do a full backup, then you do a series of incremental backups, and then you do another full.
right? That’s the, that’s the sort of the typical way. There is something called a cumulative incremental backup. Um, I prefer that to the term differential backup, and I can talk about that in a minute. But a cumulative incremental backup is essentially just that it backs up everything that has changed since the last full different, um, different product.
Uh, do different things. Uh, the differential backup is a term that you see a lot in windows and, um, I, I, and I, I don’t use it because I find the term, um, the, the, because depending on where you ask the term differential means different things.
And so I, I don’t like it. Right? So I like the term cumulative incremental.
Even if your backup product doesn’t use it, just learn what, whatever they call it. What?
wait. So as far as I understand it, , you have the thing that’s like full then. So say I do a full on the weekend, right on Monday, I do, uh, incremental,
Uhhuh.
right? Which will be the differences for whatever happened between Sunday and Monday,
Yeah.
right? Tuesday I do another incremental
Yeah,
right? That’ll be the differences between Monday and Tuesday.
yeah,
And then Wednesday I do another incremental. That’s the difference between Tuesday and Wednesday. Now, for different, uh, for cumulative incrementals, it’s, I did my still, I’ll do my Sunday, full Monday I’ll do my cumulative. Incremental, or yeah, incremental, which will just be Sunday to Monday, differences.
Tuesday, when I do the cumulative incremental, it’ll be from Sunday to Tuesday
Correct.
and Wednesday when I do a cumulative incremental, it’ll be the differences between Sunday and.
Correct. And so what, what we used to do back in the day was I would do a monthly full weekly cumulative incrementals and daily incrementals, right? So that way, even if I’m 28 days into the month, I would only need three backups to get the job done. . Right. Um, I, I don’t use the term differential.
Differential is what a lot of places call accumulative a criminal backup. But again, different products use it to mean different things, so I don’t use it. Um, but interestingly enough, what we haven’t talked about is backup levels, which is the term that, again, you don’t see too much these days. Have you, have you, are you seeing it?
Very, very rarely do Does anyone talk. The only place I’ve seen levels really is when you’re talking like database backups, but most.
what? That is true. Right?
Yeah. But most backups software itself no longer talks about the levels. Right. It’s either fulls or incrementals.
I think it was too complicated. Yeah. The basically if to, to do. To do a full on a weekend and then daily regular incrementals, you would do a 0, 1, 2, 3, 4, 5, 6, 7, right? Well, seven. And then maybe, um, so then, and again, this is, this is impossible to do without a whiteboard, but if you did, um, a zero followed by all the way up to the five, if you did a one again following that five.
you would get all the changes since to zero.
Hmm.
So it was accumulative. Incremental, right. But what some people would do is they would do this, um, this, there, there’s, there was this thing called towers of Hanoi, which, um, it, it comes from the game.
like
Yes. Yes. It’s like the thing with the game. Yes,
three. Yeah.
Comes from that game.
Um, and so the, and again, this is really, don’t try to understand this, I’m just putting it out there just for, this is for the true backup anoracks, so who’s gonna
Daniel Rosen Hill. We’re looking. Daniel Rose Hill. We’re looking at you.
Exactly. So, um, let’s see. So what I’ve got here is, , um, a towers of Illinois Schedule 0 3 2 5 4 7 6. And what it does is it, is it like every file that’s changed ends up on two different, um,
Mm,
two different, um,
days.
backups.
Yeah,
Um, and um,
you don’t run the risk of if a backup is bad, you’ve lost the file, which could happen if you just had a full and you never did another incremental again.
Yeah.
Or sorry you never did another full again, I, I think one thing we forgot to mention Curtis though, is the benefits of doing a a, we might have talked about it implicitly, but the benefits of doing a cumulative incremental versus a normal incremental when it comes to restores.
Well, you know, here’s the thing. The benefits were much bigger when we were using tape.
Hmm.
I, I think that as, and, and what we’re gonna discover is that this is why a lot of this has, I think, gone by the wayside is because, because when we were using tape, you’d gr you’d grab the full tape, you would grab the cumulative incremental tape, and you would grab any incremental since then.
So you’ve got maximum like eight tapes if you did a full, followed by 30 incremental. , you would, um, you would need 28 tapes, you know, at the end of the month. So this way you only need like a handful of tapes. And what it allowed you to do was it allowed you to do less frequent full backups. Because it was a, it was a, an extra cost. You’re like, well, by doing a weekly cumulative incremental, I could do a monthly full backup instead of a weekly full backup. Thus really reducing how big the backup system needed to be, how many tapes I needed to use. Um, And, and by doing that cumulative incremental, it made the restore much, much faster.
I would argue if you have a decent backup system today and you’ve got a monthly full and 30 days of incremental, or even 60 days of incremental, the difference between restore times, um, depending on how you store the backup data, shouldn’t be as significant as they were back in the day when we were loading tapes, cuz every tape loaded was like two minutes
Yeah. But,
to the first.
but I do wonder though if when you are doing those restores, depending on what you’re restoring, like I could see a file system maybe not being a big deal for doing the 60 days. I’m just wondering like database backups, right. To do those incremental restores.
Yeah. Um, I think that’s a val, I think that’s a valid. Uh, concern, right? This is why to this day we still do more frequent
Fulls that need it.
on, on database backups. Right. The, that’s, that’s a good point. Um, and since a significant portion of the world is on, is, is structured data then, uh, yeah, that’s a really good point, right?
So, um, the. So now we’ve got a just some different types of, so we’re done with the levels. Now we’re talking about different types of backups. What about a block level incremental backup? What is that?
Block level, incremental. Are you referring to like database slash v. Where you’re just copying the differences in whatever the application block size is, rather than block backing up the entire image or the entire file potentially again.
Yeah, so basically this is the difference between, and again, this really only applies if you’re looking at things at the block level, which you tend to do at the database, but there are some file systems that are so dense that backing it up at the block level would be faster than backing it up at the, at the file level.
Yeah.
And so you can do a block level incremental, basically. The point is, when I’m doing an incremental. A file system in increment, a file system incremental. I’m backing up the files that have changed since the full, if I’m doing a block level incremental, I’m backing up the blocks that have changed since the full or since the previous bite.
Right. And then here’s something that hopefully you’ll know something about, and it’s called Source side deduplication. This is another way to do incremental. What can you tell us about.
Yeah. Source side de-duplication. So when de-duplication appliances came out, um, what you ended up happening is whenever you would do a full right, you were talking right Curtis, about doing weekly fulls, daily incrementals. Every time you did that full, probably 98% of the data was the same. So on the source you would read it, you would send it over the network, it would go to your de-duplicated appliance and it would throw away 98% of the data.
And so source side de-duplication is, Instead of sending all of that data to be thrown away. On the appliance, you basically split the deduplication algorithm such that you’re running some of the processing on the source. So you figure out what is duplicate, usually at a smaller block level or some other, uh, granular size and only send the unique blocks over to the target appliance.
And the rest of the data is sort of reconstructed, if you will. There are different methods you could look at. It’s sort of rebuilt because you still need to account for those pieces of data, even though you haven’t technically sent the data over the wire.
Yeah, so if we think about it, like the only reason ever to do an incremental backup is to save something, save bandwidth, save CPU time, save storage. And what, you know, what you’re talking about is even when you had target site dation with like data domain and the like, you, you. , you save storage. You, you can do full backup every day and you save storage.
But you were still using up all that CPU in that network by doing Source I duplication and making that decision before you send it. You save CPU time, you save bandwidth, you save storage, you save all the things. Right? Um, source ID is not as prevalent as targets. I dedup. There’s, there’s only a handful of companies that, that do it, but uh, they are,
Yep.
Um,
it does require more integration into the backup product on the source in order to take advantage of it.
exactly right. Uh, for the record, Druva is a source site duplication product. So the next thing we talk about here is called synthetic full backups, uh, which is and interesting. I remember when this concept first came out, again, we’re trying to minimize the impact of the backup. on the system, we’re backing up.
So the idea was we already have all of the bites necessary to create a full backup. Why do we need to go get, you know, we’re doing a, we’re doing an increment. We’re doing the next backup and 99% of what’s on the system we already have on disc somewhere. This is really a disc thing. Uh, we did it on tape too, but it was a lot more work.
Yeah.
So we’ll go get the 1% that has changed. and then we’ll create a full backup by copying the stuff we already have and the stuff that we just got. So we create a synthetic full. Um, have you, have you run into those out in the wild?
no, it’s very common, uh, especially when you think about VMware images.
Mm-hmm.
depending on how it’s being backed up, a lot of folks would back up a VM as incrementals and then, because like you said, most of the data is the same, they would sort of synthesize a full image on the target side in order to have that full copy.
Therefore, when you need to restore, you have that. What image you could pull from or would a lot of people start doing is you take that one image and you can now mount. , right. And access it directly. Instant access is what simple some people call it. So you can spin up the VM from your target system and then vMotion it over or whatever else.
So you have shorter RTOs than trying to first restore the data before you spin it up.
and there, and there’s a couple different ways that you can create a synthetic full. You can either do it by copying, which is what I was talking about before. There’s also this concept of virtual synthetic full, where you can just create one. , you know, magically, right? Like the, I know that again, data domain I know supported that on the back end, right?
That if the backup product could say, you have all the bites you need to make a full, why don’t you just put ’em together and, you know, everything’s beautiful,
You know what else does it too,
what’s that?
Oracle? When you do incremental merch,
That’s right. Incremental merge creates. Exactly, exactly. Um, but here’s the thing. Wherever we can get rid of full backups, I think we should get rid of full backups, Right, because they’re just the dumbest thing. Right? They’re just, they’re just, they’re, we did ’em because that’s the way we did ’em. And we still do ’em because that’s the way we used to do ’em. Uh, so I’m a fan of, go ahead.
I, so I, I wanna be careful, right? I think we should. Get rid of the concept of backing up full copies of data every single time or on some increment. I a hundred percent agree with that statement.
Isn’t that what I just said?
No, no, no, no, no. But, but I don’t agree with, but there’s still a necessity to have the equivalent of periodic fulls, right?
Something that represents a full backup. Now it could be stored on de-duplicated systems to save storage space, cuz most of it’s the same, but I’m just saying you don’t wanna go the route of, it’s just a full once a year and just incrementals physically stored on disk. So using virtual synthetics or some other mechanism to end up with full copies periodically, I think is beneficial.
So I would argue that that’s what Incremental Forever is a true incremental forever system. Right. That basically you, which is what, what I was about to talk
Okay. Okay.
Um, and that, you know, the idea is store each backup in such a way so that a, any restore from any backup, you just, the, it’s stored in such a way that it looks like a full right and a restore from it behaves like a full, I, I don’t think we need to keep making fulls or make or synthetically creating an occasional full.
I think that if you design it from the beginning, so that you do incremental forever, and then you store that data in such a way that every backup is essentially a full
Yep.
from, from a way it behaved the way it behaves during a restore, then I think that’s as good as it’s gonna get. Right.
I, I think that works in the cases of file systems and virtual machines. I do think in the case of databases, it is a little database is always that oddball.
I had the caveat back earlier where I was like, wherever we can, right. Um,
Databases have so many nuances with fulls and incrementals and log archived, redo logs, and all the rest that I think, yeah, for everything else, a hundred percent agree with you, Curtis
Yeah. Um, and there are, there are a handful of products that back up this way that basically, said full backups is so from the tape land, right? Except for database backups. Right. Um, generally speaking, there’re gonna be products that have come out in the last 10 ish years. Right. Um, products that have been around for 20 years.
They’re gonna have spent time in the tape land and they’re gonna have parts of their architecture that are left over from the tape plan.
I also do wonder if some of that is just the people who are banishing the systems. It’s something that they are used to, and so change is hard, you know?
is hard. Inertia is something. Yeah, exactly. And I would say like if you, if you’re doing source side dup with incremental forever, I think from an efficiency perspective, assuming that the way you store it, everything behaves like a full, an efficiency perspective, I don’t see it.
I don’t know how it could get any more efficient in that. Right. You could maybe argue the block level incremental forever where you’re. Right. So there, there are like c d P and near CDP systems that’s continuous data protection where they’re doing block level changes and they’re replicating that.
That’s also quite efficient. It’s not source I, it’s still, it’s still, so I would say, so I’ll restate my statement. Either source iddu or block level of replication and then incremental forever stored in such a way so that everything behaves like a full.
Yeah.
you know, that includes a handful of products , one of which is Druva.
But, um, you know, there, there are other products that behave like that. Like, you know, your old employer, NetApp, right? That was a block level incremental replication. Um, yeah. Um, so I’m gonna throw out one, um, this is gonna be one of our shorter episodes. I think. We’ll see. We’re almost done. I’m gonna throw out one, like, uh, what’s the cranky old man?
Comment
get off my lawn.
this is the archive bit and Windows. I hate the archive bit and windows. Um, so what is the archive bit? It’s a flag on every file that when your backup software backs up that file, it can unset that flag. If the file is new or it changes, the archive bit is set. And then when the flag, uh, when the back, the, uh, backup software backs it up, the archive flag is unset.
Why.
This is the old way and it’s still the current way. Uh, not looking. This is if you’re doing a file system level backup in Windows. Um, I think it’s so dumb for multiple reasons. The first of which it’s name,
Archive.
it should be the backup bit.
Yeah.
They called it the archive bit anyway, but that’s not really my problem is you can’t, if anyone comes along with like, any, cuz there are a number of like third party tools that can back up your Windows system.
If you run a third party tool to create your own backup, like let’s say you, you know, a lot, some people don’t trust the backup system, right? So if you, if you being an admin go and decide to make your own backup and you use something that uses the archive bit, it will clear the archive bit on all the files that have changed since the last full. and then when you go, when the real backup software comes in, it actually
not gonna find,
until the next full backup.
and I bet people don’t
problem with it.
Yeah.
Yeah. And of course people don’t notice it. Um, and so I’m gonna go back to the story, um, that I talked about. I don’t know what the, the one about me being kidnapped, was that in this episode or was
That
was a previous.
Okay, well I hinted at this story from um, this other, from the other episode, but here’s the full story cuz we got time And you know what, if you don’t want another old backup story then you know, thanks for joining and uh, see you next episode. But this is a good story and if figures in a good buddy of mine, Rob Worman. I know Rob listens to the podcast. , he figures into this story. So I was at this large, you know, national entertainment company and I was helping them to redesign their backup system. We were doing a massive redesign. And one of the, and one of the things, you know, like we pushed them from, you know, weekly folds to monthly folds.
We, we changed all their multiplexing setting. We changed what, uh, we changed how. , the schedules were, I mean, we changed everything and we were doing this to, to do massive efficiency change. They were running like, it was like 18 tape drives simultaneously, and I told ’em that that was their pro, they were gonna buy, uh, two
Oh, this is a sh when you’re, you were, you’re talking about the shoe shining problem and they were trying to tape goes too fast for what they need.
Yeah. And so they were gonna buy two new tape drives and that was gonna fix the problem. And I’m like, no, it’s not. It’s actually gonna make it worse. And I explained to ’em and I’m like, listen, give me a few weeks and I’ll redesign your backups to submit. It’ll cost you less than what those tape drives were gonna cost.
Cuz they were expensive. These were the storage tech tape drives, like the 90, what? I can’t remember the tape name. Tape names. But anyway, um, so we did this and. , um, and things went really well, but so I knew about the archive bit. This is all about the archive bit. I knew about the archive bit and I knew that if I had, I basically left their parallel backup system.
I left their parallel policy.
Up and
I left their existing policies running in parallel. Well, I was gonna do the regular backups, uh, during the day, um, and something like that. It’s been a while. And, but I knew that I would screw them up if I had the archive bid on both systems. So I used, this was net backup. I used a feature of net backup that said, okay, don’t use the archive bit on.
On the sink and what I didn’t notice or what happened over time, over about two weeks is the arc, the incremental backups got bigger and bigger and bigger and bigger, and they got huge
on your new
and they on the new system and they got so big that they weren’t finishing. By the way, when I got there, they were backing up 28 hours a day.
So my, my opinion was no matter what I was doing, it still wasn’t gonna be that bad. , but backup, but the backups weren’t finishing and I didn’t understand, right. So, so we shut off the old backups. We’re like, okay, we we’re, we’re beyond the point of no return. We shut off the old backups, backups still weren’t finishing.
And uh, you know, I was working through diff, you know, multiple levels of support. And I actually at, at one point ended up with my buddy Rob,
Hmm.
I’m like, dude, I don’t understand. And we, you know, we figured out that when that. Doesn’t use the archive bit. What it does is a, again, this is, this is a hundred years ago.
I don’t know what it does today. If it doesn’t use the archive bit, it would, it would traverse the directory tree and then it would, as soon as it encountered a directory that whose modification time had changed, it would back up everything in that directory tree,
Okay. Including data that does, yeah,
including data that hadn’t changed, right? It, it was really inefficient. And so it tried to add that level of efficiency by working up the, you know, so long story short, the back, the incremental backups were, were getting close to full backups. And so once we figured that out, we’re like, okay, okay, okay, okay, we figured it out.
We’re gonna tell it to use the archive bit now. And, um, and we turned on the, we turned on the archive bit. and it was the same the next day. Right?
Okay,
Now I look, now I look, and meanwhile this, this manager, he is looking at me, you know, looking at me over the, you know, he’s beginning to think I’m a complete moron.
And, um, he, um, I said, look, I, you know, we figured it out. This is what, you know, he’s like, but it, but it didn’t fix the problem. And that’s when I was really talking to Rob. I’m like, I don’t understand. And again, I was on the edge here. I would, no one had done what I was doing, you know, and so, so what we figured out was when you tell net backup not to use the archive bit, it doesn’t use the archive bit.
Now, what that means is that when it backs up a file, it doesn’t clear the archive bit, which was the whole point of me using the thing. And so that meant that
The
we,
when you turn it on
to run the, we had to run the one backup. that would use the archive bit to clear everything. But because we had run one in a long time, it was also gonna be really long.
And so then it’s like, okay, tomorrow night is gonna be all better. And that’s when the boss said, okay, smarty pants, no one’s going anywhere. He was, he was really upset at the level of instability. And I can understand him being. But he was really upset at the level of instability in his mind that I had introduced into his system.
Nevermind that the new backup system was backing up 50% more data than the old system because I changed the, the inclusion factor. And they had, they had, they had left out a lot of really important data. So it went from 20 terabytes to 30 terabytes and. Nevermind the fact that we were now creating two copies, whereas before he wasn’t even getting one copy done.
So I was feeling pretty good. But all he sees is his
Stability. Yep.
Yeah. And so he’s like, well, nobody’s going home until, and literally he just, he stayed with us. I mean, to his credit,
Okay.
stayed with us and like he ordered pizza and stuff, uh, but he literally wouldn’t let us leave. I couldn’t even say like, well, I kicked off the back house.
We’re gonna go get dinner. I ain’t nobody going anywhere.
He’s like bathroom. You already had your bathroom break.
but we’re gonna sit here and Yeah. And so we were there until pretty late. Um, I don’t really remember how late, but I remember like, it was like a whole second day, you know, like it was, it was a long, long, long, long night.
And that my friends, . That’s what happens when you don’t understand backup levels. Anyway, All right. Well I think this is enough torturing people with backup level information.
I, I a hundred percent
talk about, go ahead. Yeah. Next week we’re gonna talk about metrics.
Ooh,
yeah. I think it’s a little more exciting.
yep.
It’s backups. Like how exciting could it be
Come on, Curtis. What are you talking? Telling our audience, our listeners,
Woo. I dunno what to tell you. You know what, maybe you’re excited as you’re as excited about backups as I am, maybe, to which I say, welcome to the party. All right, well, uh, thanks for listening folks, and remember to subscribe so that you can restore it all.
