Hear the incredible story of Albert Uy, who was handed a two-year project that had already wasted 18 months. They told him he had four months to complete it. He looked at that timeline and told them there was only one way he could do, by switching horses in mid-stream. Not only did they finish it in time; they finished it early. Albert has also managed backup teams for many years, so before we cover this project story, we talk about backup has progressed over the years. We also talk about how much easier his job got once he started using Druva has his backup solution.
Transcript
This week on restore it all, we’ve got a backup practitioner that has used multiple backup products, including the one where I happened to work. I hope you enjoy this week’s episode. Hi, and welcome to Backup Central’s Restore it All podcast. I’m a host w Curtis Preston, a k a, Mr. Backup, and I have with me my colossal Chore completion celebration companion Prasanna
[00:00:44] Prasanna Malaiyandi: Woohoo, .
[00:00:45] W. Curtis Preston: Malaiyandi
[00:00:47] Prasanna Malaiyandi: How does it feel, Curtis that.
[00:00:50] W. Curtis Preston: my Lord. it’s, you know, uh, there are no words. I, I, I will technical, you know, basically, you know, to, so to be honest, I have finished the part of the project that. everyone can see. I’m still sitting on carpet in this one room, which is my office. Um, and uh, but everything else has been completed, which means that we’re gonna have guests over for Christmas.
So that’s
[00:01:21] Prasanna Malaiyandi: And, and this is basically for the listeners who may not realize this is Curtis’s. I’m going to replace all the flooring in the downstairs by myself and take up tile, which has been. Glued down with super glue, basically
[00:01:36] W. Curtis Preston: Yeah.
[00:01:37] Prasanna Malaiyandi: and
[00:01:38] W. Curtis Preston: was, there was what? There was carpet, there was tile, you know, large, large format ceramic tile and also Pergo in one room. So I took all that up and it was approximately 1500 square feet of luxury vinyl tile that I laid down.
[00:01:57] Prasanna Malaiyandi: it was like three pallets that were sitting in your garage, right?
[00:02:00] W. Curtis Preston: Yeah. Uh, and I’d say about, um, I don’t know, about a 25% into the project I was. what have I done? Like, because I, I actually got hurt. Like, I, I, you know, I’m, I, you know, I’m in my fifties, I’m down on my knees. My, my, that, that was a mess. Um, and then I got sick, so, but yeah, the really funny thing is, um, for the, for the long time listeners, you know, that sometimes my granddaughter comes on here and at the beginning of the project when she saw how much work it was, her comment was, this is gonna take like five months. and I was really insulted at the time, but, uh, it took actually a little over five months due to my injury and my, and my sickness. But, but yeah, at this point it looks beautiful out there and I, I, I don’t think the mic would pick it up, but right now, just outside my door is our new, uh, Roomba that we bought,
[00:03:00] Prasanna Malaiyandi: To keep it.
[00:03:01] W. Curtis Preston: of that.
Yeah.
[00:03:02] Prasanna Malaiyandi: it and span
[00:03:03] W. Curtis Preston: Yeah, exactly. Because the one thing I noticed with having, you know, the hard floor versus the co carpet is that you see and feel every little bit of dust. And so, um, that, uh, so yeah, so we, we added that, that it’s, it’s now done, its first full vacuuming and now it’s trying to do its second. So it’s a fun, it’s a fun time.
At the Preston household
[00:03:27] Prasanna Malaiyandi: Well, at least it’s all done before Christmas,
[00:03:30] W. Curtis Preston: Exactly. Um,
[00:03:32] Prasanna Malaiyandi: I’m sure your wife is quite pleased.
[00:03:35] W. Curtis Preston: yeah, she really likes, she really likes decorating the house for Christmas and we really weren’t able to because of, you know,
the fact that yeah, it was a construction zone up until just like two days ago.
[00:03:46] Prasanna Malaiyandi: so now here’s a question, Curtis, are you gonna quit your day job and go do INS flooring installation as a full-time gig?
[00:03:54] W. Curtis Preston: absolutely not. There was this guy, and by the way, shout out to there, there’s a website called for anybody who’s thinking about doing it. It, it wasn’t like difficult per se other than the fact that you’re on your hands and knees the whole time. And, and one thing that was invaluable was this, there’s a guy, um, who.
Has a site called, so that’s how you do that.com . And he has a service where you give him like, I think it’s like a hundred bucks. and then he guides you through the project. You give him your floor layout and he’s like, start in this room, go to this room and here’s the room where you’re gonna work backwards.
And um, you know, and that was invaluable throughout the entire process. Um, so shout out to him. Um, yeah. Thanks Joe. Anyway, so let’s get on to our guest who’s sitting there going, what kind of podcast have I come on At this point, I didn’t sign up for home improvement stuff. He has been in it for over 20 years and was most recently the Vice President of Technology, architecture and Innovation at Maximus, a government administration company.
If you’re looking for an experienced bright IT manager, we think he’d be a great addition to your team. Welcome to the podcast, Albert. Uy,
[00:05:12] Albert Uy: Uh, hi Curtis. Good morning. Thank you for the introduction and I’m glad to be here.
[00:05:17] W. Curtis Preston: uh, have you recently completed any major? Do it yourself projects
[00:05:23] Albert Uy: Yes, actually, like you’re saying, I’m doing the Christmas lights and, and kind of upgraded my laundry, so did some cabinets and sink. So I’m a, I’m a diy like you, so d i
[00:05:34] W. Curtis Preston: right. All right. Fellow, fellow DIY person
respect. Um, sometimes, you know, you bite off a little more than you can chew. Uh, you, you know, you’re halfway through the project and you’re like, why? You know, why didn’t I just call the guy?
[00:05:48] Prasanna Malaiyandi: The solar project, Curtis
[00:05:51] W. Curtis Preston: Yeah. We won’t talk about the solar project. The solar project was, yeah.
Speaking of taking six months. Um, yeah, but it, but so ha Is it the, the, the laundry thing? Is it finished?
[00:06:03] Albert Uy: Yeah, it’s finished. Um, do you wanna hear an interesting thing I wanna share? So, you know, I, you know, I got a point. Install the cabinet and install the sink with the plumbing, and I’m like, okay, I need it. I need to get it certified, so I need a certified plumber. Right? So they say they usually charge like $90 plan trip and stuff, but when they heard, uh, I just moved into a new house, the price went from $90 to like five, $600.
So I know when they hear like, oh, you just moved into a new house. And so I went and to uh, forget it, I can do it. I went and just did it and I’m, I’m pretty sure I, I would, yeah, I, I did a great job. So my wife said, Hey, thumbs up.
[00:06:42] W. Curtis Preston: why they would
[00:06:43] Albert Uy: Yeah. Yeah. When they hear a new house, the price just went up the roof. I don’t know why.
From 90, 90 to a hundred to like four, 500 dollars. Just, just to just install the plumbing, the, you know, the, the, the, yeah.
So
[00:06:57] W. Curtis Preston: Wow.
[00:06:58] Albert Uy: crazy.
[00:06:59] W. Curtis Preston: Yeah, it is crazy. Uh, yeah, I recently redid some, you know, some stuff in the kitchen and, and I, and I, I did call the guy cuz plumbing is one of those things where it, it’s not, Technically very hard, per se, at least much of it. Some of it is actually really hard, uh, but it’s, but it, but it’s, it’s one of those things where it’s not, it’s like, it’s not technically difficult, but you’re on your back and you’re in these tight, tight, tight spaces and all that stuff.
So it’s definitely in that category where I’m like, you know what? And, and also the one thing about plumbing is if you get it wrong, you might not know right away. Right. It might be just a slow leak and then you find it out way later and it costs you way more money. Um, we won’t talk about the bathtub incident of, uh, 2000
[00:07:51] Prasanna Malaiyandi: Well, you know another thing that reminds me though, I know you’re talking about plumbing, but I feel the same way about cars and brakes. Right? It’s like a brake job. It’s like get someone to do that. And I know, Curtis, you’re laughing because I know you do your own breaks, but
[00:08:06] W. Curtis Preston: Yeah, I, that was, that was the way I felt for a long time until I, until I found out it, that, that,
that they actually weren’t, that. Yeah, they’re not, and, and, yeah. Yeah. Uh, and generally with breaks, they either work or they don’t, like, it’s not like you, you know what I mean? Anyway. Yeah. But I, but I, I, I respect the, the not doing it as well.
[00:08:27] Albert Uy: And uh, and I was gonna share the trick is you have to put the right washer and clamps. And like you said, I went and put a pan after my work is done. Uh, just to make sure there’s no leak. I left it for a month and there were no leak. So,
[00:08:39] W. Curtis Preston: Nice, nice, nice. Yeah, so, well, we’re not talking about plumbing today. We’re
[00:08:45] Prasanna Malaiyandi: Are you sure we could
[00:08:46] W. Curtis Preston: Yeah, we could, we could do a just a DIY podcast where I talk about what I’ve done and you talk about what you’ve seen on YouTube,
[00:08:54] Prasanna Malaiyandi: Yep, exactly. Yep.
[00:08:57] W. Curtis Preston: Um, and, um, maybe, maybe Albert and Albert and I can, can start a d i y podcast.
Um,
[00:09:05] Prasanna Malaiyandi: I was recently watching a YouTube video of a guy trying to clean up a 30 by 60 saltwater pool,
[00:09:13] Albert Uy: Wow.
[00:09:13] W. Curtis Preston: clean it out. What do you
[00:09:15] Prasanna Malaiyandi: sorry, clean it up because it was full of algae and everything else. He had just moved into a house and he had no pool experience, and he gave up and he just hired a pool service. They wanted five grand to bring it back to life, and then 1200 a month.
[00:09:28] W. Curtis Preston: See, this is why I do stuff myself. I had the same problem, five grand.
[00:09:33] Prasanna Malaiyandi: Yep. Well this is a 30 by 60, Curtis.
[00:09:36] W. Curtis Preston: Oh, but still, I mean, 30 by 60 is big,
[00:09:40] Albert Uy: Yeah, it is
[00:09:40] Prasanna Malaiyandi: Yeah.
[00:09:41] W. Curtis Preston: okay. One, one more. So, one more. So years ago when my, I have a pool, and by the way, you don’t need a pool. You need a friend with a pool. Um, that’s a life lesson to anyone. Please take that lesson.
You don’t need a boat. You don’t need a pool. You need a. with, with both of those, or, or one friend with one each. Uh, my pool, my pump went down and I didn’t have the money to replace it right away, and by the time I replaced it, I had a swamp in the, you know, I had a, a, it was disgusting and I actually, if you can believe this, and a bunch of stuff had been dumped in the pool like, like leaves and twigs and all that kind of stuff, I actually put on my scuba gear. and went in my pool and went down and cleared out all the debris in the bottom of my pool. And by the way, I was completely blind cuz it was just green, right? Just a hundred percent green. And I’m down there with a, with a bag and I’m just, just crawling around in the bottom. I had myself super weighted so that I was just laying on the bottom and I’m pulling out the, the twigs and the rocks and the, you know, And, and because I knew that like any one of those could, could mess up the, the, the pump, right?
The Well, no, the, the, you know, the,
[00:10:59] Prasanna Malaiyandi: Oh, the Impalas?
[00:11:00] W. Curtis Preston: crawley thing, you know? Yeah. So, so I, I, I, so the, and then, and then I just dumped it. I just dumped. The pool, right? Um, and instead of trying to spend hundreds and hundreds and hundreds of dollars of chemicals trying to bring what it was back to life, uh, but that was an expensive, just the process of dumping the pool and refilling the pool, uh, was an incredibly expensive process because they charge you, you know, uh, what it is, is it, it impacts both your water bill and your sewer.
um, because they’re like, oh, they’re, you know, cuz they, they look at how much it impacts your sewer bill for like the next quarter because they see how much water
you put out. Yeah. And so, uh, so the secret is to fill it with your neighbor’s hose,
[00:11:53] Albert Uy: Good tip.
[00:11:54] Prasanna Malaiyandi: Oh, Curtis.
[00:11:55] W. Curtis Preston: i y tip, fill your pool with your neighbor’s hose. Um,
[00:12:00] Prasanna Malaiyandi: Hmm.
[00:12:01] W. Curtis Preston: I kid. I kid. So, so, Hey Albert. Uh, you, you’ve been in it for a while, for a minute. As the kids, say,
[00:12:11] Prasanna Malaiyandi: Hmm.
[00:12:11] W. Curtis Preston: um, and, you know, and, and I’m sure you’ve been in and around backup systems quite a bit. From your perspective, how has backup gotten.
Better and worse, like over, you know, you’ve seen, uh, if you’ve been in it 20, you’ve been through the tape world, you’ve been through the conversion to disc world. You’ve been into the, you know, into the, um, obviously into the cloud world. Uh, what, what’s that? What’s that been like right.
[00:12:42] Albert Uy: I would say people are lucky to be, uh, managing backup today compared to 20 plus years ago because like you said, when I first started, there were only like one cartridge or there’s no robotic, and, and the tape is, can only do like megabyte versus gigabyte and all that. And then, and then you know, you have to start with on-premise and then now you got cloud and all you need to do is good internet connection.
So, you know, it’s almost like night and day comparison.
[00:13:14] W. Curtis Preston: Right. So talk to me about, um, so first, you know, we took, you know, you, you were in that tape world and I, and I grew up in that tape world, right? My, my. The backup tape that I first, well, technically the first tape I did a backup to was actually a nine track tape, which I don’t, I don’t remember what the capacity of that, but it wasn’t much because, um, because I remember that the, the density of that tape was 120 bits per inch.
It was a single. It was a single, um, recording stream. Right. Um, and then I, I have, I’d have to look up, I, yeah. Anyway, so that was my first, and then I remember working with, you know, eight track tapes. I’m sorry. Um, eight millimeter tapes, four millimeter tapes, uh, then, um, dlt, then L T. , right. Um, grew through all of those.
Um, d do you remember, were things ever good with that world Because I remember like, for a while they were bad because the drives themselves were really unreliable. Didn’t the drives? My memory is that the, the drives got more reliable, but then they got too fast. There. Was this, the shoe shining problem of the tape drive being too fast for the job?
Uh, how, how about.
[00:14:35] Albert Uy: Right. Yeah. Um, well, I was gonna say, well, the key thing is when you’re doing backup and disaster recovery is r p o and r t o. RPO stands for recovery point objective, and RTO stands for recovery time objective. So, like you said, um, more, uh, two decades ago because of the speed wise, um, there’s three types of, uh, backing up full backup differential and incremental.
Um, because of the slow speed you have to do of a lot of incremental, right? However, people don’t realize when you restore, restore, that means you have to restore a lot of incrementals. Right. So as the tape device goes fast and fast, you can go less incremental and do more differential. And today with the speed, you can also do full backup.
So what I’m trying to say, we use it when you restore, you always want to start restoring, using full backup and then whatever differential based on r p on R T O based on, you know, restore your differential. And if not, you’re incremental. Uh, if I make sense.
[00:15:38] W. Curtis Preston: Yeah. You do. Yeah. Prasanna, you, you remember? We just, what? It was
[00:15:42] Prasanna Malaiyandi: I think it was like two episodes ago. Yeah.
[00:15:44] W. Curtis Preston: We just did an episode on backup levels and, uh, I prefer the term cumulative incremental
[00:15:50] Albert Uy: Yep, yep, yep.
[00:15:51] W. Curtis Preston: different companies
use the term differential to mean
different things. Um, but yeah, that was a, that was a huge thing back then was the, the, how often we’re gonna do a full, how often we’re gonna do a differential or cumulative, incremental.
And then obviously we did incrementals every day if we weren’t doing one of those other
[00:16:10] Prasanna Malaiyandi: Wasn’t there a Hanoi Tower?
[00:16:12] W. Curtis Preston: Yeah, there was, the Tower of Hanoi thing was about different levels of backup real. That that was a huge thing. Compare that to what we do now, which I, I would say when you look at, um, and by the way, I, I, I’ll throw out our usual disclaimer Prasanna, and I work for different companies.
I work for Druva, and, uh, Prasanna works for Zoom and uh, this is not a podcast of either company. Uh, and the opinions that you hear are ours and sometimes they’re all mine.
[00:16:44] Prasanna Malaiyandi: They’re all Curtis’s.
[00:16:45] W. Curtis Preston: I don’t give Prasanna a word in edge, a word to speak. Um, and, um, uh, and by the way, be sure to subscribe to this podcast so that you don’t miss anything.
And also, Yeah, just click right. Just click subscribe right now so you don’t forget. And be sure to rate us, just, uh, scroll down to the rating thing. Give us, you know, five stars and, uh, give us a comment. We love, we love comments. Always helpful. Always. It helps. It helps other people find our podcast, the more comments that we have.
And, um, But, uh, and, and also, uh, you know, in full disclosure, uh, Albert is a former Druva customer. Uh, he’s not, uh, you know, he’s not employed right now, so he is not a current Druva customer, but, uh, he was a former Druva customer. So we’ll be talking about Druva a little bit in this episode. A little bit more than normal, I think.
But, um, so what I was about to say was, You know, with a product like Druva, and we’re not, we’re not the only ones that do it, but with a product like Druva, you do a, what I call a forever incremental. You don’t do repeated fulls, but. , but the backups are stored in such a way that each backup behaves as a full, from a recovery perspective.
[00:18:00] Prasanna Malaiyandi: I know we always talk about sort of the incrementals, but something that I always like to bring up is when you talk applications is after the incrementals, then you gotta worry about all your archived redo logs and whatever the nomenclature, the application uses, right?
Which just adds more and more to the time it takes to recover.
[00:18:20] Albert Uy: And, and that’s a very good point. So I said the first requirement you knew to know is the R P R T O. The second I’m a big, um, follower methodology of people, first process, and then technology. So just to share with you, I also used to work at GameStop, which is also a Druva customer.
Um, I was able to reduce the 72 hours of Dr. Uh, r t o to 24 hours. and when I mean people process technology, um, because, uh, they were like siloed the service team by themselves and database. So when we were doing disaster recovery, um, the server guys have to restore the file first and then database comes in and then use their database back up.
To restore the database. So, you know, there’s two different methods of, you know, um, restore, right? So yeah, that’s why people, you know, get together with the people first and then create the process to make the people work together and then apply the technology.
[00:19:19] Prasanna Malaiyandi: Yeah, sometimes you people tend to forget about those first two parts, right? They’re like, oh, here’s an awesome technology. It’ll solve my problems. But then they apply it, and then it doesn’t work the way they think, because the people in the process part, they haven’t worked through. And it’s like, oh, the technology’s bad.
But it’s like, no, the technology isn’t bad. It’s just you only did one third of the problem. You have to also worry about the people in the process.
[00:19:42] Albert Uy: And, and another thing I wanna share is when I was at Maximus, um, because of the people, there were only two tier support. People requesting the restore and then the backup team restoring. But the thing is, you know, it queues up because the backup team are the only restoring. So I created a third tier, like a middle tier.
Um, we gave, um, the database team power, user rights, server team power, user rights, site administrator, power, user rights. So instead of going to a backup team, the site support team can, can restore site files, database can restore database files, and server team can restore server team server files, and then application can restore application files.
By doing that, the sla, um, we were able to meet SLA because it’s not entirely, uh, dependent on the backup team.
[00:20:29] Prasanna Malaiyandi: it’s almost a little bit like self-service.
[00:20:31] Albert Uy: Yep, exactly.
[00:20:32] W. Curtis Preston: Yeah. You know, you, you bring up a really, I, I think a really good, another part of technology that. That has happened in the last, I’m gonna say 10 years, and that is the idea of role-based administration, right? It is probably a little more than 10 years, but, um, what I remember again, back, back in the day, uh, if you were gonna administer a backup system, you were root.
Right. You, you had root, you logged into the system as root. You pulled up the, the, the ui, uh, we called it a gooey back in the day, right? You pulled up the gooey and you ran that gooey as root. And if you didn’t root, because root was the, was the, the user ID that had the power to get all the files. That’s why you had to run it as route.
Right. Um, and that, but that was a horrible thing because I remember back in the day, Saying, you know, I, I, I would joke about it, but I would say, you know, be friendly to the backup person because with, with, in order to do backups, not only did I have root on the backup adminis, the backup system, I had root on every system because I needed to.
So I had the ability to log into every. Destroy your world, right? And then go to the backup system, or maybe before that, go to the backup system and destroy all the backups, right? So be really kind to the backup person because they have the ability to destroy your entire world and leave no evidence, right?
Um, that’s, that’s what we did away. With this idea of role-based administration, this idea that you gave, because back then, if you wanted, you know, the Oracle DBAs to, to, to do their own thing, you had to give them root, right? Which I think we could all agree. They don’t need root, we don’t need Oracle, uh, permission.
Right. Um, and of course I had Oracle permission back then too. Um, so this idea that you could give certain groups of people the ability to do things, The way you described it was it was a way to make things better for them, but it’s also a way to make things better for you in that you gave them just, you know, we, we have this concept of least privilege that we talk a lot about, right?
You gave them just enough power to do the job they needed to do, but without giving them the ability to mess up the rest of the world. Does that sound about.
[00:23:09] Albert Uy: Exactly role based
[00:23:11] Prasanna Malaiyandi: don’t give him the keys to the Kingdom
[00:23:14] W. Curtis Preston: Yeah. Yeah. So, so let’s talk about that a little bit. So, I know Druva has role-based administration. What other things did you do with that? Like how, how did you divvy up responsibilities?
[00:23:26] Albert Uy: yeah, so role based, so we’d just like to say database team can only restore database application, application server server, uh, site administrator site. . And the reason is, um, you know, we have data centers, but we also have remote offices. Uh, because of, um, government services. What we have is what you call a tl, uh, authorized to operate.
So some of the files has to be in the remote office before we can put it in the cloud or data center. So that’s when the site administrator come in and they restore whatever locally to their
[00:23:56] W. Curtis Preston: Also when when you say site administrator, you mean administrator of that particular
[00:24:01] Albert Uy: office. Yes.
[00:24:03] W. Curtis Preston: And what about, um, I know, I don’t know if you’ve, if you’ve done this, but what about separating. Other things within the backup system. Like some, I’ve seen some people where they’re like, we, we give this person the ability just to run the backups, but then they don’t have any ability to say delete backups.
And then, you know, we have another person who can configure backups, but they can’t restore backups. I’ve seen some very interesting stuff. Uh, can you think of anything else from the people that run the backup system? Anything that you did?
[00:24:32] Albert Uy: So, so the backup team, we used to call them, well, the team was called e edp, enterprise Data Protection Team. So they schedule a backup, so they do all the backup based on the project requirement.
[00:24:44] W. Curtis Preston: Mm-hmm.
[00:24:45] Albert Uy: So in the, on the roll base, um, the roll base are able to restore, but usually it’s the EDP team that schedules the backup.
[00:24:53] W. Curtis Preston: Mm-hmm.
[00:24:54] Albert Uy: So on the database team, um, because of the project requirement based on the r p rtl, you know how many times you do full versus incremental. So they schedule it. And then one thing we made it easy is because we went to the cloud, we also, um, uh, created a tagging. Um, you know, you can create a tag on your cloud instances.
So we have a, um, a tag called backup, and then they would put a value, uh, which is a string, and then they use regex to go to the string to say, Hey, how often do we want full? Uh, you know, is this a database? Is this a file? Is this so.
[00:25:32] W. Curtis Preston: Mm-hmm. . Mm-hmm. . Mm-hmm.
[00:25:33] Albert Uy: all the,
[00:25:35] Prasanna Malaiyandi: without having to require some two people talking together.
[00:25:38] Albert Uy: that’s correct. That’s correct.
[00:25:40] W. Curtis Preston: Did you have anything? What? What would you do if you had a new resource? Cuz I’m assuming you used like auto discovery and things like that. If you had a new resource that needed to be backed up but someone forgot to put a tag, did you, did you have anything for that?
[00:25:57] Albert Uy: Yes. So, um, we have a delivery team. Um, so we automated everything with the help of Druva. We created Lambdas on aws, so for, um, for operational readiness. So when a project goes live in two weeks, we already checked two weeks prior to going live. Those tags have values, so everything’s shared. So, did I answer question Curtis?
[00:26:20] W. Curtis Preston: that warms my heart. Uh, Albert the idea that someone would wait to go live on a project, they would wait to make sure backups work before that project goes live. That, that warms
my heart.
[00:26:33] Albert Uy: And, and, and Curtis. With GRS Health, we create what you call backup of service. So every, every instance cloud instance being created has the site value of the backup. We already know, uh, if that instance will be backed up, if that instance will be patched, and if that instance will be monitored.
So those three functions are in the tags, and we have a operational readiness team that goes in and makes sure those tags are values. And then we also have, uh, uh, account owner, account owners. We have technical point of contact, uh, for those project and for those instances. So everything’s labeled, everything has to be labeled.
So if not, uh, um, which we partner with the security team, we have what you call compliance. So before they go live, we make sure the compliance is a hundred percent before they go live. If not, uh, if not, we create, um, incident priorities. Um, we create, um, a day, um, well, two weeks before it goes live, we create a P two, and then one day before it goes live, it becomes a P one.
In other words, you need to fix it before it goes live.
[00:27:37] Prasanna Malaiyandi: now the two weeks before. Are when someone tags it. Are you actually doing backups at that point to make sure that the entire process and everything works, that things can be restored because it hasn’t quite been pushed to production yet, correct?
[00:27:52] Albert Uy: Right, right, right. So, so we follow, um, two SLAs, a corporate SLA and a project sla. So before you, you know, when we deliver everything, follow the corporate sla, and then if the project SLA is stricter than the corporate O overrides the corporate sla.
[00:28:10] Prasanna Malaiyandi: Hmm.
[00:28:12] Albert Uy: So, so when we deliver, we based on those sla, so those values are filled out based on those SLAs. Make
[00:28:20] W. Curtis Preston: that. I love, yeah. I, you know, when I started again, a hundred years ago, when I started backup sla sch, I mean, we didn’t have anything like that. I
mean, at, I mean, at best we did have, we had, we, we didn’t, I, I think I was in the backup business. Probably five years or so before I heard the terms RTO and RPO
and by the way, I agree with you, that’s where it starts, right back then again, you’ll remember this back then. We spent so much of our time talking about like backup window and um, you know, we didn’t, we hardly talked about restore window. We talked about backup window because that was the thing we had to do every night, whether or not we could fit our backups within the backup window.
So this idea that you basically, you agree on the SLA first. and then you design backups and other things, uh, to meet those SLAs and that all of this is done as a cooperation between the business and the technology folks before the server goes into, or the application goes into production again, that this is what Prasanna isn’t just what we talk about all the
time, right?
That. That con that, that, that cooperation between the business and the Yeah. The stakeholders and the, and the people. Uh, or the, or the technology folks. There, there’s another thing by the, there was another group that you talked about, and again, this warms my heart as well. You talked about partnering with the security folks, um, you know, in this world of ransomware.
[00:30:03] Prasanna Malaiyandi: Hmm.
[00:30:05] W. Curtis Preston: I think that partnership is more important than ever before, um, that folks on the security side need to have some basic understanding of what, what the way backups work. And folks on the backup side need to have a basic understanding of, you know, security. Well, thi this concept that you talked about, you know, role-based administration, that’s a security concept, right?
Least, uh, least privilege is a security concept. . And, um, so, so let me ask you this. How have you, over, over the years, how have you dealt with one of the problems that, that I see in it, which is the, the, the idea that nobody wants to do the backups , so nobody wants to be in charge of the backup system. Uh, how, how did you deal with that before and how was that any different, uh, when, you know, when you, uh, went with Druva.
[00:30:59] Albert Uy: Right. Well, going to the cloud is very helpful and since Druva is also cloud native, it’s very helpful and one advantage of working in government services were highly regulated, right? So, so we, you know, because we’re highly reg regulated, security kind of dictates what we need. So actually our operational readiness is monitored by, How about compliance?
We monitor all the, all the security compliance and operational compliance. So before going live, so let’s say if your backup choice has, you need backup agents, make sure backup agents, you know are installed, we check on that and then make sure the tag values for the backup agent is full, you know, field.
And then same with patching, and same all that. So compliance is almost like our key, like you said, security, uh, and it covers both security and operational.
[00:31:52] Prasanna Malaiyandi: and
[00:31:53] W. Curtis Preston: what about get, getting the person to actually getting somebody to actually take the responsibility for the backup process? Is that, is that less of a problem for you in that, in that,
world?
[00:32:04] Albert Uy: Um, yeah, because, uh, the delivery team, so whenever any instance, um, gets, um, created, it follows the corporate sla, everything. So all the engines install, everything’s automated. Uh, the, the security agents installed, the monitoring agents installed, the backup agents installed. So everything’s automated.
And then what we do is we use the compliance report to go in and check, make sure those, um, those agents have, he. and make sure those agents also have values on their tagging. So it’s all automated and we
[00:32:36] W. Curtis Preston: like.
[00:32:37] Albert Uy: actually Yep.
[00:32:37] W. Curtis Preston: It sounds like it’s part of a team rather than one person that’s just responsible for backups. It sounds like the team is responsible for backups. Is that okay? That, I think that’s perhaps one way to solve
[00:32:52] Prasanna Malaiyandi: Yeah, and I’m guessing that having this automated process also helps you avoid sort of like the shadow IT issue that we heard so much about like five, seven years ago, right? Where a department’s like, Hey, I need to spin up something, but I don’t want to go through normal it, so I’m just gonna go swipe my credit card, get an AWS account, and start running right.
[00:33:13] Albert Uy: Right, right.
[00:33:15] W. Curtis Preston: Yeah. And, and, and it sounds like I, I think that automated process was also enabled by the way Druva works, you know, again, before and after. W you know, you could automate to a certain degree, pick your favorite backup product. You can automate that to a certain degree with one major caveat, and that is capacity, right?
Um, a, a backup server or, or, you know, cluster of backup servers can only back up so much. It can only handle it from a, from a comp, uh, compute perspective, from a throughput perspective. Most importantly, from a capacity perspective, when the bits are full, the bits are full. And you can’t just, you can automate all you want.
Your backup system’s just gonna be like, uh, I don’t know where you expect me to put this stuff. Uh, you know, but with, but with Druva, you can a hundred percent automate it because the compute and additional capacity that you need is just automatically added to the system. Does that, does that seem about.
[00:34:23] Albert Uy: Yep, that’s right.
[00:34:25] Prasanna Malaiyandi: Just following up on that sort of the automation side, I think one thing I’m curious about Albert is. Enabling of this automation, the APIs, that a lot of, because previously, like Curtis was talking about, right? You went into a gooey or a UI and you manually clicked all these buttons and now you have sort of this automation side of things.
How important are APIs as you start to look at this new world, right? And the enabling of integration of backups into these process.
[00:34:54] Albert Uy: Um, APIs are very important and don’t forget, there’s different flavors. APIs, you have public APIs and private APIs, so most time you get public APIs, but you have to see if the public APIs support role-based, blah, blah, blah, everything, right? So APIs important. One thing though I wanna add is, uh, what’s more important than API is culture. Like automation. So when we start implementing automation, everybody’s like, Ooh, if I automated, I’m gonna lose my job. Right. So, so, so the one thing I did was promote the culture first before, you know, you have api, but people don’t wanna use it. It’s useless. Right.
[00:35:30] Prasanna Malaiyandi: Yeah.
[00:35:31] Albert Uy: So, and how I promoted Yeah. And how I promoted it.
I, um, I create, you know, sometimes it’s nice to mix fun with work. So I started with automation.
[00:35:42] Prasanna Malaiyandi: Hmm.
[00:35:43] Albert Uy: So, and I create a process in automation games. Hey, automation games is not just for developers. So I kind of come up with a, uh, idea board. So if, if, if a project manager have an idea, uh, and then other people will see the idea and say, Hey, I wanna join you.
So the developer will have a developer joining him, have a QA joining him, or. and then together as a team, they create this automation. They, and then they submit it on the automation games and then, you know, and all that. So, so kind of created automation games to promote the culture, and then people get the point, Hey, this is actually helping me.
Uh, I can do, I, I can spend my time doing more important stuff than just doing backup. Right? So,
[00:36:29] W. Curtis Preston: Right. Um, yeah, I like that a lot. Be because, you know, you, you really, um, I flash back to a. I remember, um, you know, a hundred years ago, uh, when I worked, and here comes the name again, when I worked with Stewart, um, . It’s a name that comes up a lot on our podcast. Uh, there was a, there was a guy at, at that company.
That his thing was, he was the NetWare administrator, remember NetWare.
[00:36:59] Albert Uy: Yes, no.
[00:37:00] W. Curtis Preston: And, and he had zero interest in modernizing the infrastructure because it meant that he wasn’t gonna have anything else to do. , right? Uh, that, that problem is. I think that, I think you, you really hit the nail on the head that this, this problem with au the, the problem with automation is people think, well, if I fully automate the world, then I’m going to be out of a job.
And, and I would say that what you do is you automate the stuff that you, that’s just the mundane, the stuff that has to happen every day. One perfect example is backup. So that you can then do the interesting stuff, which is like, we can talk about hunting for the bad guys, right? We can talk about watching the environment.
Something that that can be assisted by technology. Right? And I know, I know Druva has that as well, where it has the posture and observability features and it can. Look for bad things going on. But I think that that’s also a very human thing to do is to, number one, watch for the, for the bad actors. And number two, um, continually, you know, you talked about that idea board continually figuring out how we can make our security better, right?
Uh, because in in this world, there is, there, there has been no time where I think cybersecurity is more important.
[00:38:26] Albert Uy: Yes. Uh, actually I’m glad, uh, Curtis, you touch, uh, what people don’t understand is there’s um, um, there’s three data situation. You call it data at res. Data in motion and data in use, right? So like you said, mostly when you’re doing data backup, you wanted data at rest. So one thing we’re looking at is actually we’re looking at Druva is data resiliency, right?
So that’s when security and cybersecurity comes in. So you don’t just bank it out when it’s a data arrest. You also have to protect it during InMotion and uh, at use, right? So,
[00:38:56] W. Curtis Preston: Right.
[00:38:58] Albert Uy: so that’s in terms of cyber. Uh, one thing I wanna mention too, on automation, um, One trick, um, because my pre my predecessors tried automation game it, you know, coupled them and it failed.
Uh, one thing I also learned is you need to make it open. You know, the guy, the person before me said, Hey, okay, automation game, everybody has to use Ansible. But the thing is not everybody knows Ansible. So when I create automation, I open it. You can use dos batch file, you can use Lambda, you can use, uh, batch, uh, you know, batch script, and. and that’s how it’s successful. So, and, and then what, what we needed to do is find a tool that will support all open scripting or, or programming. So,
[00:39:39] W. Curtis Preston: Nice. Uh, yeah, I like that. Yeah, because not, not everybody knows. I mean, I remember back um, everybody was jumping on Pearl. and all I knew was, all I knew was Bhe. Right. Um, I was an old Bhe guy and I, I learned Pearl and I, I wrote, I remember writing, uh, just to learn Pearl. I remember writing a monitoring program that, that created a webpage.
of how net backup was running at the time. Right. Um, this was, uh, I actually remember the first time that, that, that, that, uh, system, I dunno what to call it, program, uh, the first time I went live was actually at Amazon back in, back in 1998. Uh, it was a long time ago. Um, I, you, you mentioned earlier about how you made Dr.
Faster. Um, when you talk about that 24 Dr. Pro, 24 hour DR process, uh, talk to me about how, how does that 24 hours break down?
[00:40:52] Albert Uy: Um,
[00:40:52] W. Curtis Preston: what, what does it consist of?
[00:40:54] Albert Uy: yeah, because, uh, everybody was operating in, uh, silo. So the, the, the server team, uh, used a product called, um, at that time was using a product called CommVault, and then the database team was using SQL backup, so the old database between SQL backup and then the server team will use Combo to back up the sql. So when you do diver dr, you do reverse, right? So the server team comes in and restore the combo file, and then after that, the database team has to use that file and, and, and, and restore it into a SQL database, right? So that’s like a, just the process alone is 72 hours. But since they, we created the process.
So instead of using SQL backup, it’s just one file. So when the server team restore it, it’s already the. You don’t need the database team to come and restore this file into a SQL database, if that makes.
[00:41:46] W. Curtis Preston: you, you made the actual restore process itself. You, you took out steps in the restore process itself.
[00:41:53] Albert Uy: Right, right. And then it cut costs to this. And I was talking about at that time there were no cloud yet. So when we are doing VR exercise, um, we had to fly in everybody. The file backup team, the database backup team, you’re talking about a good 15 people. So, uh, 15 people, um, not doing their day-to-day job, just doing the DR for three days, so,
[00:42:17] W. Curtis Preston: Right, right. Yeah. We, we did that back in the day. We would, we didn’t have to fly anybody in cuz we were all in the same place. But, but we did it over the weekend cuz we were a bank. Um, so it’s not like we’re gonna take the bank down for a few days. We did it over the weekend. . Um, and, um, we would, uh, one of the, one of the rules that we had back then was that the person responsible for the backups, which was me, was not to run the Dr. right. Um, because the idea is, it’s supposed to be properly documented enough that the, the other person, you know, that another person that’s capable should be able to follow the documentation and do it. I will tell you, in the three years that I was there, not once did we successfully do the , the restore from beginning to end without having to come in and, and, and help.
I mean, it did work. It’s just. You know, the, the process was so complicated and so convoluted that, um, you can only document it to a certain point and then, right. And, and so I, I, I do think that’s, you know, you talked about making it so much simpler with a, with a single step restore process. That seems like that would be the, the biggest boon to getting things restored in.
[00:43:35] Prasanna Malaiyandi: it’s interesting that sometimes people forget about that. Instead of looking at how do we document this process? It should be, how do we simplify this Right. and start thinking about that.
[00:43:47] W. Curtis Preston: Yeah. How can we make this process so simple that documentation is like, pull up the UI and log in, or, or
[00:43:56] Prasanna Malaiyandi: Well, well here, well here’s a, here’s a question for you. Right. Whenever you go buy a product today, how often or do you go and like open the instruction manual, read through the instruction manual before using it? Right, Curtis? Or you just got a Roomba, right? Did you actually read the instruction manual or were you like, oh, I’m just gonna use it?
[00:44:12] W. Curtis Preston: Well, that was the quick start card that I did need because there, there were some steps that I had to, but it was just a two-sided of a card. Right. Um, but you, you, you did make me think about, again, in this process we bought a new, um, we bought a new dishwasher. And there was this, I, I was putting in the dishwasher and I had put it in essentially without, without really, I mean, I had looked at the manual a little bit and, and, and I had it installed.
And then we wanted to, um, I wanted to level it and it was level, but I wanted to raise the level of it so that it fit better in the, in the,
in the uh, counter and. I was doing it. Like, I was like, well how hard could it be? You just turned the things right. Um, right. And my wife in the middle of this and I was getting frustrated cuz it wasn’t working the way I would expect it to work.
And my wife was like, have you looked at the manual? And I got mad. I was like, dang, I don’t need the manual to turn some screws. And it turned out I looked at the manual and I was doing it all wrong because they had changed. They had changed, uh, Yeah, so the manuals can be helpful, but yeah, the process should be simple enough that you can do that.
Uh, so you know, Albert, I wanna thank you a lot for coming on. Is there anything that we, that maybe you wanted to talk about that we, that we haven’t covered yet?
[00:45:34] Albert Uy: If I could go back to what you said about how backup has progress. So if you remember 20 years ago, there’s no cloud. So, and then most of the time you change your DR provider every two years. Right? You know, popular ones are like Sunguard and ibm. 20 plus years ago, if you go with a provider, uh, you say, how many servers do you need?
So 20 years ago you send this team going in and then they installed the os, they installed the app. and then 10 years later, uh, I don’t know if you heard, they’re searching as bare metal restoration. So you, so I did that too. So what you do is you put a bare metal orchestrator on that location. Uh, you, you, you, you say how many servers and then, you know, based on the nick, um, then you have to spare bare metal restoration to restore it, right?
And then now today you got the cloud. So it’s gets simpler, simpler, uh, and faster.
[00:46:25] W. Curtis Preston: Yeah, because with, with the cloud doing Dr. I, I’ve always thought that Dr. Cuz you know, some people are like, oh, he says that cuz he works for Druva. I, I’ve only worked for Druva for five years, I’ve said this for many years. And that is that. The Dr. The, the DR is the killer app for the cloud. Because what you want is you want, uh, you know, a thousand servers right now.
and you don’t wanna pay for them until they arrive. Right. Until you need them. And, but with, with a, with a product like Druva, you can do the restore part in advance, right? And then it just automatically brings the servers in when it, when it needs. And, you know, you talk about a simple DR process, it’s just literally a one button and then it automates all of that stuff.
Bringing all those servers online, all of the network settings, all of that stuff. . Um, so yeah, I, I, I think if, if, if you haven’t experienced DR in the Cloud, uh, I’d say give it a try. If you’re pr, if the product you have currently offers it, give that a shot. If it doesn’t, uh, you know, they’d be happy, happy to talk to you over at Druva, uh, to see how easy Dr can be.
Uh, and, and I’d also say cloud, it makes it so much less expensive that it can bring Dr down to. The masses, if you will, right? It used to be it was only the, the companies who had lots of money that could afford the sun guards, right? This, this is literally companies of any size should be able to automate their DR environment.
[00:48:00] Albert Uy: And, and Curtis, I would like to add has to be a hundred percent cloud native. So just to share with, that’s reason why we went with Druva. So when I was at Maximus, um, they had a project to go from on-premise to cloud. . So the backup team was reporting to the engineering team. At that time, I was managing the operations team.
They had two years to do it. after 18 months, they just did one remote location and, and then they gave me the team and say, Hey, finish it. And I got six months left from the 24 months. And you know, you’re talking about several remote sites and big data centers. So they said, here’s the solution, implement six months.
I’m like, okay, you tried it for for 18 months and you couldn’t do it and you’re giving me six months. Fortunately I was technical and I look at the solution, I’m like, this is not cloud native. This is hybrid. And that’s the reason why it’s taking forever. Because when you talk about hybrid, that means including pro premise and on premise equates to you need provisioning.
So, you know, I spent two months using their solutions that there’s no way this is gonna work. I only have four months left and I have like so many sites and because I had Druva experience in my previous job, which is with Gamestop. I went in to the executive team and said, Hey, if you want me to get this done in four months, uh, uh, we need to go with my new solution.
And they gave me a thumbs up and I was able to do it in four months. And how I did it in four months is because I know how Druva works I’ve spent the first month with, uh, what I call A P O V, I don’t call it poc, proof of value.
[00:49:33] W. Curtis Preston: Yep.
[00:49:33] Albert Uy: And I had the Druva team help me. We created playbooks. So all I needed to do is I know the solution works, I just create the.
A playbook for database, a playbook for application, a playbook for server, a playbook for sites, right? Spend a whole month creating the playbook and then project management team help PMO help me come in and spend the three months. And actually I told ’em I don’t want to get in done in three months. I wanna get it done in two months.
Because if you give them three months, they’re gonna use three months. Right?
[00:50:02] W. Curtis Preston: right,
[00:50:03] Albert Uy: So, so I only have four months left. One month for p. Three months employment, but I told him two months and we were done in two and a half months. So two weeks before the deadline, because of expensive, on-premise, uh, license renewal, um, we just, we had time to check and make sure everything works.
So we got it completed in four months. Using the solution. So just share. And I did apply the people process tools. I know the technology works. Now I have to figure out the people, the database team, the server team, the role base, like you said, in creative process, how we can make all of them work together and how everybody follow the same playbook and all sort.
So
[00:50:41] Prasanna Malaiyandi: Nice.
[00:50:42] W. Curtis Preston: I like it. I like it.
[00:50:44] Albert Uy: Right. Just to share
[00:50:45] W. Curtis Preston: Thanks so much for, thanks so, so much for coming on the podcast
[00:50:48] Albert Uy: uh, thank you for inviting me. Uh, pleasure meeting.
[00:50:52] W. Curtis Preston: and Prasanna. Thanks again and thanks for helping me celebrate my little.
[00:50:56] Prasanna Malaiyandi: Woo-hoo. Nice to meet you, Albert.
[00:51:00] Albert Uy: Likewise. Thank you. Happy holidays.
[00:51:04] W. Curtis Preston: yeah, absolutely. Happy holidays and thanks to the listeners. Remember to subscribe so that you can restore it all.
