A few weeks ago, Mr. Backup (W. Curtis Preston) said he didn’t understand why people used flash for backups. He said it was overkill. A few days later, Howard Marks of Vast (friend of the pod) took issue with that statement, and asked for the chance to defend Vast’s title, so to speak. Howard is a friend of the pod and we were happy to say yes. We also take the opportunity to get an update on Vast, and discuss their data reduction techniques in more detail. Bonus points if you get the cover art reference.
Transcript
[00:00:39] W. Curtis Preston: Hi, and welcome to Backup Central’s. Restore it all podcast. I’m your host, W. Curtis Preston, AKA Mr. Backup. And I have with me my continuing advisor on my consumer backup project Prasanna Malaiyandi, how’s it going? Prasanna.
[00:00:57] Prasanna Malaiyandi: I’m good. Curtis. You know what? We have the expert, the backup anorak Daniel Rosehill coming next week on the podcast, so
[00:01:05] W. Curtis Preston: yeah.
[00:01:06] Prasanna Malaiyandi: can definitely pick his brains.
[00:01:08] W. Curtis Preston: Yeah. You know, it’s been interesting cuz, and it was funny, it was on the podcast where I really, I really, I had a moment where I was like, I’m not really backing up I photos. Right? Like, I’m, I’m backing up, you know, I, I use iCloud, but as, as we have discussed, iCloud is not a backup.
iCloud is a sync. Right. And if something catastrophic, if you, if I ever got hacked, um, and somebody got a hold of my, my iCloud password or my iPhone and then just decided to massively delete everything, I, if I caught it soon enough, I would be okay. Cuz I do have like a deleted items thing. Right. And as we discussed, I have, well, I don’t think we’d discuss on the pod, but as part of this project I found out I have 11,000 photos in, in iCloud. So,
[00:01:59] Prasanna Malaiyandi: isn’t as much as a lot of other people, you know. I’m sure there are
[00:02:02] W. Curtis Preston: know, I am not,
[00:02:03] Prasanna Malaiyandi: you.
[00:02:04] W. Curtis Preston: yeah. As, as I, as you and I were talking earlier, I’m not, you know, on one end, I’m not Cecil b Dilla and I’m not, you know, photographing and filming everything. On the other hand, I’m not Prasanna because you use your phone camera, like you use your, uh, Tesla,
[00:02:19] Prasanna Malaiyandi: Yeah, pretty much. Yeah. Almost never
[00:02:23] W. Curtis Preston: Okay. So you’ve had your Tesla, how long now
[00:02:26] Prasanna Malaiyandi: four years,
[00:02:27] W. Curtis Preston: and how many miles do you have on it?
[00:02:30] Prasanna Malaiyandi: I think like 11,600, 700, something like that. And it works great because my maintenance cost has been zero. My electricity cost is really minimal versus gas,
[00:02:41] W. Curtis Preston: Cars are incredibly reliable when you don’t use them. Um, anyway, yeah,
[00:02:46] Prasanna Malaiyandi: powered cars, even if you don’t use them, you still gotta change the oil. You still gotta do everything else, you know? So,
[00:02:52] W. Curtis Preston: I’m not, we’re not doing a, we’re not doing an e an ecar thing. But anyway, but yeah, we’ve, we’ve been having some fun with, uh, with this project of figuring out the various ways. Right. Uh,
[00:03:05] Prasanna Malaiyandi: and, and I think, yeah. I was just gonna say, I think you should mention to the listeners what you’re current, what you are currently trying to do for backing up your iCloud photo
[00:03:14] W. Curtis Preston: my current, uh, uh, uh, I don’t know what, I don’t know what, what, what, I don’t know these different methods. My current method that I am trying is Google Photos, and it turns out Google photos, it, it’s the only one that I’ve found so far. Uh, well, the only one that I’ve. Well, there’s maybe one other, which is iDrive, but Google Photos, cuz the problem is that on an iPhone you can turn on optimized storage.
And so I have like, I don’t know, somewhere between sixty and a hundred gigabytes. We’re not quite sure of photos up in, uh, iCloud and, but I only have four and a half gigabytes on my phone because it’s, I’m using the optimized storage. But apparently Google Cloud photo, Google Photos pulls down a high res whatever high, the original version from. iCloud and then backs
[00:04:14] Prasanna Malaiyandi: that, that’s our theory. That’s our theory.
[00:04:16] W. Curtis Preston: That’s the theory. Well, it’s, I, that’s what it says in documentation. We shall see what we shall see. Um, and then we will report on the results here and then, and, and I’ll blog about it.
I’ll backup Central, uh, because IO iCloud is not a backup. The number of articles that I read that told me to use iCloud to back up my iPhone pissed me off. Right? Like, it, it was like 95% of the articles that I found on how to back up, uh, my photos basically said, OI cloud. I’m like, ah.
[00:04:49] Prasanna Malaiyandi: Because. Because for most consumers, right, they’re probably not going to do what you’re about to do, and they don’t care. And so turn on iCloud. At least you have something else other than whatever’s on your phone,
[00:05:00] W. Curtis Preston: Yeah. So we’re gonna, we’re gonna have an answer for the three people in the world, all of which are probably already on this recording, the three people in the world that actually care about having an actual backup of their, of their photos. Anyway, all right. Well, we’re gonna bring back a, a longtime friend and a returned guest to our podcast.
Uh, he is one of the few people in this industry that, um, make me feel young. Uh, we welcome. We welcome. And he’s also the, uh, the technologist, extraordinary and plenty of potentially at Vast Data. Welcome to the podcast, Howard Marks. How’s it going, Howard?
[00:05:43] Howard Marks: I’m really happy to be here cuz you know you guys went on your little podcast and you said something about using flash for backup being stupid
[00:05:55] W. Curtis Preston: Yeah. Yeah. I might have
[00:05:56] Prasanna Malaiyandi: But, but, but, but wait, I wanna clarify, Howard, that was from Curtis. I was the one who was like, yes.
[00:06:04] W. Curtis Preston: Oh, he’s
[00:06:05] Prasanna Malaiyandi: What about Howard? Oh,
[00:06:07] W. Curtis Preston: throwing me right under the
[00:06:08] Prasanna Malaiyandi: throw you under the,
[00:06:09] W. Curtis Preston: So, we’ll, we’ll, we’ll
[00:06:10] Howard Marks: you don’t have to explain that to me. I’ve known Curtis 35 years.
[00:06:15] W. Curtis Preston: that’s that. We will, we will, uh, we will, we’ll get to that topic. I will give you a chance to defend your, your, your Honor. Um, why don’t we start with an update. It’s been a while since we’ve had you on the pod. Why don’t we start with an update on, uh, how much more vast, vast data is, uh, since we had you on.
[00:06:40] Howard Marks: Well, you know, from the financial side, um, we announced at the beginning of this year that we’ve hit a hundred million a year in annual recurring revenue cuz we’ve organized ourselves as a software company even though the experience customers have is look, appliances on my data center call vast when something goes wrong.
Um, we arrange for customers to buy the hardware so that we are a software company, makes life easier for us. Um, the other big thing is that our friends at HPE
just made announcement of a product of theirs called GreenLake Files that will be powered by our software. So before today, if you wanted a scale out, expandable, low cost all flesh file and object system, we would facilitate your buying hardware from the OEMs that we deal with, and we’d sell you the software and you’d have a system that was running.
Now you can buy that from HPE as part of GreenLake, and that includes management through the GreenLake Cloud front end. So you can manage the GreenLake for files along with GreenLake for Block and the Compute and all the other servers that are part of GreenLake. So they’ve taken our software, married it to their control plane and run it on their hardware.
[00:08:10] Prasanna Malaiyandi: And is for, sorry, for those who may not be familiar with GreenLake. GreenLake is more of a, I don’t know, a managed or a hosted environment done by
[00:08:20] Howard Marks: It, it, it, it’s an as a servicey. So there, there are both consumption and CapEx models the way I understand it. But you know, you don’t log into the block array that and create a LUN. You go to the cloud website and you create a LUN and their control plane does that for you.
And so it’s got more controls and you do don’t have to keep the detail.
[00:08:53] Prasanna Malaiyandi: Yeah.
[00:08:54] W. Curtis Preston: So then you’re, you’re probably paying for what you provision then,
[00:08:58] Howard Marks: Uh, you can do that or you either way. And, you know, that’s, that’s, you know, kind of an H p E finance question.
But my understanding is they do it either way.
[00:09:08] Prasanna Malaiyandi: And I’m sure for vast data, right. That’s a huge win, being part of that offering.
[00:09:14] Howard Marks: Well, it’s, you know, first of all, it just gets hundreds and hundreds of boots on the ground out selling our software and our whole concept of you can do all flash for as cheaper, cheaper than other guys can do spinning disk. And why do you want spinning discs as opposed to flash? Other than that, they’re cheaper.
You know, I can’t think of another advantage. Um, and so we, you know, for most workloads have narrowed that down or reversed it, and so, All flash cheaper than disk. What a great idea. Um, so we’ve got, you know, a, all those HPE sales guys going out there selling it as an HPE product, you know, it’s not like Qumulo or Scality, where HPE was reselling those products to run on HPE servers.
[00:10:09] Prasanna Malaiyandi: Mm-hmm.
[00:10:10] Howard Marks: Um, and you know, who you called for support was. Well, is this a server problem or is this a software problem? It’s HPE GreenLake for files. HPE takes the support calls. It’s a full HPE product. Um, it’s our software underneath it.
[00:10:29] W. Curtis Preston: Yeah, it’s interesting. So you know, you talked about the, you, you know, you said you, you. You, you have a r r. So basically your customers are paying an annual fee to you based on the size of their storage
[00:10:47] Howard Marks: we, so, so we make, we make our money on a, what we call a Gemini subscription. That is, you know, in capacity units, we sell it at a hundred terabytes. HP can sell it in different ways, um, and it’s per year. And we guarantee that we’ll write that agreement for any piece of hardware for 10 years at the
[00:11:10] W. Curtis Preston: right, right. right. I remember
[00:11:12] Howard Marks: Because, you know, it’s not spinning disks. They don’t start failing a lot more often in year five and six. And so if you want to keep it for 10 years, keep it for 10 years. If you decide you want to replace some of your hardware in your five or six, because the new denser or faster hardware is more attractive to you, uh, but you bought seven years of support, we’ll transfer it on the, you know, terabyte per terabyte basis.
[00:11:41] W. Curtis Preston: Right. Gotcha. Um, yeah, that’s a pretty good deal for you. And by the way, I’ll, I’ll, um, I, I was gonna, uh, compare it to something else, but, but it, it made me remind me of our, uh, disclaimer Prasanna. And I work for different companies. I work for myself, he works for Zoom. And, uh, these are our opinions, not theirs.
And, uh, be sure to rate us at, uh, your favorite podcast or give us all the stars and comments. It helps other people find us. If you think we’re amazing, then maybe other people will do so as well. Uh, reach out to me, uh, @wcpreston on Twitter, or w Curtis Preston at gmail, and, you know, to be part of the conversation.
And we’ll see. Um, you know, we’ll get you on. So your arrangement with HP reminds me of our arrangement with Dell. Basically it’s the whole boots on the ground thing. Uh, you get to put your product in front of a whole, you know, giant number of other people and it’s great for you. It’s good for them. Their customers get the benefit of your, uh, technology with, with the company that they already, you know, know and
[00:12:51] Howard Marks: And you know, and, and we all know that there are loyal H P E customers who you know now it’s a lot more likely they’ll buy this product cuz it’s got that stamp of approval on it,
all of which works for us.
[00:13:05] W. Curtis Preston: Yeah. Yeah. Well, congratulations on hitting a hundred million. Um, wish you the best of luck on your way to the, you know, doubling that and triple in that. Um, last time you were on, we talked about, we, we alluded to, I think, a little bit about how you do dedupe or dedupe-like stuff that’s a little different than the rest of the world.
And, and, and that it’s better, you know, these are the, you know, the, the, you’re saying it’s better. So I, I want to give you a chance to talk about that, and then
[00:13:42] Howard Marks: we, we guarantee it’s, we guarantee it’s better because I’m a vendor and without a guarantee you shouldn’t believe anything I say.
[00:13:49] W. Curtis Preston: Okay. All right. That sounds good. So how so how, so first off, how is it
better, and then why?
[00:13:56] Howard Marks: It’s better cause it reduces data further. Um, And the why is how it works. So, you know, at the beginning it’s really pretty simple. We do variable chunk deduplication with a variation of the rock soft method.
So if there are insertions, we re re re-sync relatively quickly and the deduplication gets more effective.
[00:14:22] W. Curtis Preston: Mm-hmm.
[00:14:23] Howard Marks: Um, we do z standard compression on the data. We then throw some data specific encryption algorithms at the data for things like, oh look, it’s numeric data. Well that means it’s only gonna vary within this range.
We’ll store deltas. And so whichever of those compression methods reduces this block of data most.
[00:14:50] W. Curtis Preston: Mm-hmm.
[00:14:50] Howard Marks: we run. Um, because we’re doing so the, the data path is writes go to storage class memory.
[00:15:00] W. Curtis Preston: Mm-hmm.
[00:15:01] Howard Marks: then get act and then all of this data reduction happens as we migrate from that writebuffer to the capacity flash layer.
And since it’s after the act, as long as we’re draining that buffer fast enough, l how long in time it takes to move any piece is irrelevant. And so we have time to go, ah, let’s try five different compression algorithms. Use whichever one works best.
[00:15:30] W. Curtis Preston: Interesting.
Yeah.
[00:15:31] Prasanna Malaiyandi: do you do,
[00:15:32] W. Curtis Preston: go ahead. Go ahead.
[00:15:33] Prasanna Malaiyandi: do you? And that’s actually very interesting how you can, like you said, by storing it in memory, right. You’re not impacting client latencies at all. Right. For them it’s like, Hey, right. Went through. And then you have this time to do the parallel, uh, computation.
[00:15:48] Howard Marks: Yeah, just, just accept it. It’s storage class memory, so it’s an S s D, so it’s persistent and there’s no batteries and protection and you know, a panic when power goes
[00:15:59] Prasanna Malaiyandi: Yeah. Now, when you are running these algorithms, like I know AI and ML is all the hot topic everywhere you look these days, right? Are you guys doing anything around that in terms of trying to smartly detect which compression algorithms based on
[00:16:16] Howard Marks: We we’re, we’re not doing that in the data path right now. You know, frankly, the running the five doesn’t use that much compute that it’s worth it. Um, we’re using AI in our cloud platform, so if you have multiple clusters, there’s a cloud site you can go to and see one dashboard. Um, and we’re using it for the capacity projections.
So it’s like, oh look, here’s how much capacity you’re gonna need six months from now while you’re filling out your budget request. Let me tell you what you’re gonna, there’s AI behind that so that it smooths things like, oh look, every three months they do a cleanup. And so let me factor that the AI is good enough to factor that kind of thing
in, but not in the data path.
But let’s get back to the data path.
[00:17:05] Prasanna Malaiyandi: Yep.
[00:17:06] W. Curtis Preston: yeah, your, um, your comment when you, you know, there was a comment you were like, as long as we’re clearing the buffer quick enough. Um, and, and I would agree with you, um, you know, how, how do you ensure that that happens, I guess is, is one question.
[00:17:28] Howard Marks: Well, first of all, it becomes a parallelism issue. So we have a large number of compute nodes, all of which are draining this buffer in parallel. And so when the buffer hits a high water mark, more threads to D stage, it gets spawned and allocated across the parallel system. Now, if there’s a huge influx of writes, and you know, we’re talking. Tens of gigabytes per second for hours on the smallest system.
[00:18:06] W. Curtis Preston: Mm-hmm.
[00:18:07] Howard Marks: Um, then we’ll start introducing latency into the writes and apply back pressure.
[00:18:14] W. Curtis Preston: Okay. Okay, that makes
[00:18:16] Howard Marks: But, but you know, that’s, you know, literally, you know, it, it don’t
ha it, you know, the mechanism is there just in case,
[00:18:26] W. Curtis Preston: right.
[00:18:26] Howard Marks: happen.
[00:18:27] Prasanna Malaiyandi: And when you, and because you have more than more capacity, I guess, more throughput at the capacity level than at the storage media level, is that why you can just increase the number of parallel threads and you don’t have to worry about the backend being a bottleneck?
[00:18:46] Howard Marks: so in, in, so our, our building block, we call a D Box or a data box, and it’s got some. S scm SSDs. We started with Opta. We now mostly use K oxia FL six, and then a larger number of capacity SSDs. And they, you know, it’s whatever the cheapest we can get or the cheapest that our OEMs use is, um, the P C I E lanes.
Feeding the small number of S C m SSDs is generally the bottleneck.
[00:19:24] Prasanna Malaiyandi: Okay.
[00:19:26] Howard Marks: And so we can paralyze reading data out of s C m, the writing to cut to the capacity. We have a lot more capacity ssd, so there’s plenty of bandwidth to write
[00:19:37] Prasanna Malaiyandi: Gotcha.
[00:19:39] W. Curtis Preston: So what, why’d you stop using OC Octane?
[00:19:44] Howard Marks: Um, well first we decided just to get a second source because it’s a good idea. Um, and then I, Intel
[00:19:55] W. Curtis Preston: turned out to be a really good
[00:19:56] Howard Marks: of, then, then Intel decided to get out of the business. Um,
and, you know, we have supply agreements with Intel. They still have a warehouse full of wafers. Um, but it, you know, the, the performance advantage wasn’t worth the complexity.
So we’ve chunked on these variable sized 32 K average blocks and we de-dupe them. But in addition to running a strong hash. To validate identical, we run a series of weaker hashes against the same data blocks, and these weaker hashes are designed to generate the same hash value for inputs across a narrow range of cryptographic distance. So if two blocks have a sm, so cryptographic distance is the number of bits you have to flip to turn block A into block B. If block A is within X bits of block B, this hash will generate the same hash value
[00:21:09] W. Curtis Preston: Okay.
[00:21:11] Howard Marks: from a data reduction point of view. If two blocks generate the same hash value and are a small cryptographic distance part, they have long common strings between them. And will therefore re compress with the same compression dictionary. So the first block that generates one of these similarity hashes, we just compress and store when the second through MTH block generates the same hash. We recall the first one and we used the dictionary from the first one to compress the second one
[00:21:51] Prasanna Malaiyandi: So you get better compression
[00:21:53] Howard Marks: we can store it compressed without the overhead of storing the dictionary a second time.
[00:21:58] Prasanna Malaiyandi: yeah.
[00:21:59] W. Curtis Preston: Yeah. Yeah.
[00:22:01] Howard Marks: and it becomes essentially the difference,
[00:22:03] Prasanna Malaiyandi: Yeah. So instead of storing a bigger block, it’s like just very, very small Deltas because they are cryptographically
[00:22:13] Howard Marks: right?
[00:22:13] W. Curtis Preston: that,
that’s
[00:22:14] Howard Marks: similar. The mathematicians would say it’s a limited cryptographic distance.
[00:22:19] Prasanna Malaiyandi: That’s unique. I’ve never heard of someone doing that. Have you, Curtis?
[00:22:27] W. Curtis Preston: just this guy that we had on the podcast a little
[00:22:29] Prasanna Malaiyandi: Yeah.
[00:22:29] W. Curtis Preston: that looks a lot like Howard.
[00:22:32] Howard Marks: It, it, you know, nobody else is doing it now. Um,
[00:22:38] W. Curtis Preston: Is it, are you patenting it or,
[00:22:41] Howard Marks: there are patents around it. I don’t, I haven’t looked to see exactly
[00:22:46] W. Curtis Preston: gotcha. Gotcha.
[00:22:47] Howard Marks: to.
[00:22:48] W. Curtis Preston: Yeah. That
[00:22:48] Howard Marks: Um,
cause reading patent applications makes my brain hurt.
[00:22:53] W. Curtis Preston: I see, I thought that you would, let’s say you got two chunks, right? And you run the really weak, but much faster, I’m assuming, uh, hashing algorithm, and that you would say these two blocks definitely aren’t the same. And so let’s not do anything else other than com.
They’re not, they’re nowhere. They’re, they’re, they’re cryptographic distance. I think you said so far apart, there’s no point in running the stronger ddu, uh, thing on it. Um, that’s
where I
[00:23:23] Howard Marks: it, it turns out, it turns out even with a weak hash, the number of identical hashes that are not identical data is so small that the cost of testing is ignorable.
[00:23:39] Prasanna Malaiyandi: Yeah.
And especially if it’s in flash, like probably
[00:23:43] Howard Marks: the,
[00:23:44] Prasanna Malaiyandi: is
[00:23:45] Howard Marks: compare is so rare. It doesn’t matter that it’s expensive.
[00:23:49] Prasanna Malaiyandi: Yeah.
And the fact that you’re not doing this in line, right. So it’s all been de
Or
[00:23:56] Howard Marks: it, it’s not trench. It’s in line, but it’s not.
[00:24:03] Prasanna Malaiyandi: client.
[00:24:03] Howard Marks: act,
[00:24:04] Prasanna Malaiyandi: Yeah, yeah. Exactly.
[00:24:05] Howard Marks: it’s, it’s it’s post act, so it doesn’t have any impact on latency. But you know, the, the S CM is a one-way writebuffer. We write new data into it, it gets demoted to the capacity flash layer and there’s so much bandwidth in the capacity flash layer that reads from there actually faster than from the scm.
So there’s no reason ever to promote it back.
[00:24:32] W. Curtis Preston: Right,
[00:24:32] Howard Marks: Um, but the other thing is we keep all the metadata in that s scm. So as you expand the system, you add another enclosure that’s got more s cm and more capacity, the DUP hash table and the similarity hash tables all grow
with it. So it’s one data reduction realm regardless of how big a cluster is.
We don’t have to store that DUP table in memory.
[00:25:00] Prasanna Malaiyandi: Yep.
[00:25:01] Howard Marks: And so you know the whole, well, flash would be great for backup, except I can’t afford it as well. If you’ve got three or four conventional PBBAs,
[00:25:14] Prasanna Malaiyandi: Mm-hmm.
[00:25:15] Howard Marks: you know, first of all, the vendors of PBBAs charged you a lot for that disc storage. You know, they, that’s a high margin product.
[00:25:25] Prasanna Malaiyandi: Yeah.
[00:25:26] Howard Marks: as soon as you have two of them, you have two deduplication realms.
[00:25:31] W. Curtis Preston: Right,
[00:25:32] Howard Marks: And we might talk about data duping 10 to one. That doesn’t mean all your data dupes 10 to one
[00:25:40] W. Curtis Preston: right.
[00:25:41] Howard Marks: 50% of your data at least is unique.
[00:25:44] Prasanna Malaiyandi: Yep,
[00:25:45] Howard Marks: Some of your data ddus a hundred or a thousand to one, and most of the benefits you get is from that data that ddus a hundred or a thousand to one.
Well, when you got two boxes, it’s not a hundred or a thousand, it’s 50 to 500.
[00:25:58] Prasanna Malaiyandi: yep.
And every time you add a new box, you’re, you lose some of that benefit as well. Right. So,
[00:26:05] W. Curtis Preston: you can dup all our episodes down to like four or five comments. Right? Like back up, back up, all the things.
[00:26:12] Howard Marks: Well that
[00:26:13] W. Curtis Preston: 3, 2, 1. Just 3, 2, 1.
[00:26:16] Howard Marks: that requires the next version of, uh, AI deduplication that can take out the idle banter.
[00:26:22] W. Curtis Preston: Exactly, exactly. Our episodes will be like five minutes long.
[00:26:26] Prasanna Malaiyandi: So just to summarize or just to close on that, so we talked about the how you guys do it. So because of all these technologies that you’re leveraging or mechanisms, right, that’s how you’re able to offer that guarantee, right? That’s better than anyone else.
[00:26:43] Howard Marks: Yeah, we, you know, we use Z Standard. It’s a slightly newer compression algorithm than anybody else does, cuz we started a little bit later than everybody else. So we got to pick the latest one. Um, and then we have those, you know, the additional, well, oh, they’re numbers, let’s just store the differences, tricks.
And then we do deduplication on variable block, which is as well as anybody does it. And then we throw in similarity as, oh, here’s another unique trick nobody else does. And so the combination is, we are confident that as long as you’re send, you know, we guarantee as long as you’re sending us unencrypted data, that will reduce it better than the other guy, whoever the other guy is.
And if we don’t, we’ll provide the capacity so that you didn’t pay any more money. Cuz
[00:27:33] Prasanna Malaiyandi: Yeah. Yeah. No, that’s a great guarantee for end users and customers, especially with budgets these days, right? It’s like, Hey, I bought this system. It doesn’t quite meet my expectations. I can’t go back to my boss and ask for more money. So
[00:27:47] Howard Marks: Well, you know, the other side of that is, you know, just, it’s really a very simple scale out architecture. So you don’t buy today what you think you’re gonna need in three years. You buy today what you think you’re gonna need in a year, and then you can buy more when you need it later. Or if, as some of our customers have found out much to therin of our sales guys, their data reduces better than they expected and they don’t need anymore in the next year.
Well then you’re just ahead of the game.
[00:28:19] Prasanna Malaiyandi: So, and I know maybe you could talk in gener generalities, but sort of like if I was a customer who had one of the competition, PBBAs, right. And I now use Vast, right. I buy a vast system, sort of like, is there, like what is the savings that I normally see in terms of storage? Like if I had like a hundred terabyte P B B A, actual
[00:28:42] Howard Marks: if, if you, you know, a hundred terabytes is small for us.
[00:28:45] Prasanna Malaiyandi: okay.
Or say
a
[00:28:46] Howard Marks: So if you had, if you had a petabyte P B B A, um, then you know, you’re probably storing four or five petabytes of logical data on
it. Um, and you bought a, you know, a petabyte of usable from us and you’d probably store 25 or 30% more on it.
[00:29:08] Prasanna Malaiyandi: Okay.
[00:29:09] Howard Marks: But that petabyte, P B B A is as big as you can buy that P B B A, there isn’t a two petabyte P B B A.
[00:29:17] Prasanna Malaiyandi: Yep.
[00:29:18] Howard Marks: And the real difference is at restore time
[00:29:24] Prasanna Malaiyandi: Hmm.
[00:29:26] Howard Marks: because PBBAs are scaled. For backup speed, not restore speed. They don’t even have restore speed on the spec sheet anymore. And backups are not sequential operations nearly as much as you think they used to be. And you
know, when
Curtis, when when Curtis changed block, block tracking, incremental forever,
[00:29:53] W. Curtis Preston: Right?
[00:29:54] Howard Marks: of those things make the backup and the restore much more random.
[00:29:59] Prasanna Malaiyandi: yep,
[00:30:00] Howard Marks: And so if you’re backing up to a disc based p v a, your restore speed is like a fourth or a fifth, you’re backup speed.
[00:30:09] Prasanna Malaiyandi: yep.
[00:30:09] Howard Marks: If you’re backing up to a vast, your restore speed is five times your backup speed. Cause we’re, cuz we are designed to serve. Re primary storage applications where reads happen much more frequently than writes cuz the reads come from all the capacity SSDs, the writes have to go to the s scm.
Um, and so what, where that really starts to get important is when, when we start talking about ransomware attack, cuz 10 years ago Curtis and I used to teach seminars and we’d go, yeah, 90, 95% of your restorers are, you know, the file. Somebody screwed up. And you know, if it’s on A P B B A it’ll be restored in a couple of minutes.
And if it was on
tape, you’d go find the tape and then a couple of minutes. And so, but you don’t know you’ve been ransomware attacked till thousands or hundreds of thousands of files have been encrypted.
[00:31:05] Prasanna Malaiyandi: Yep.
[00:31:07] Howard Marks: And so now you have to like use something like instant recovery to check back, you know, is this backup good?
You gotta do three or four quick looks without restoring, which
is a great feature, but you know, requires a relatively high speed backend to work relatively well. And then you’re gonna find, okay, this is my last non good point. And then you have to restore and you are gonna have to restore a lot of data and restore speed starts to become really important then.
[00:31:39] Prasanna Malaiyandi: Mm-hmm.
[00:31:40] Howard Marks: And then the kicker is, and the lawyers in the insurance company won’t let you use the, the system that was infected for another couple of weeks cuz it’s evidence or we have to get somebody in to clean it and certify that it’s cleaned well, if you know you can run a VMware NFS data store on VAs, you can just restore to VASc.
Now it’s a bad idea to run your primary and your backup on the same system for more than a day or two,
[00:32:12] W. Curtis Preston: Right,
[00:32:12] Howard Marks: but, Compared to not running your primary and just, you know, if your choice is backup only or primary and backup, and if this one system dies, I’m really in trouble. Not that part of choice for me. I want my users back up and running.
As soon as the lawyers let me get tacked to the old system, or my VAR gives me a new system, or I have someplace else to storage, VMO
to, I’m getting that stuff off there right away. But that might mean I’m up a week earlier and a week earlier is a lot of time.
[00:32:45] W. Curtis Preston: my objection to flash for backup has been for two primary reasons.
One is, is expensive af right second. Do I really need it? Right? Like, because there’s, there are a lot of things that we can buy in life, right? Uh, like I, I need to move fertilizer. I can totally borrow Prasannas, uh, Tesla and it will do it, right? But, but is that what I should be using for that? Do I need, do I need a Tesla to move fertilizer or will
my Prius do
[00:33:53] Howard Marks: need Prasannas. Tesla to move fertilizer if you ever want prasanna to speak to you again.
[00:33:59] W. Curtis Preston: no, that’s, that’s true. By the way, the Prius has been used to move fertilizer just for the record. Um, but so, so that’s the thing. It’s like, there, there are a lot of things, like, this goes back to the c d P, the c d P argument that I made back in the day. It was the same thing, the same two arguments.
One was c D P was too damn expensive, right? And then the other was, does anybody actually need. The, the, the, the functionality that C D P provided. And the answer is yes. 0.1% of the population needed what C D P provided. And that’s why you don’t really see c D P as a choice very, very often these days.
Right there, there, there’s a one or two companies that do it now, um, and all the other products have died. So those are my two arguments. It’s, I, I already know what your argument to the second one is gonna be because you just gave it, I think. Um, so
why
[00:34:58] Prasanna Malaiyandi: about for cost?
[00:34:59] Howard Marks: Well,
[00:35:00] W. Curtis Preston: talk about costs?
[00:35:01] Howard Marks: so, for cost, it depends what flash systems you’re talking about.
[00:35:05] W. Curtis Preston: Mm-hmm.
[00:35:06] Howard Marks: Um, I will give you that most all flash systems are designed to be fast as possible for a small amount of data, because that’s what you need to run the Oracle databases that make companies work. And so if you, you know, it’s a block, you know, it’s block storage to be low latency to support O L T P and therefore expensive because that’s, you know, if that system goes down, you count by the second how much money you’re losing.
And so you have always bought expensive storage for that. Um,
[00:35:45] W. Curtis Preston: sort of the, sort of the true normal sort of, if I, if I can use this word, pure flash array.
[00:35:55] Howard Marks: Yes.
[00:35:56] W. Curtis Preston: the, that type is designed for that, right? Um, that’s technically pure with a small p, but it works the other way as well.
Um,
[00:36:06] Howard Marks: talking either way, you know? Yeah. You know, I could name half a dozen other products, but
[00:36:14] W. Curtis Preston: And they’re just too expensive.
[00:36:16] Howard Marks: of it, you know, it’s, we’re gonna design a system based on having a, a pyramidal tiered system.
this is the one at the top.
[00:36:26] Prasanna Malaiyandi: Yeah.
[00:36:27] Howard Marks: And if you assume you’re gonna build a tier system, then you want the one at the top to be as fast as possible, and you kind of don’t care how much it costs because
you’ll just put stuff that doesn’t deserve it on the next tier.
[00:36:40] Prasanna Malaiyandi: Yep.
[00:36:40] Howard Marks: Philosophically our idea was we’re gonna make something that delivers performance for everything but the very, very top there.
[00:36:50] W. Curtis Preston: Mm-hmm.
[00:36:50] Howard Marks: And goes down in cost to where Well, if you use enough of it, you don’t need those tiers. You don’t need the complexity. Right. So, you know, part of our story is as you consolidate workloads, you have workloads that need performance, and you have workloads that need capacity.
When you add capacity, performance comes with because in,
you know, spindles, how many SSDs? Yeah. A hundred SDS is so much performance. 200 SDS is twice that much performance. So if you take the applications that need capacity, And you put them on the same system as the applications that need performance but don’t need capacity.
The performance that the capacity creates is used by the applications that need the performance and the cost of the performance is brought down because you’ve used that much capacity and you get in a vir virtuous cycle.
[00:37:53] W. Curtis Preston: I think I followed that.
[00:37:56] Prasanna Malaiyandi: yeah, it, it, it’s basically
by
[00:37:59] Howard Marks: if you, if
[00:37:59] Prasanna Malaiyandi: that’s
common.
[00:38:00] Howard Marks: if you, you’re, paying 10 x for 10% and one x for 90%, then you’re paying a hundred. If you have one tier that costs a hundred,
[00:38:18] W. Curtis Preston: Mm-hmm.
[00:38:19] Howard Marks: why have two tiers?
[00:38:20] Prasanna Malaiyandi: Yeah.
[00:38:22] Howard Marks: And when you use capacity, that capacity comes with performance.
[00:38:29] W. Curtis Preston: Mm-hmm.
[00:38:29] Howard Marks: And that means that performance is available to other applications that didn’t need the capacity. So you don’t need to have separate systems, you just have
[00:38:39] W. Curtis Preston: So, so if I could, if I could try to put this in, in, in just different words, but it’ll say the same thing. If I’ve got a hundred QLC disks, right. Um, and, and these are how big
[00:38:53] Howard Marks: 15 or 30 terabytes.
[00:38:57] W. Curtis Preston: the each, each one, right?
[00:38:59] Howard Marks: Each one
[00:39:00] W. Curtis Preston: So if I’ve got a hundred, I’ve got one and a half tear, one and a half petabytes. Did I
do that
right of raw? Yeah. Okay. All right. So I’ve got one and a half petabytes of raw capacity. And what you’re saying is we can just take a slice off the top, if you will. You know, we used to call short stroking the discs.
The, obviously you don’t need to short stroke a, a flash, but you’re basically saying, we’re just gonna take a slice off the top, uh, of these 150 discs and we’re gonna get this massive performance slice, uh, for, for the 10% that need that performance. And then the rest will just put wherever we need to put it.
Is that, Does that sound about
right?
[00:39:42] Howard Marks: I’m, saying all those SSDs create one pool of performance and one pool of capacity,
[00:39:49] W. Curtis Preston: Right.
[00:39:50] Howard Marks: and a workload can draw from either one as much as it needs.
[00:39:53] W. Curtis Preston: Gotcha.
[00:39:56] Prasanna Malaiyandi: Is it separate capacity and performance pools that you
then assign to Applic? Okay. It’s just one
pool that includes both
[00:40:06] Howard Marks: that, and now, you know, and you can use q o s, you can say, okay, this workload gets a hundred thousand iops, or, or 50 gigabytes per second, and this other one gets
different.
[00:40:19] Prasanna Malaiyandi: yeah,
[00:40:19] W. Curtis Preston: yeah,
And
you’ll just use
[00:40:20] Howard Marks: And so you
[00:40:21] W. Curtis Preston: you need to
[00:40:22] Howard Marks: performance, right? But you know, if you’ve got, um, you know, your backups and you’ve got the developers who wanna do run, live copies of the database, well run it all on one system.
It’s, it’s an all flash system. It’s fast enough to run the database.
[00:40:44] Prasanna Malaiyandi: It’s almost as if you’re saying, You’ve built a system that works for all workloads except that 1% or whatever, that’s like that very, very, very high end. And you’re saying you have one common architecture that allows it to deal with, regardless of if your workload is capacity focused and not very performance, it doesn’t need a lot of performance or it’s high performance and maybe a little capacity.
It’s all a
[00:41:09] Howard Marks: and and it doesn’t matter whether your definition of performance is bandwidth or iops. You know, it’s like all but that very lowest. You know, we, you know, we’re an all flash system lightly loaded. We deliver one millisecond latency.
[00:41:23] Prasanna Malaiyandi: yeah,
[00:41:23] Howard Marks: You know, some systems can deliver half that and some rare applications care.
But you know, between that and the 10, Tencent, a gigabyte, well, there are 20 terabyte hard drives and super micro servers and you know, they don’t do any iops, but you can write to ’em pretty fast. You know,
[00:41:45] Prasanna Malaiyandi: Yeah.
[00:41:45] Howard Marks: in between we can cover.
[00:41:47] W. Curtis Preston: So we’re dancing around. You’re saying why you could be cheaper, but let me,
let me just put a, lemme just put it right, you know, sort of, I’m assuming that you get into competitive bids with PBBAs on a regular basis.
[00:42:08] Howard Marks: Yes, sir.
[00:42:09] W. Curtis Preston: Okay. How do you do there?
[00:42:14] Howard Marks: They’re easy. Those are very high profit margin products for
the
[00:42:19] W. Curtis Preston: so you’re, so you’re, saying you can come in less expensive than the effective price of the typical P B B A, even though you’re using all this flash.
[00:42:29] Howard Marks: Yes, sir.
[00:42:30] W. Curtis Preston: Okay. Because that, that’s the short answer. I like the long answer. That’s, I like the long answer. You and I
live in long, right.
Um, yeah, but in the end, it doesn’t matter if it’s still more expensive.
[00:42:44] Howard Marks: yeah, the, you know, the long answer is, you know, we use the cheapest flash we can get because we designed the system to treat flash well and understand how to minimize wear. We ha our erasure codes have 3% overhead at I at large scale, so we’re not wasting.
Space on raid, we reduce data better than anybody else does. So you know, we’re getting as much capacity in there. Um, and then when you start saying, okay, it’s 30 terabyte SSDs, so you get a lot of capacity and a little bit of space and a little bit of power, and the power and Rackspace start to add up as costs.
Um, especially when you start looking at the fact that the leading PBBAs are still using eight terabyte hard drives because that rehydration tax of turning,
[00:43:40] W. Curtis Preston: Hmm.
[00:43:41] Howard Marks: making everything random, well the bigger the hard drive gets, the worse it is cause.
One hard drive is a hundred iops.
[00:43:49] Prasanna Malaiyandi: Yep.
[00:43:50] Howard Marks: Doesn’t matter whether it’s a one terabyte hard drive or a 20 terabyte hard drive.
And so they’re just reaching the, the world. The land of diminishing returns on IO density. They can’t go any lower. And now the sheet metal and the power supplies and the SaaS expanders are becoming a larger and larger percentage of their cogs. And they mark ’em up a lot cuz there’s a lot of IP in there in terms of
software and they have to make a margin.
Um, and so we just don’t have most of those problems.
[00:44:30] Prasanna Malaiyandi: Yep.
[00:44:30] W. Curtis Preston: you’re, you’re, also marking up due to your ddo, right? I mean, you
[00:44:36] Howard Marks: Yeah. Or you know, some of it, you know, some small portion of the difference is, you know, compared to the guys, those the best P VBAs, we still do a little bit better.
but but when we go into a customer who says, no, no, no price, this as if you de-dupe the same, we’re still coming in with a lower selling price.
[00:44:58] Prasanna Malaiyandi: Yeah, I, I’m not surprised about that. The other thing, Howard, I wanted to bring up, I know you mentioned sort of dis drives and the a hundred iops limit, right? That each of them typically have. The other thing that I’ve also seen is as the drives get larger and larger, anytime you have to do a raid, rebuild, right?
And you’re talking like a 20 terabyte drive and it just takes longer and longer, and now there’s a potential for failure,
[00:45:22] Howard Marks: Yeah. Well,
[00:45:23] Prasanna Malaiyandi: becomes a lot worse.
[00:45:25] Howard Marks: you know, I do a lot of, you know, resilience calculations and people just don’t realize how big a factor rebuild time is in the probability of data loss. Uh, we had one customer share with us. They ran, you know, the leading. Scale out system before us and the average for their rebuilds was 53 days.
[00:45:54] Prasanna Malaiyandi: Oh
wow.
[00:45:57] W. Curtis Preston: That’s two months.
[00:45:59] Howard Marks: yeah, that’s two months during which time your data is exposed and you know, could be slightly exposed if you are already running. N plus three could be really exposed if you’re running n plus one, like some vendors recommend. So it all depends.
[00:46:20] W. Curtis Preston: right. Okay. So, so I think, I think, you know, you, you’ve definitely covered the cost argument. Um, the, and, and it’s, I think if we just back up, you’ve already covered the why, right? The why.
would, why
is today’s Restore different?
[00:46:41] Howard Marks: Ransomware.
[00:46:42] W. Curtis Preston: Yeah.
[00:46:43] Howard Marks: stores. The stores bigger. And the restore location is less well known
[00:46:49] W. Curtis Preston: What do you mean by that?
[00:46:50] Howard Marks: you may not be able to restore back to the infected system cuz it’s still evidence,
[00:46:54] W. Curtis Preston: okay. Understood. Okay.
[00:46:57] Howard Marks: right? You need someplace to restore to. And you know, having it where the primary in the backup are duped to each other probably gives you that in a pinch.
[00:47:08] Prasanna Malaiyandi: Yeah. Have you
seen.
[00:47:10] Howard Marks: emphasize in a pinch, cuz
[00:47:12] W. Curtis Preston: Yeah,
[00:47:13] Howard Marks: you know, we’ve, we’ve all been trained, no, no bad idea. Don’t mix the strip, don’t cross the streams.
Um, but, but that’s the, if you have a backup on the primary, you don’t, you don’t have a backup. But if you have to choose between backup and primary, I’d rather have primary.
[00:47:30] Prasanna Malaiyandi: Have you seen customers actually do this in the field with fast systems?
[00:47:36] Howard Marks: Oh, we have several customers doing really large scale backup to Vasst. Um, we had one customer who was kind of shocked cuz they were doing encryption in net backup. And so they expected us to not reduce data at all, uh, but they were doing encrypted net backup backups of Oracle dumps of the same database over and over again, encrypted with the same encryption key.
And we started seeing about 20% reduction just because even when you encrypted, if you’re backing up the same data, it looks
the same encrypted as it
[00:48:15] W. Curtis Preston: right.
There’s like sort of two questions in my head here. One is, and, and they’re, they’re very much related. One is the, the whole backup container problem, right? Meaning that you get the net backup container and the. Arc serve container and the backup exec container, you know, and they all stored backup data differently.
And you have that issue. And then you, but you, there was something that you alluded to that I found interesting. You said commonality between the backup and the primary, but the backup is in some weirdo format. So are you able to get backup or commonality between the backup and the primary?
[00:48:56] Howard Marks: Now in that case, it’s
more likely we’ll see commonality between multiple primaries. You know, it’s more like you
restored 17 windows VMs
[00:49:04] W. Curtis Preston: Does the way that you’re doing ddo make the format problem any less problematic? Right.
[00:49:11] Howard Marks: O Only in that we reduce them all as opposed to if you were relying on the data movers to do reduction. So, you know, kind of the most common case is the storage guys like backup, you know, com Vault or net backup or Veritas or Veeam or whatever they use. And the Oracle DBAs don’t trust them and insist on doing, doing, dumps.
[00:49:37] W. Curtis Preston: Right.
[00:49:38] Howard Marks: And so, you know, if you’re doing both to, you know, they just give the Oracle DBAs, okay dump to this NFS mount point on the vast. Well then we’ll reduce all of those dumps as well as anybody could reduce all of those dumps and your data mover. You’ll do data reduction at multiple stages to manage the network traffic.
And then we’ll do the final dup at the end cuz we’re finer grained. And the sim similarity works really well for things that are duped course grain, cuz the edges all look similar. And so when
we, when we run, you know, we have a probe you can get as a VM that scans your data and reports back, this is how much it would reduce.
And this is how much of that comes from each of these techniques. And so when we run, when we do that with data from a data mover, d duper, those are usually, you know, 128 K or big blocks because they have limited memory available. And so we see more similarity cuz we’re finding those pieces Finer
[00:50:52] W. Curtis Preston: just to, just to make sure I understood correctly. So the, one of the question that I didn’t really ask was, you know, when you buy a, you know, pick your favorite P V B A, they tend to support. These five backup products, and if you buy a different backup product, well, they’re like, well, we don’t understand that format yet.
And so then they have to go and do some development work to figure out how to crack that container. Do you not have that problem or have you done that
[00:51:23] Howard Marks: We, We, have not optimized for any of these backup applications,
[00:51:29] W. Curtis Preston: And yet you
still get better duped than the other guys.
[00:51:33] Howard Marks: Our, our general case data reduction against all of these reduced data types still gets better reduction. You know, we are not, you know, scanning for the timestamps in Oracle rack dumps and, you know, that level stuff.
[00:51:51] W. Curtis Preston: Right.
[00:51:51] Howard Marks: Not to
[00:51:52] Prasanna Malaiyandi: agnostic, right?
[00:51:52] Howard Marks: to say we won’t in the future, but our, you know, our data reduction was written for primary storage. It just so happens that being an
N F S or an S3 target for a backup data mover is a simple case of primary storage,
[00:52:11] Prasanna Malaiyandi: Yeah.
[00:52:11] Howard Marks: it just works.
[00:52:13] W. Curtis Preston: Right. Right. Hmm. What do you think Prasanna,
[00:52:17] Prasanna Malaiyandi: in the future? So I, I had no complaints to start with. Uh, the one question
[00:52:24] W. Curtis Preston: I lost this argument? I think I might
[00:52:26] Prasanna Malaiyandi: I think, I think you lost this one. Uh, Howard, the one last question I had was, I know some of these backup vendors support the ability to do source side due duplication by integrating with the purpose-built backup appliances.
Does VAs support that? Are you guys planning to support that? I know you’re looking, you just
previously said, right, that you’re
[00:52:47] Howard Marks: don’t, we don’t, um, I’ve never been really comfortable with the use of client side CPU for that cuz client side CPU is valuable for other things. Um, I think, you know, doing a pass at some level in the data mover.
[00:53:08] Prasanna Malaiyandi: Mm-hmm.
[00:53:09] Howard Marks: It’s like, okay, we’ll we’ll de-dupe at the media server at some large grain so that we’re not transferring 50 copies of windows over the network.
Um, is perfectly reasonable thing to do cuz it’s a network bandwidth management technique. Um, things like Didi Boost are, you know, let’s offload this from the, the P B B A to the client and we’d just rather do the work ourselves. Um, and in our architecture, since you can just add more servers at the front end
and you just have to buy the servers, we don’t even charge for that software.
If you need more compute. To do more
[00:53:51] Prasanna Malaiyandi: you just scale out.
[00:53:52] Howard Marks: to more dup, you just add a few more servers as opposed to stealing 5% of the cycles of all of your VMware hosts, which means you now have to not just buy servers, but you have to buy another VMware host, another VMware license. All the other stuff you put on a VMware host starts to add up.
[00:54:08] Prasanna Malaiyandi: Yeah.
Gotcha.
[00:54:10] W. Curtis Preston: Well since, well, since you stepped into my neighborhood now Howard, I will have to say that source side, DUP done correctly, speeds up the backup and reduces the C P U utilization on the client. But I, I want, I can’t speak to the, to the implementations you were talking about. Um, I can only speak to the one that I am obviously very familiar with.
Um, cuz there, you know, there that, that is the off discussed thing of like, well there is a,
[00:54:41] Howard Marks: It’s
[00:54:42] W. Curtis Preston: know, there’s a
pen.
[00:54:43] Howard Marks: it’s also a different case because of the assumed bandwidth at all the stages.
[00:54:48] W. Curtis Preston: right.
[00:54:49] Howard Marks: You know, I’m, I’m kind of assuming that there’s a lot of bandwidth for short
distances in the data center
[00:54:56] W. Curtis Preston: Well, all right. I, I concede this battle, Howard, I lay down my sword. Um,
[00:55:05] Howard Marks: Okay. I
[00:55:06] W. Curtis Preston: you know, I mean, you, what’s that
You expect
to what?
[00:55:10] Howard Marks: in the mail,
[00:55:13] W. Curtis Preston: Um, yeah, I’ll send you, I’ll send you something. Um, alright, well, uh, Howard’s been great. Uh, glad to hear the update and glad to, you know, I, I remember we did, now that I heard you describe it, I, I think we did cover it in the last one, but I think you went deeper this time and that, that’s good to hear this idea
[00:55:37] Howard Marks: probably.
[00:55:37] W. Curtis Preston: you can, that you have the, that you have the, the bandwidth to, to, to, to how many different ways did you say you try each block
for
[00:55:49] Howard Marks: there’s five compression algorithms and, and then there’s a strong hash and a number of similarity hashes. I can’t remember offhand what they are,
[00:56:05] W. Curtis Preston: Gotcha.
I got, I thought I heard you say 15 total ways. I thought I
heard you say
that.
[00:56:11] Howard Marks: it on that order
[00:56:14] W. Curtis Preston: Gotcha. So the fact that you can take each chunk and try 15 different ways to compress it and pick the one that works the best is pretty damn cool. Um, and, um, you
know, it’s just on
[00:56:28] Howard Marks: of it’s just cuz we can parallelize it so well
[00:56:30] Prasanna Malaiyandi: Yeah.
[00:56:31] W. Curtis Preston: that too, right? Uh, the
fact that, you can, you know, that’s a.
[00:56:35] Howard Marks: one takes. We’re doing a lot.
[00:56:37] Prasanna Malaiyandi: Yeah,
[00:56:38] W. Curtis Preston: it’s that, that beauty of the scale out architecture, right? That you can just pass that out, um, like that. All right, well, thanks for coming back, especially to, it’s really funny how we had you, you’re like, you’re listening and you’re like, Hey, you said mean things about the way I do things. I will, I accept your challenge.
And I’m like, all right, come on back. Uh, happy to do that. And we’ll do that with other people. By the way, if you’re out there listening and you’re like, the thing that Curtis or Prasanna said is wrong, we will be happy to have you on and have us prove to you why you’re wrong or, or in this case,
In this case, uh, right.
Yeah, we concede. Well, I mean, I mean, I think my concerns are certainly valid and there are certainly vendors out there that are like, yes, we can certainly sell you this appliance for the purposes of backup, because recovery speed is really important. I’m like, but it costs five times the cost of this thing over here.
I don’t like how much better could it possibly be Anyway, so that’s, that’s where those arguments tend to come from, so. Alright, well thanks.
Thanks Howard for
[00:57:52] Howard Marks: and given, given the marketplace, they’re not completely unreasonable.
Uh, you know, we, we just do things sufficiently different that, you know, if you think restore speed’s important than we do,
[00:58:05] W. Curtis Preston: right, right.
[00:58:06] Prasanna Malaiyandi: Yep.
[00:58:07] W. Curtis Preston: Yeah. Yeah. Ransomware. Ransomware. All right. Once again, ransomware, you know, I don’t know. What do you, what do you call it? Uh, trump’s all, um, although I don’t enjoy that word as much as I used to for some reason. Um, anyways, thanks for, thanks for your questions for,
[00:58:30] Prasanna Malaiyandi: Uh, I try. I try. And Howard, great to have you back on the podcast. Hopefully you’ll come again.
[00:58:35] Howard Marks: Always pleasure. As long as I keep winning, I’ll keep coming back.
[00:58:40] W. Curtis Preston: and we thank you to our listeners. Uh, you know, we’re nothing without you. Remember to subscribe so that you can restore it all.
