Vast is a massively-scalable storage system designed around multiple pieces of technology that weren’t available just a few years ago (e.g. NVMe, Storage class memory, QLC) that offers both file and object functionality, immutable snapshots, and integration with the cloud to address the “smoking hole” problem. Their typical sale (of which they’ve made many) is north of $1 million, and they have many exabytes of disk in the wild. It’s a scale-out storage system without all the typical East-West traffic such systems have. We do our best to poke holes in their offering, but Howard Marks goes toe-to-toe quite well. This one went a little long (one hour) but we truly were fascinated with the Vast story Howard was telling.
[00:00:00] W. Curtis Preston: I’m pretty sure we’ve said smoking hole more times than we’ve ever said on this podcast. Just for the record. Just saying
[00:00:07] Howard Marks: Well,
[00:00:07] W. Curtis Preston: um, it’s getting a
lot of play today.
[00:00:29] W. Curtis Preston: Hi and welcome to Backup Central’s Restore it All podcast. I’m your host, W. Curtis Preston, AKA Mr. Backup and I have with me, my Bhangra dance consultant, Prasanna Malaiyandi, how’s it going Prasanna?
[00:00:41] Prasanna Malaiyandi: I’m good Curtis, but I have to warn you. I have not a dancer at all. So probably the wrong person to be seeking advice about dancing from,
[00:00:50] W. Curtis Preston: But you said that you knew about Bhangra dancing and that you could advise me on these things.
[00:00:55] Prasanna Malaiyandi: well, I told you that it’s like a Indian dance style, if you will. And you had asked a question of, have I seen it because I bet I’ve seen a bunch of Bollywood movies.
[00:01:07] W. Curtis Preston: You expanded my horizon, so, I, I bought my tickets, uh, my wife and I will be going to see the show it’s called Bhangin’ it! It’s it’s banging, bangin’ was spelled would be H so it’s like, it’s trying to like, you know, do an homage to the Bhangra. Um, so it’s a, a new musical at the LA Jolla Playhouse, which is a very nice, uh, Playhouse that I’ve actually never been. I’ve lived here 20 something years. I’ve never watched a show there, but a lot, a lot of like big Broadway shows actually start out.
I’ve never started. I’ve I’ve, I’ve always watched the Broadway shows after
[00:01:44] Prasanna Malaiyandi: Like gone to Broadway
[00:01:47] W. Curtis Preston: This is the kind of show that could possibly hit big on Broadway. And so we’ll see it, you know, and, and, uh, we’ll see if it’s any good, you know, and I’ll, I’ll
[00:01:56] Prasanna Malaiyandi: waiting for rave
[00:01:57] W. Curtis Preston: review.
[00:01:58] Prasanna Malaiyandi: Yes. I think our listeners will be curious. And by the way, for those in San Diego, when is it running? Do you know how long?
[00:02:06] W. Curtis Preston: It’s running. It’s running until April.
[00:02:09] Prasanna Malaiyandi: Okay. So that
[00:02:10] W. Curtis Preston: So it might, Yeah.
Yeah, depending on when this goes live, if it goes live less than a month from now, then you have like two days left to go see it because it runs until April 17th at the LA LA Jolla Playhouse, by which time all the tickets will be gone and you won’t be able to see it.
So, sorry, I don’t know what to tell you, but, um, so we, we have a longtime friend on the podcast here today. Prasanna. I’m excited to bring him on I, and not just because he’s one of those people that make me feel young. Um, as, as been in the it industry for an awfully long time, makes me feel like a young whippersnapper sometimes.
He is now the technologist extraordinary and plenipotentiary at Vast Data.
Welcome to the podcast, Howard Marks.
[00:03:09] Howard Marks: Thank you. It’s very nice to be here. Uh, I was always about Fauci guy, so I don’t know much about Indian dance.
[00:03:16] Prasanna Malaiyandi: Curtis didn’t either before he met me. So it’s fine.
[00:03:21] W. Curtis Preston: I, I, my knowledge of, of Indian dance, you know, it basically includes, uh, the reference to it in, uh, what was that movie? Uh, the bright, the BR the bride and prejudice.
[00:03:39] Prasanna Malaiyandi: Oh
[00:03:40] W. Curtis Preston: There’s a, there’s,
[00:03:41] Prasanna Malaiyandi: yeah. I th I think
[00:03:42] W. Curtis Preston: there’s a, it’s a pride and prejudice,
[00:03:47] Prasanna Malaiyandi: yeah.
[00:03:47] W. Curtis Preston: like knock off done with what’s her name? .
[00:03:51] Prasanna Malaiyandi: Ashwaryia Rai. I think.
[00:03:54] W. Curtis Preston: She remember she, she, in the movie, she, she gives two D two dance moves. It was petting the dog and screwing in the light bulb. I don’t know if you remember that. She says that. Uh, that’s literally the extent of my knowledge of Indian dance. Well, that, and the fact that I’ve watched a bunch of Bollywood movies, but that’s all thanks to Prasanna.
[00:04:13] Prasanna Malaiyandi: Yeah.
[00:04:15] W. Curtis Preston: So you never know what you’re going to get when you’re listening to the Backup Central Restore it All podcast. Speaking of which, let me throw out our usual disclaimer, Prasanna and I work for different companies.
Persona works for Zoom. I worked for Druva. This is not a podcast of either company and the opinions that you hear are ours, and be sure to rate this podcast ratethispodcast.com/restore, or just go at your, you know, on your favorite pod catcher, you know, like, uh, apple podcasts and just scroll down to the bottom and, and give us some stars.
And if you really want to make my day, actually put some words there. Yeah, absolutely. And, um, if you are interested in the things that we’re interested in, like backups and storage and resilience and ransomware recovery and cyber warfare and all of these things. Uh, then, then, you know, just send me a note @wcpreston on Twitter, or wcurtispreston@gmail, and I’ll be happy to, uh, get you on the podcast.
[00:05:13] Prasanna Malaiyandi: We’re friendly. lots of questions.
[00:05:17] W. Curtis Preston: we even apparently, although the last episode I said, unless your name was Stewart and apparently Stewart has now reached out to you Prasanna
[00:05:25] Prasanna Malaiyandi: Yes, he has. He has.
[00:05:30] Howard Marks: So even if your name
[00:05:31] W. Curtis Preston: Even Stuart can get on this podcast. So if we’re going to let you know a mouse on the podcast, then, um, surely we can let you, his name is Stuart Liddle for those of you that didn’t get that reference
[00:05:45] Howard Marks: way to make me feel honored here.
[00:05:50] W. Curtis Preston:
We literally let anybody in the door,
including guys who always wear Hawaiian shirts.
[00:06:00] Howard Marks: um, you know, they’re comfortable. They come in my size and at this point I’m just known for them. I have been known to tell people I’m going to meet at the Starbucks at some conference. Um, just look for Santa Clause in an Aloha shirt. That will be me.
[00:06:20] W. Curtis Preston: Pretty much. It pretty much narrows it down.
[00:06:24] Howard Marks: Yeah. Well, you know how many 350 pound guys with a gray beard are there walking around the average tech show, wearing an Aloha shirt? Two.
[00:06:33] W. Curtis Preston: I’m going to,
yeah. Two, yeah. At most. Absolutely. And one of them is going to be you. So, so how long have you been at Vast Data?
[00:06:45] Howard Marks: I’ve been at Vast Data, uh, three years and 15 days.
[00:06:50] W. Curtis Preston: Wow.
[00:06:52] Prasanna Malaiyandi: And the company is fairly new as well. Right?
[00:06:55] Howard Marks: Um, I joined Vast Data the day before we came out of stealth. Um, my, my first official act at Vast Data was a briefing for Chris Mellor, um, followed the next day by Storage Field Day.
[00:07:09] W. Curtis Preston: Wow.
[00:07:10] Howard Marks: You know, nothing like starting off running
[00:07:14] W. Curtis Preston: Right, right,
[00:07:16] Howard Marks: I joined Vast from being an independent analyst. So there were a couple of weeks there where I was, you know, getting brought up to speed and such before my official start date. But yeah,
[00:07:30] W. Curtis Preston: And why don’t you give a, you know, for those that aren’t familiar with Vast Data, give us a, you know, the elevator
[00:07:40] Howard Marks: sure.
[00:07:40] W. Curtis Preston: and
[00:07:41] Howard Marks: The really short form on Vast Data is that we make very large scale all flash file and object storage systems. And when I say very large scale, um, our average selling price for our cluster is well on the north side of a million dollars. It’s multiple petabytes. Um, today we’re just introducing a new storage enclosure that brings our building block down from 675 terabytes per HA enclosure to 338.
So we’re taking it down by factor of two. We’re going from a two U to a one U enclosure. We’ll talk about that in a little bit, but the innovative thing about Vast is the architecture. If you talk about a large scale system, like we build traditionally, that’s been done with a scale out, shared nothing model where you have a lot of x86 servers.
Each of those x86 servers owns some set of media and they communicate on a backend network and software makes it look like one big system. Um, but those systems start to break down at really large scale. And so we’ve come up with a new model. We call DASE the dis-aggregated shared everything architecture instead of having a field of peer nodes, each of which owns some media, we disaggregated the media into these HA enclosures that I was just talking about.
So no single point of failure, 400 gig connections to an NVME fabric and that’s typically a hundred gig Ethernet. Some of our HPC customers like to run InfiniBand so we can do InfiniBand as well.
[00:09:45] W. Curtis Preston: Um,
[00:09:47] Howard Marks: All those enclosures do is hold data. There’s no services there. All of the services, everything that you would think of as the controller function of the system runs in stateless Docker containers in the front end servers. So when a user makes a request to a protocol server to one of those front end servers could be NFS, could be SMB, could be S3. That server looks in the metadata that’s stored in storage class memory in the enclosures, finds the data the user’s requesting in the data in QLC flash in those same enclosures, retrieves it over the NVME over fabric’s fabric and delivers it to the user. So there’s none of the traffic from node to node required to reassemble data, everything’s north, south across that NVME over fabrics connection.
And since the metadata is in storage class memory, it’s fast enough to directly access by all of the front end servers that they can just share it. They don’t have to cash it.
[00:11:11] Prasanna Malaiyandi: So
[00:11:12] Howard Marks: having the cache, we don’t have all the complexities of keeping the cache coherent.
[00:11:17] Prasanna Malaiyandi: I was just going to ask about that, Howard. So it looks like though you’re sort of, dis-aggregating the actual storage and metadata from sort of all the front end processing, which allows, I would assume the front end to scale independently of the backend.
[00:11:31] Howard Marks: right. So each of those front end protocol servers, mounts all of the SSDs in the cluster at boot time. And then it looks at all of those SSDs, like. And at those are the SCM SSDs that hold the metadata and the QLC SSDs that hold the data. So everybody has access to everything. And instead of sending messages back and forth between the front end servers, they simply write a single version of truth in the shared metadata, so that the old so that you can place a lock on the metadata or update the metadata.
But you never have to tell everybody else you updated it because if they want to know what the state is, they’ll go look in the one place where it’s true.
[00:12:22] Prasanna Malaiyandi: Yeah. And because everything is stateless in the front end, you don’t have to worry about that necessarily to everyone always trust that backend for consistency.
[00:12:30] W. Curtis Preston: So the backend has both SSDs and QLC.
[00:12:35] Howard Marks: What has SCM sort of storage class memory SSDs, and that can be Optane or Um, and it has low end QLC SSDs.
[00:12:48] Prasanna Malaiyandi: So super fast high-speed.
[00:12:51] W. Curtis Preston: yeah, the, the storage class memory is what’s holding the metadata and the
QLC is, is what’s holding the data. Right.
[00:12:59] Howard Marks: Primarily. It’s also used as a write buffer.
[00:13:02] W. Curtis Preston: Okay. Okay.
[00:13:03] Howard Marks: So writes come into the storage class memory and get mirrored to two different SCM SSDs and then get ACKd. And then the migration from SCM to QLC happens after the act. So we have more time to do things like compress more fully.
[00:13:22] W. Curtis Preston: This is a very different game than. I mean this idea of all of the front end nodes, being able to mount the entire, uh, storage in the background
[00:13:36] Howard Marks: Yeah. We, uh, we eliminate the whole concept of ownership and all the complexity that, that creates. And now I’m going to blow your mind because when I say the metadata is in the SCM, I don’t mean just the element store metadata, the metadata for our merged file system object store, but also the data reduction metadata. And so when you add another enclosure to the cluster, you add more SCM, which means you add more room for that metadata. So regardless of the size of cluster, the cluster is one data reduction realm across tens or hundreds of petabytes.
[00:14:18] Prasanna Malaiyandi: Because everything’s sort of looks like one cluster, if you will, or one system.
[00:14:22] Howard Marks: right. And, and we don’t have to hold the data deduplication hash table in memory any place. It’s all in SCM where it’s fast enough we don’t need that. So we don’t have the limitations of how big a deduplication realm can be that most deduplication systems have.
[00:14:42] W. Curtis Preston: right. They typically top out around a, you know, a petabyte or so, and then you can’t get any bigger than that. I don’t know where to start on my questions!
[00:14:54] Howard Marks: Well, so from that, from the backup point of view, we’re discovering that the customers are starting to demand higher restore speeds that traditionally all a customer worried about when they were picking the storage for their backups was, was it fast enough that I can make my backup within the window?
[00:15:21] W. Curtis Preston: Right, right.
[00:15:23] Howard Marks: And so we got systems like Data Domain and other disk based deduplicating systems, where there was a big write read asymmetry where you could write data faster to them than you could read data from them. Because reading data that caused the system to rehydrate turned sequential IO into random IO.
And they had disks on the backend. And as disk drives have gotten bigger, this has gotten worse because a 20 terabyte disk drive today delivers exactly the same number of IOPS that a one terabyte disc drive delivered 10 years ago. So now 20 terabytes of data gets a 20th as many IOPS. And so you discover, yes, it takes me eight hours to back this up. It takes me 82 hours to restore it and ransomware,
[00:16:23] W. Curtis Preston: dedupe has never been very friendly for, for large restores, especially if you’re doing any sort of, if you want to do a live mount, forget it right. From a directly, from a Data Domain. Um, it’s possible, but you know, in the same way, it’s possible that…
[00:16:39] Howard Marks: That’s, but that’s, you can bring up the Oracle or the SQL server VM. So that the it guys can access the passwords database, not so that everybody can start at running ERP on it again.
[00:16:55] Prasanna Malaiyandi: Yeah. Don’t use it as production. That’s a bad thing.
[00:16:58] W. Curtis Preston: Right.
[00:16:59] Howard Marks: Um, and so, you know, we’re discovering that, you know, people’s requirements are, are getting tighter. Um, you know, you start thinking about, uh, software as a service providers where, you know, if you run some account, some industry specific accounting as a service for a thousand customers, that’s a thousand databases.
And when something goes wrong, you want to restore those databases as fast as you can, because your customers are going to be standing over your shoulder, yelling at you. And you know, the last thing that’s kicked, a couple of our potential customers over the edge is the ransomware threat. Because the size of the restore grows so much with ransomware.
You know, you start off with, I need to protect my data against ransomware and use various methods to do that. And so we have indestructable snapshots. So you can say snapshot this folder at 6:00 AM when the backup window closes and retain it for 30 days. And even if the administrator wants to delete it he can’t.
[00:18:11] Prasanna Malaiyandi: So I wanted to talk about that.
Uh, so I did read a little small blurb about that. So
[00:18:18] Howard Marks: Um,
[00:18:20] Prasanna Malaiyandi: what prevents, like is that locked down forever? Like an admin can’t delete it no matter what, or is it just, there are additional safeguards in place to make sure that someone doesn’t compromise the admin password,
[00:18:32] Howard Marks: Anyone who ever talked to any customer of EMC Centera knows that if you build a system where you literally can’t delete data someone will get themselves in trouble and fill it a hundred percent up with junk, and it will be a bad situation. So you have to provide some mechanism for overriding this because customers will paint themselves in corners.
Um, as I said, our average selling price is well over a million dollars. We don’t have small customers who we only know third hand through VARs. We are in relatively intimate contact with every one of our customers. And so we don’t have a fixed policy that says, if you jump through these hoops, then we will let you delete the undeletable snapshots we, and the customer agree what the hoops are.
Yeah, multifactor authentication must be three of the five people on this list. They have to know the passphrase and the proper response to the passphrase. And if they respond with this other response to the passphrase, then for the next 24 hours, do not give anybody the secret, you know, as complicated as you want.
We’ll as long as we can write it down, those are the rules. And then once you’ve jumped through the hoops, we give you a time limited token that allows you to delete snapshots for a short period of time. And that token is a one-time pad. So that you can’t re it’s not good for
[00:20:26] Prasanna Malaiyandi: Yeah.
[00:20:26] Howard Marks: an hour whenever you use it.
It is good for the time when we issue it for some limited period of time. And then you have to know the next one. And so, you know, it’s just, it was the best solution we could come up with.
[00:20:43] Prasanna Malaiyandi: And this is probably helps in cases where someone attacks a company, they get access to the, to a storage system. They start deleting back-ups or what have you, right. It sort of gives you that extra layer of protection.
[00:20:56] Howard Marks: Well, I’ve seen ransomware , you know, we think of ransomware as being on the order of the viruses we’ve dealt with. And the ransomware reports I see are much more frequently and this ransomware opened a door and then someone physically hacked for a long period of time. And they took over some workstation, eventually that some administrator logged into and they have an administrator password. And, you know, if we’re just worried about, if we’re just worried about the script kiddies in a, I can protect against the script kiddies in building my backup infrastructure and architecture and those permissions. Um, but we’re talking about more sophisticated attacks than that. And frankly, you know, we talk about it as ransomware, but it’s also rogue administrator protection. Then it’s also just the, the guy who is disgruntled and decides on his way out the door, he’s going to make life for his employer. You’re protected against that too.
[00:22:08] W. Curtis Preston: Right. Yeah. Yeah. And, and sometimes rogue administrator is a true rogue administrator, meaning it’s a, it’s someone masquerading as an administrator as well. Right. Uh, that hacker that you talked about. So let me, let, let me ask, you know, You know, call it a difficult question, call it whatever you want to call it.
But when I hear about, you know, boxes that where you’re not supposed to be able to delete data, but then there is this other way where you can delete data. I immediately ask, you know, I, I have to ask the question, well, doesn’t that sort of suggest that there is a, you know, this is, I’m assuming this is a, Unix-based like, OS and that there’s that there is a root account,
[00:22:58] Howard Marks: It, we, we run in containers under linux
[00:23:01] W. Curtis Preston: So there is an account, there is a, you know, a root account and that if someone did some sort of, you know, just the right attack against that box. And again, you know, you’ve already mentioned that there is that these are sophisticated attacks. If someone did a privilege escalation attack against
the CoreOS, and now they’ve gained access to a privileged Couldn’t want
[00:23:31] Howard Marks: if someone has
administrative access to the management network, because the ports that face users as storage
ports, can’t be logged into
[00:23:51] W. Curtis Preston: Right. Okay. they’re
cause they’re back. Cause they’re backend, Right.
[00:23:55] Howard Marks: Right. So if you’re wondering, if you want to log into
Linux as root on one of our appliances, then you need, then the management network has to be set, has to be compromised. And now we start saying, are you looking for protection against destruction? Because if your data center is compromised, everything can be destroyed, but that’s not really the level of attack that we’re, we’re concerned about. We’re not talking about, you know, and someone walked into the data center because we hadn’t disabled their key card and left 20 pounds of thermite in the middle of the floor, who would do such a thing. I’ve done that on video, but you know, I was being paid. Uh, so you know, I, it is a vulnerability, but it’s kind of the generalest of the vulnerabilities. You’re pointing out that if I have sufficient
access, I can destroy anything.
[00:25:13] W. Curtis Preston: Right. The, um, but it sounds like you have protected from the rogue administrator, the stupid administrator. Right. And, um, and someone gaining access to those. But let me, let me just ask you to clarify something from your previous answer, when you said that means the management network has been compromised, what do you mean by that?
[00:25:40] Howard Marks: Well, so you manage the system through different ethernet ports, then you access the system. And so too, you’re if there’s a vulnerability where a user could log into the appliance as the Linux root user, um, well that Linux root user can only log in on the management, physical Ethernet port on the appliance, not on the hundred gigabit NVMe over fabric port.
[00:26:13] W. Curtis Preston: Gotcha. Okay.
[00:26:14] Howard Marks: so network security should keep that from being an internet connected network and available to attack.
[00:26:23] W. Curtis Preston: Gotcha. Gotcha. That makes sense. Okay. Um,
[00:26:27] Prasanna Malaiyandi: I had a question. So Howard, before we dive more into the data protection side, one thing that was curious to me was you mentioned that Vast supports file and object. Could you talk about some of the use cases that you see your customers using Vast Data? And then I think maybe some of the protection stuff will probably come alongside that.
[00:26:47] Howard Marks: Sure. Well, so, you know, we have the majority of our customers use us for primary storage. Um, you know, and that includes, you know, one of the biggest travel sites who uses us for their big data analytics and are using the S3 Presto connectors to store all of their analytic data on us. So that we’re much faster than a disk based object store, obviously.
And they can do that processing faster. Um, we have a lot of hedge funds who do time series analysis of trade data against large databases to try and predict the market. Um, we have a lot of life sciences customers who are doing things like. Molecular modeling and, uh, cryo electron microscopy where, you know, one microscope generates many terabytes of data a day because we have very high resolution images. Um, and you know, we have a major, um, motion picture studio who makes movies.
[00:27:56] Prasanna Malaiyandi: So, and so it looks like they are using both sort of the file and the object interfaces for a lot of these use cases. So specifically around data protection and backup. A lot of times you hear the vendor’s customers say, object store doesn’t need to be backed up. Right.
[00:28:17] Howard Marks: this, this is a subject that personally I find myself on the fence about that part of me goes, you know, I’ve built a huge amount of resiliency into this single system. And for durability, if you know, for, for availability, I may need to have it in another location, but for durability, assuming that the whole data center doesn’t end up being a smoking hole in the ground, um, I could get away without backing this up. I am N I remain firmly on the fence there. Um, but
[00:29:05] W. Curtis Preston: assuming you have the second copy somewhere, right. You’re going to
[00:29:12] Howard Marks: I may decide that it’s, it is data that If, the whole data center goes away, I don’t need.
[00:29:17] W. Curtis Preston: Okay. Yeah. Agreed. If, if yeah, if we have That, data, I would, I would, I would argue why did we make it in the first place, but,
[00:29:24] Howard Marks: That, you know, the risk, the risk of that is the risk of that is small enough that I’m going to go, you know, once every thousand years this is going to cost me a million dollars, but it’s going to cost me a million dollars a year to protect. So I’m going to take that risk. Um,
[00:29:41] W. Curtis Preston: So, so such I will agree to such data classes exist. I don’t run into them much, but I will agree
[00:29:47] Howard Marks: yeah. And, you know, and, and then, and then we get to the, you know, okay, so this is the object store that does a deep, uh, dispersal coding, and they have, you know, three locations and I can lose one. So do I need to back that up? Well, that starts getting really close to, well, now I need to back it up because there could be a bug in the software that loses my data. ’cause, that’s the only thing that could cause that, you know, it’s kind of like unprotected against one of my three data centers being a smoking hole. What, um, I guess, you know, again, it’s kind of like, I could see you going, I want to be safe and I can see you going, it’s not worth it.
[00:30:28] W. Curtis Preston: And.
[00:30:28] Howard Marks: Um, now for us, most of our users use us for primary storage.
And so, you know, for someone like that, big data analytics data, they may not back it up because it’s regenerate Hubble, and it’s not actually in the form it’s in on the object store, but it’s extracts from other things and they can run the ETL again and it would be really annoying, but it is replaceable.
Um, and then we, you know, and then for other use cases, you know, this is primary data. I gotta protect it. Um, and so we can do snapshots to an S3 compatible object store and back ourselves up that way. Um, or you can back us up, you know, the usual ways.
[00:31:16] W. Curtis Preston: And could you, could you use, you know, one of the, like ones that are like glacier deep archive where I hope I don’t ever have to use this. I know it’s going to cost me a crap ton of money, but it’ll save me a lot of money. In the meantime, can you use that kind of storage?
[00:31:34] Howard Marks: Um, the, the risk reading data out of that kind of storage requires a few manual steps. If you just use S3 standard, um, then data in those snapshots is available in a .Remote folder, like the .Snapshots folder in the file system. So users can do self-service restore, but that required, but this, that feature means the object has to be immediately readable. So, and so if you, if it went to
Glacier, then. And you, you know, it would be like, uh, your net backup this backup isn’t in the catalog anymore. So I got to put those files someplace where I can catalog it and then I got a catalog and then I can restore it. So if you really think you’re never going to need
[00:32:32] W. Curtis Preston: it doesn’t sound like it’s very, you know, well, it’s the smoking hole copy, right?
[00:32:38] Howard Marks: it is, it is annoying. But if it’s just, but if you’re protecting against the smoking hole, then you know, you may be willing to put up with the annoyance.
[00:32:48] W. Curtis Preston: I’m pretty sure we’ve said smoking hole more times than we’ve ever said on this podcast. Just for the record. Just saying,
[00:32:55] Howard Marks: Well,
[00:32:56] W. Curtis Preston: um, it’s getting a lot of play today.
[00:33:00] Howard Marks: I spent way too long as a disaster recovery planner.
[00:33:03] W. Curtis Preston: Yeah. Yeah. Um, so so the majority of your customers use you for primary storage, but clearly you’re trying to expand your TAM,
[00:33:15] Howard Marks: Well, w you know, we, we deliver all flash at a substantially lower price than anybody else does. You know, we start with using the cheapest QLC flash. We have a file system designed to treat that flash properly. So we never do small writes that would consume a lot of write amplification. We do very wide erasure code stripes.
So we’ve got under 3% overhead, and then we do guaranteed better data reduction than anybody else in the business. Um, and so that combination means that on an effective byte basis, from whatever backup data mover you’re planning on using, we’re going to be cheaper than a Data Domain. When you start saying that it’s, you have more than a petabyte of data and you need multiple Data Domains.
And each one of those is going to be a separate deduplication realm. Then the gap starts to grow substantially. So if so for these very large customers who have five or 10 or 20 petabytes of data across a bunch of Data Domains, simply the fact that we’re one reduction realm makes that makes us much more efficient than that can be. And it’s one system to manage. It’s one namespace it’s one 20 petabytes or 50 petabytes system.
[00:34:52] W. Curtis Preston: Hmm. So you’re saying, so let me just make sure I understood what you said there correctly. You’re saying on a, regardless of the size of the system, you should be priced competitive with a Data Domain, but then the bigger you get,
[00:35:07] Howard Marks: I haven’t done anything under about 500, any pricing experiments under about 500 terabytes, because we’re interested in the large end of the business, but yes.
[00:35:16] W. Curtis Preston: Right.
right. That is interesting though, that sort of. Jump into that end of the business. Right. And you, you know, you, you had another, th you know, there was another large, all flash competitor that’s doing very well, but they have a very different architecture, right. They’re referring of course, to the orange company. Um, and they have a very different architecture than you.
[00:35:44] Howard Marks: um, well, you know, if you’re talking about Flash Blade, that’s really a shared nothing architecture it’s instead of being pizza box servers, they’re blade servers, and each blade has flash modules built in, uh, And they, you know, they don’t scale nearly as large.
[00:36:07] W. Curtis Preston: So it, so it sounds like you, you just took, you’ve built an architecture based on several new pieces of technology that simply weren’t available, say, five years ago,
[00:36:21] Howard Marks: Yeah. We, we, we are the storage system designed from a clean slate around the 2016 toolbox. So QLC, flash, SCM, NVMe over fabrics and other people shoe horn one or two of those technologies into an existing architecture, but we built the whole architecture around having those technologies. Yeah, putting all of the metadata in SCM with no cache meant it had to be in SCM.
And it meant the connection between the compute server and that SCM had to be fast enough that we weren’t going, you know, if we cached this, it would be a lot faster. So that meant it had to be NVMe over fabrics. And then the QLC flash gives us the cost. But it, it, it really is, you know, if you look at any storage system, it’s by definition built with the parts that the industry is making when they sat down to design it.
[00:37:35] W. Curtis Preston: Yeah.
[00:37:37] Howard Marks: And, and so, you know, you see that when x86 processor, you know, when Mahalum came along and the memory bandwidth and the number of PCI e-lanes on processors got big enough. All of a sudden we stopped seeing FPGAs and ASICs in storage systems, we started seeing software defined storage, cause what was available for the designers changed and the NVMe over fabrics has been used by most of the storage vendors for that last mile connection going well, it’s going to be fast and then fiber channel or iSCSI for the user machine to access the storage.
But it hasn’t been as effectively used for the server that is the logical controller to access the media on the back end and the way we use it, we broke the traditional limitation that a drive had to be owned by one or two controllers. Cause I drive, you know, a SAS drive where an NVMe drive has one or two ports.
[00:38:47] Prasanna Malaiyandi: Yea.
[00:38:48] Howard Marks: Well, we connect that NVMe SSD to what we call a fabric module, which is an NVMe over fabrics router. And in fact, in the new box, it’s going to be a pair of Nvidia Bluefield cards and the Bluefield card routes, NVMe over fabrics requests from the ethernet network to the SSDs and routes the responses back. But that’s all it does. We don’t need x86 servers in the enclosure. We can do it on the ARMs and the offloads and the Bluefields.
[00:39:25] Prasanna Malaiyandi: and these are the DPUs, correct?
[00:39:27] Howard Marks: Yes. Yeah. The Bluefield is, is the DPU it’s, um, the Nvidia Mellanox version of that. And so it has an ARM, you know, some ARM cores and NVMe over fabrics and RDMA and other built-in offloads in the chip. And so we leverage that to do the routing of requests from the front end servers, where everything is, all the work gets done to the SSDs and get that clean fast, more cost-effective channel
[00:40:03] W. Curtis Preston: Let me go back in time when you did that, that first presentation that you did to the Storage Field Day folks, did how, how did, how did that go over, um, with, with those folks?
[00:40:16] Howard Marks: It went over, pretty well. There was a little, um, being from Missouri and,
[00:40:23] W. Curtis Preston: Yeah. Cause you weren’t because you were brand new. at that point. Right.
[00:40:26] Howard Marks: were brand, we were brand new. Um, and now we’re going, okay, look, we’ve sold a couple of exabytes of storage. Now at this, you know, we, we, our go to market model’s a little different, we sell software. We arrange for customers to buy the pre-approved hardware at cost. Um, and the
[00:40:51] W. Curtis Preston: okay.
[00:40:52] Howard Marks: are,
[00:40:53] W. Curtis Preston: a little interesting.
[00:40:54] Howard Marks: and the software licenses are transferable. So you license a petabyte of software. And you upgrade the hardware when you feel like you’re want to upgrade the hardware. Cause you want the denser faster one that is always coming, but we’ll write the support contract for 10 years for any appliance from install date. So
[00:41:18] Prasanna Malaiyandi: That’s very different
[00:41:21] Howard Marks: well, a typical vendor, you would buy an appliance, it would come with an oEM software license. They would write five years of support. And in year six they would encourage you very strongly to rebuy.
[00:41:35] Prasanna Malaiyandi: yep.
[00:41:36] Howard Marks: Um, and then when you rebuy, you have to buy another appliance and the software license isn’t transferable. So you have to buy another software license. So with us, you gotta have your VAR know, you go to a VAR, a hundred percent channel you go to a VAR. Your VAR, goes to Avnet, says, I want this hardware for Vast. Now $1.2 million average selling price. One of our sales guys is involved. We’re writing, you know, the high touch sale. It’s not, you know, somebody went on a website someplace. but essentially the VAR, essentially the VAR writes two POs: one to Avnet for the hardware and one to us for the actually he writes one PO to Avnet, Avnet cuts us a PO for the software and, and, and that’s a capacity So if you bought a 675 terabyte, raw enclosure and an appliance, that’s got four servers that provide the front end, which is our usual entry point. You could license a hundred terabytes for a year. Multiples of a hundred terabytes for multiples a year.
[00:43:06] W. Curtis Preston: And so that, I think that addresses the question that I had. Cause I listened to the Chris Evans podcasts that you guys did. Um, and there was this talk of the 10 year thing. And, and again, you know, I’m gonna, I’m gonna just acknowledge that I live in a SaaS world where we preach against, you know, large capacity licensing and capital purchases and all of that stuff. Right. So when I heard 10 year purchase. I was like, what? I gotta, I got to decide now how much I need for 10 years, but that doesn’t sound like what you’re talking about.
[00:43:47] Howard Marks: No, no, no, no. You th you buy the hardware.
[00:43:52] W. Curtis Preston: Right.
[00:43:52] Howard Marks: We will write a support contract and software license. One agreement. For that hardware for up to 10 years from install date at the same rate. So if you want to keep it for 10 years, you keep it for 10 years,
[00:44:13] W. Curtis Preston: And so, and so I could, I could buy a smaller one and then add capacity.
[00:44:19] Howard Marks: yeah. Our NRR is three. Lots of people buy small and add capacity. We had a 300% NRR.
[00:44:37] Prasanna Malaiyandi: I think you meant NRR,
[00:44:39] W. Curtis Preston: Thanks for explaining.
[00:44:41] Howard Marks: Yeah,
[00:44:42] W. Curtis Preston: Yeah. NRR, you said ARR. That’s why you have me confused there for a minute.
[00:44:47] Howard Marks: Yeah,
[00:44:48] W. Curtis Preston: Um, I was like an annual recurring revenue of three, three. Uh,, so you rate, you’re saying? yeah. So you’re saying 300% your customers start out at X and they end up with three X, uh, very regularly. Okay. and and the way your system works, it just grows as they need it to grow.
[00:45:12] Howard Marks: Yeah. And you can do it in the hardware, you know? So if you want to start really small, then you can buy hardware and partially license it
[00:45:21] Prasanna Malaiyandi: oh, interesting.
[00:45:23] Howard Marks: you can buy, you can buy, a 600 terabyte box and a hundred terabytes software license, and the 600 terabyte box you bought at what would be our cost. If we were still selling hardware, right. We negotiate the cost with the intel and key Aksia and those vendors.
[00:45:42] Prasanna Malaiyandi: so you used to sell hardware and then you sort
[00:45:44] Howard Marks: We started off in an appliance model. Um,
[00:45:49] W. Curtis Preston: why would I do that? Is that just like ease of large capital purchase thing?
[00:45:55] Howard Marks: Yeah.
[00:45:55] W. Curtis Preston: Meaning why would I buy a bigger box
[00:45:57] Howard Marks: university, we had a university had this much money in this year’s budget.
[00:46:04] W. Curtis Preston: Oh, okay.
[00:46:05] Howard Marks: We won’t put more than a hundred terabytes on it before the next budget comes around when we renew, we’ll renew it as a 400 terabyte license.
[00:46:15] Prasanna Malaiyandi: and I think this is where at the beginning, you said Howard, that you’re looking at releasing a smaller unit.
[00:46:22] Howard Marks: Yeah. So the new, the new box is one. You,
it uses the ESS F one L the ruler form factor as, as DS. So we can, we have 2215 terabyte SSDs for 3 38 raw bat, 300 usable. Um, and that’s half the physical size, half the capacity, because what we have now, it holds 56 SSDs and two U
[00:46:51] Prasanna Malaiyandi: Gotcha.
[00:46:53] Howard Marks: um, Yeah, the new one is, goes from, you know, the fabric module is those NVMe routers today. Each one has to be a dual Xeon. So we have enough PCIE lanes and the processors don’t do hardly anything. So there’s just like, well, there’s costs there. We don’t need, if the Bluefield
[00:47:13] Prasanna Malaiyandi: Yeah.
[00:47:15] W. Curtis Preston: right. right,
right. So let’s, let’s focus for a little bit on. Uh, I mean, the only reason I have historically been when, when I historically heard the idea of using flash for backup, I’m like, that sounds ridiculous because
for the same for cost reasons, too expensive, um, I’m hearing you that, um, so I would, I would put it this way, that, that in, in this upcoming world, in this current world, um, you know, in a world where we have large nation states invading other nation states and then large ransomware, you know, organizations in those countries, we had this, this was our last, you know, th they’re they’re talking about.
So we’re, we’re talking about being retaliated against because of this other country. It’s crazy. Right? So you have this, you have this, um, this need more than ever before for large recoveries. And I, I do believe strongly that there’s really only one of two ways to be really successful in any sort of ransomware situation.
And, and, and it’s basically about fighting the laws of physics .Either you have to have already restored it. So you already have a hot standby ready to go to switch over to, um, or you’re doing live mount directly from your backup and live mount directly from your backup is only going to happen if you either aren’t, deduplicating, uh, like, like the way Data Domain does, or
[00:49:04] Howard Marks: Right.
[00:49:04] W. Curtis Preston: you have flash as far
[00:49:06] Howard Marks: And, and even if you’re not, even if you’re not, deduplicating when you start talking about big, hard drives the IO density just isn’t there, you know, it’s better
[00:49:19] W. Curtis Preston: Some somewhere between you and Data Domain, I would put Exagrid, right. Because exa grid has that front end. It’s not de duplicated now they’re there. They’re nowhere near the size of you.
[00:49:30] Howard Marks: right, no. And they have some, and they, have, some flash cache. And you know, if you look at, you know, the, the guys who do integrated appliances where the software and the target are one thing, those are typically hybrids. And, and so they’ll do an instant recover for one or two VMs pretty well. Cause there’s enough flash for that. But when you start going, I need, you know, the database server behind my ERP, instant recovered, or I need all 50 of these VMs, instant recovered, then it’s, you know, then you just, then you just don’t have enough flash and you’re going to get hard drive performance,
[00:50:08] W. Curtis Preston: Right. And so what it sounds like you’ve replaced the hard drives with QLC
[00:50:14] Howard Marks: right,
[00:50:15] W. Curtis Preston: Help me because I don’t live in this world QLC from a cost perspective versus regular.
[00:50:22] Howard Marks: it’s, not just QLC. So QLC means quad level cell holds four bits per cell.
[00:50:30] W. Curtis Preston: okay?
[00:50:31] Howard Marks: the more, bits you hold, the closer, the voltage levels that represent the differences are, and the more sensitive the cells become to a few electrons escaping. If you have SLC, it’s like a light switch it’s on or off,
[00:50:50] W. Curtis Preston: Right.
[00:50:51] Howard Marks: it doesn’t matter if a few electrons escape, you can still tell whether it’s on or off. QLC. You got 16 values. The difference between value 13 and value 14 might only be a handful of electrons. So QLC has less endurance. Cause every time you erase it, the insulating layers wear down a little and a few more electrons have opportunities to escape. Um, and it’s slower to write because you have to adjust the voltage level just right to be one of those 16 voltage levels.
And that takes a little bit longer. Now the slower to write, we don’t really care about because we acknowledge the writes while it’s still in the SCM. So as long as we are flushing that data out of the SCM, in bandwidth terms fast enough, Latency is unimportant. And the endurance we specifically do a lot of things in our software to manage endurance.
So we write very large writes so that the SSD doesn’t have to garbage collect internally to accommodate small writes. We erase very large erases so that we delete all of the data in an erase block in the flash so that the SSD doesn’t have to garbage collect internally. And that means not only can we use QLC, but we can use dirt cheap QLC SSDs that don’t have a DRAM buffer in them to protect the QLC from wear. Well, if you have a DRAM buffer, then you can aggregate multiple small writes, but yet, but now if power fails, it’s DRAM, you lose the data. So you need a power fail protection circuit, and you need big capacitors to power, the power fail protection circuit so that you can so that you can dump the DRAM into flash and, right, and it all starts to add up. So the SSDs we buy, the other customers are hyperscalers. They put them in servers. They only need one port they’re writing long tail data. It’s not like they’re overriding this stuff all the time. It’s just too many people are looking at that drunken fat frat boy picture on Facebook for it to be on disk so it’s on flash.
[00:53:20] W. Curtis Preston: A.
[00:53:21] Howard Marks: so, you know, we’re, we’re leveraging all of that to keep so that we can literally use that lowest cost flash. And do the 10 year support because the 10 year support includes if the SSD wears out, we’ll replace it.
[00:53:41] Prasanna Malaiyandi: ’cause normally QLC isn’t rated for that long. I believe. Right. SLC is years?
[00:53:48] Howard Marks: S SLC SLC is the very high endurance flesh, but the typical flash that you see for volume use today is TLC triple level cell. So it’s three bits instead of four bits. So QLC is 30% cheaper to make because it holds more bits per cell. Um, and QLC has substantially less endurance. So when you start looking at enterprise SSDs on newegg. The, the 0.1 drive write per day, SSD is slightly better than the ones we use. And the three drive write per day, SSD, you notice has less capacity because it’s got the same amount of flash. It’s just more over-provisioned so they can wear level across more of it. And the three drive rate per day, SSD probably has a DRAM cache and all this stuff to protect it. Um, and that’s what most enterprise storage systems need because how they put the data in the drive dates back to when it was a disk drive. And you were trying to keep data logically adjacent, not try and manage the write pool inside the drive.
[00:55:07] Prasanna Malaiyandi: yeah,
[00:55:07] Howard Marks: The requirements were different.
[00:55:09] W. Curtis Preston: Yeah. Interesting. Yeah. So again, going back to. the fact that you built this from the scratch, with that, with that toolbox from 2016, and you were like, we need to, we need to manage write leveling,
[00:55:21] Howard Marks: And look, our founder Renen Hallak was the chief engineer at Extreme iO. And when he got tired of working for Michael Dell, he got to talk to Extreme IO customers and find out what they wanted. And nobody said we want faster, Extreme IO was already all flash. They were still adjusting to all flash. Um, and it was plenty fast, but everybody wanted to be able to use that all flash for more things. And so our whole system is designed to provide very high, random read performance, across large amounts of flash at an affordable price.
[00:56:04] W. Curtis Preston: Got it.
[00:56:06] Howard Marks: And so our, you know, our performance asymmetry is exactly the opposite of data domains.
[00:56:14] W. Curtis Preston: wait, explain what you just said.
[00:56:17] Howard Marks: Our performance asymmetry is exactly the opposite of data domains. They don’t publish restore speeds anymore. Haven’t for years we publish, read speeds and writes speeds and reads are at eight times faster than rights.
[00:56:34] Prasanna Malaiyandi: That doesn’t mean your rights are slow either. Just for
[00:56:37] Howard Marks: No, no, no, no, no, Our, our smallest system does five gigabytes per second of rights. Yeah. Or, you know, your, your story system probably doesn’t keep up with that, but that’s the SLOs. But what that means is if you scale a system the traditional way, and you say, I need to move this many terabytes over this many hours, so you have to scale it by right performance. Your backups are going to be much faster than your restores. Excuse me. your restores are much faster than your backups.
[00:57:14] Prasanna Malaiyandi: Yeah,
[00:57:15] Howard Marks: Yeah, we, we read much faster than we write. And so if you size for backups speed, you’re a store. Speed’s going to be
[00:57:22] W. Curtis Preston: yeah.
[00:57:23] Howard Marks: nice.
[00:57:27] W. Curtis Preston: All right. Well, you know, consider me impressed, Howard.
[00:57:31] Prasanna Malaiyandi: that’s hard to do by the way
[00:57:33] W. Curtis Preston: I,
[00:57:34] Howard Marks: I I’ve known Curtis a long time. I’ve impressed him once. So this is makes twice. I’m really, I’m happy with that,
[00:57:42] W. Curtis Preston: yeah, so, I mean, it sounds like you’re, you’re clearly, you know, you you’ve been in the business a long time. You’ve seen those companies that have really interesting technology and nobody’s buying anything. You’re not that you, you clearly have the really interesting technology, but you’re also actually selling it,
[00:58:00] Howard Marks: I decided it was time to get a job. And I talked to the folks at Vast, who were still in stealth. And I said to myself, look, Howard, you’re a storyteller. And this is a really good story. And it doesn’t matter whether it succeeds or not. You’re going to have a good story to tell. And low and behold, it’s one of those cases where it was a good story and the market requirement fit. And
[00:58:31] W. Curtis Preston: You don’t have to create the need.
[00:58:34] Howard Marks: we are selling, you know, we have, for the past couple of years done comparisons, all the storage companies have gone public you. Yeah. We’re growing faster than all of them put together
[00:58:48] W. Curtis Preston: all right, well, Howard, um, thanks a lot for coming on. We might have to have you back. Cause I, I know that I know we’ve, we’ve really just begun to scratch the surface and, um, but uh, sounds like, sounds like you got a good gig over there. I’m glad. Uh, both of us could be employed.
[00:59:07] Howard Marks: Ed for the people have known us a long time. It really must be shocking to, you know, you and I both the same job multiple years, but, uh, I’m still having fun at Vast. And, you know, there’s, there’s lots of interesting stuff still to come. Uh, having taken a fresh eye to the market. We got all sorts of good stuff coming.
[00:59:35] W. Curtis Preston: Cool. All right. Well, I wish you the best. And, uh, thanks Prasanna. Uh, this is one of those cases where your background was very helpful. I think,
[00:59:45] Prasanna Malaiyandi: Oh, I try. I try,
[00:59:48] W. Curtis Preston: you know what I mean? You know what I mean? Your, your background in more, obviously more than me in storage systems. Uh, so,
[00:59:55] Prasanna Malaiyandi: yeah. Having spent a bunch of time building storage arrays. It helps, but
[01:00:02] Howard Marks: much fun as people think it is
[01:00:03] Prasanna Malaiyandi: no, it’s still interesting problems though, and, yeah. Thank you, Howard, for sharing some of the details and indulging in my questions. So.
[01:00:11] Howard Marks: Oh, no problem.
[01:00:12] W. Curtis Preston: All right. And, uh, thanks to our listeners. You’re why we do this every week and make sure to subscribe so that you can restore it all.