Public Health Expert Explains COVID-19 Vaccine News (Restore it All Podcast #71)

Lindsey Schulz MD/MPH joins once more on the podcast to discuss the great news we have had in the last few weeks around the Coronavirus (COVID-19). There are THREE vaccines that have been announced that all exceed the goals the medical community set, and will all be applying for emergency use authorization from the FDA. She explains the pros and cons of each of the three vaccinees we know about at this point, as well as giving a little info about another vaccine that is expected to announce soon. This is great news!

----- Signature and Disclaimer -----

Written by W. Curtis Preston (@wcpreston). For those of you unfamiliar with my work, I've specialized in backup & recovery since 1993. I've written the O'Reilly books on backup and have worked with a number of native and commercial tools. I am now Chief Technical Evangelist at Druva, the leading provider of cloud-based data protection and data management tools for endpoints, infrastructure, and cloud applications. These posts reflect my own opinion and are not necessarily the opinion of my employer.

Doctor & Public Health expert explains current state of the pandemic (Restore it All Podcast #70)

Lindsey Schulz MD/MPH visits us again to give us an update on the current state of COVID-19. There is good and bad news, as we are in the midst of the third wave of cases and deaths. But the good news is that health care professionals have learned a lot about the virus in the last several months that have helped to reduce the death rate somewhat, and improve quality of life for those suffering from the illness. We discuss the logic of closing schools, restaurants, and bars, and Lindsey’s interesting thoughts on that, which are actually a bit different than what I hear in the news.  We also discuss case positivity rate, and how it doesn’t mean what you think it means.  Next week’s episode will focus on the vaccines that we now know about, but this week we will just talk about the current state of the disease itself.

----- Signature and Disclaimer -----

Written by W. Curtis Preston (@wcpreston). For those of you unfamiliar with my work, I've specialized in backup & recovery since 1993. I've written the O'Reilly books on backup and have worked with a number of native and commercial tools. I am now Chief Technical Evangelist at Druva, the leading provider of cloud-based data protection and data management tools for endpoints, infrastructure, and cloud applications. These posts reflect my own opinion and are not necessarily the opinion of my employer.

Election poll site manager explains US election systems (Restore it All Podcast #69)

In a departure from our normal coverage, I decided to bring on Mark Thompson, who was the Site Manager for the election polling site I volunteered at last week. He gives us insight into how elections are managed in the US, and what we do to prevent fraud and ensure the overall integrity of the system. We talk about the technology used on the front end of the polling process, which in San Diego county including Electronic Poll Books (EPBs) and Ballot Marking Devices (BMDs). We then also talk briefly about the checks and balances in the actual counting process. Although this is a departure from our normal fare, I think a lot of people will benefit from the info we discuss.

Also, I also posted a blog post about this topic here: http://www.backupcentral.com/why-its-really-hard-to-rig-a-national-election/

----- Signature and Disclaimer -----

Written by W. Curtis Preston (@wcpreston). For those of you unfamiliar with my work, I've specialized in backup & recovery since 1993. I've written the O'Reilly books on backup and have worked with a number of native and commercial tools. I am now Chief Technical Evangelist at Druva, the leading provider of cloud-based data protection and data management tools for endpoints, infrastructure, and cloud applications. These posts reflect my own opinion and are not necessarily the opinion of my employer.

Why it’s really hard to rig a national election

I recently gained some insight into the US election process has given me much more confidence in said process, and I’d like to pass it on. I  gained this insight by working as a poll worker for both the California primary in March as well as the latest presidential election.

I know this has nothing to do with backups, but I felt strong enough about it that I wanted to publish it somewhere.

My base assertion is this: I simply don’t understand how anyone with knowledge of how our election systems work can think that it could be rigged in such a way as to steal a congressional election – let alone a presidential one.   Let me explain.

The first thing to understand is that the US election is not run by the federal government; it is run by each state.  This significantly complicates any efforts to compromise a national process that is completely decentralized. In order to rig a congressional or presidential election towards a particular candidate, you would need to do one of three things: 

  • Insert hundreds of thousands of illegal ballots into the process
  • Have thousands of people vote many times
  • Rig voting machines themselves so that they do not record what the voter said to do
  • Rig the machines that count the votes

Each of these is near impossible and has never been done in the history of the US as far as anyone has been able to find.

How people vote

According to ballotpedia, there are three ways that voters in various states actually record their vote. Hand-marked paper ballots is the most common, followed by what are called Ballot Marking Devices (BMDs), which are electronic machines that produce a paper ballot that the voter can verify matches their intention before handing in.  A minority of states use Direct recording electronic (DRE) systems, which record the voter’s vote on a hard drive.  Most states that use DREs also use a Voter-Verified Paper Audit Trail (VVPAT), which creates a paper log that the voter can verify and that can be used to verify the count at the end of voting.  Nine states (IN, KS, KY, LA, MS, NJ, OK, TN, & TX) do not use a VVPAT. 

There is no way to hack a paper ballot, a BMD that produces a paper ballot, or a DRE that produces a paper log verified by the voter.  As long as the voter can see what they voted for in paper, there’s no way to change that after the fact.  I feel less confident about DREs that do not support VVPAT (mainly because I don’t know anything about them), but it is important to note that none of the states this election hinged on use such machines.  (Texas does use them, but no one is disputing its result.)  PA used to use such machines, but PA’s Secretary of State ordered all such machines replaced in 2018.

I am therefore very confident that the votes that are being delivered in the vast majority of states are exactly what the voters intended them to be. I have no information to question the nine states that still use DREs w/o VVPAT, but I just wanted to point out that none were battleground states in this election.

Each state’s ballot is unique

Now let’s talk about the idea of “magically appearing ballots,” or “massive fake ballots via mail,” or anything like that.  The idea of secretly printing illegal ballots and sending them in or slipping them into the process would be extremely hard for multiple reasons, starting with the fact that the format of each local ballot is unique. This is not something that is going to be able to be done by a foreign entity.

Some think this is something that might be done by a domestic entity, though, so let’s suppose (for the sake of argument) that a local person was able to get a hold of the format of the upcoming ballot so that they could create illegal ballots. They wouldn’t be able to use them for the rest of the reasons I’m about to explain.

Every ballot has a unique ID

Every ballot that is sent out to polling places or via mail has a unique identifier. That identifier is a combination of numbers that includes the precinct to which it was sent, the polling location to which it was sent (or the mailing address of the voter), and a unique ballot number. So every single ballot in circulation can be uniquely identified by an actual number. (For the techies out there, I’ll say it’s something similar to a MAC address, which uniquely identifies every piece of network equipment.)

This means that if someone was able to print an authentic-looking ballot (as mentioned in the previous section) and include it via some nefarious process, it would not have a valid unique ID. It would either have an ID outside of the acceptable range, or one that wouldn’t match with the numbers for that polling place, or it would have one that would conflict with an actual voter. A ballot with an ID that was outside of the acceptable range would obviously be rejected.  And if it conflicted with an actual voter, it would obviously not be counted, either.

All ballots are tracked

(I can technically only speak to how California does things, but the news I have watched over the last few weeks shows that our process regarding the handling of paper ballots is very similar to other states that have them.) Every ballot is tracked from the moment it leaves the election office to the moment it comes back, so the election office knows the location of every ballot by this unique ID. They know the ID of each ballot that was sent to every registered voter in California, and the ID of each ballot that was sent in each carton of ballots that were sent to each polling location.

California wanted to mail a ballot to every active registered voter (due to COVID), but also knew that many would want to vote in person. If you went to a polling location and told them who you were, the system would tell the poll worker you were sent a ballot in the mail.  If you said you did not want to use that ballot, the poll worker tells the system you are surrendering your ballot, which immediately deactivates that ballot in the system via its unique ID.  Whether you physically turned it in, shredded it, or hung it on your wall, it is now an invalid ballot.  If you tried to use it to vote twice (after voting in person), the ballot would be rejected.  (This is another reason why this idea that someone will print out millions of fake mail-in ballots and send them in makes no sense to anyone who understands the process.)

Now let’s talk about in-person ballots. Before a polling location opens, the poll workers take inventory of the blank ballots they received and take note of the range of numbers on all of the pads of ballots they are given. (Blank ballots are in a stack of several stapled pads with a perforated line that allows you to tear off the ballot and hand it to the voter. A copy of the ID of the torn-off ballot remains as part of the pad.) The election office knows how many ballots each polling location was sent and the IDs of the ballots that were sent to them. The polling location verifies all of that before starting anything.

At the end of the voting day, the polling location must count the number of used ballots, unused ballots, and spoiled ballots. (A spoiled ballot is when a voter makes a mistake and requests a new ballot. The poll worker writes the word “spoiled” across the ballot and rips it.) The total of all of those ballots should equal the number of ballots the polling location was given before they started. They have to turn in documentation back to the election office that includes these numbers. All ballots are escorted by at least two people during transit and are then given to a ballot aggregator who then takes all of the ballots to be counted.

When the election office counts the ballots for each polling location, they know how many ballots were sent to them and they know how many ballots were sent back. You cannot insert extra ballots at that point in the process and add them to that precinct’s numbers, because the numbers will not add up, nor will the ballots have IDs in the range of those that were sent to that polling location.

There are way too many polling places.

If you take the number of voters in this previous election (~150 million) and divide them by the number of polling places in this election (~100K), you come out with an average of 1500 voters per polling location. It’s even smaller in Philadelphia, where there were 718 polling locations for ~700K voters that voted, or ~1000 voters per polling location.  Here’s a map of Philadelphia’s polling locations.

Philly's polling locations

If you were somehow able to subvert all of the controls I mentioned above for one polling location, you could affect only 1000-1500 votes. Inserting 10,000 votes for your favorite candidate by grabbing a bunch of unused ballots with the appropriate numbers and changing their vote would set off many bells and whistles and immediately invalidate that precinct’s vote.

Each of these polling places is like its own election, because each polling place is tracked individually. This is what I was referring to previously when I talked about how decentralized the process is.  You’d have to hack hundreds of tiny elections to affect one big election.  

There are eyes everywhere

Our polling location had 15 workers who all wanted to do the right thing, which was to ensure that every registered voter was given a chance to vote and have their vote counted. These poll workers didn’t know each other and were assigned by the election office. They were a mixture of Democrats and Republicans, and we all just wanted to do the right thing – help everyone cast their vote without interference or intimidation. The idea that one of us would somehow do something to damage that was foreign to all of us – regardless of which way we leaned politically.

There were also multiple polling observers that visited us throughout the process that were also looking for anything going wrong. Sometimes they would see things that looked weird to them and ask a question. “Why is that person doing that thing with that item?” We would answer the question and that would be the end of it. (There was one polling place where a poll worker was seen throwing an information sheet in the trash and a polling observer thought they were throwing away a ballot.) I guess what I’m saying here is that there were way too many people watching the process for you to be able to do something nefarious and get away with it.

What about rigging the counting machines?

Is it possible to take valid ballots and have them counted in an invalid way? There are so many processes in place to ensure that’s not possible that I’m going to have to say a big NO to this one as well. There are all kinds of tests that are run on machines to validate that they count the ballots as they are meant to be counted. These tests are run throughout the process to verify that the machines are doing the job that they are supposed to do. While as a technology person, I understand the concept of inserting malware into such a machine, such malware would be easily caught before and after the fact.

This is also why all but nine states use a printed ballot or paper log that can be hand-counted if there is any doubt as to the election result. You don’t have to count every ballot to do this; you just need to spot count different boxes of ballots. For example, the election office knows that 415 ballots came from the polling location where I worked. It’s not that hard to hand count 415 ballots. You do that and then run them through the machine and make sure that the count of the machine matches the hand count. You do that every so often and you validate the machines.  (I learned today that San Diego actually hand counts 1% of all ballots and checks them against the result from the machine.) 

What about getting many people to vote illegally?

This is easily the most unlikely scenario, as there are many problems with this idea, even in a state like California that does not require voter ID. The first problem with this idea is that it requires absolute secrecy from many thousands of people in order to execute enough votes to actually throw an election. In this current election, it would have required millions of illegal votes done this way to have changed the election result to the current status.

Have you ever tried to throw a surprise party? How many times has the secret been spoiled by one person saying the wrong thing? I can tell you that the surprise party we threw for my granddaughter two months ago was spoiled by two people – and we only needed 10 people to keep the secret. How do you get hundreds of thousands of people to commit a federal crime without a single one of them gaining a conscience and reporting what’s going on? The answer is you don’t.

So let’s say you get over this (in my opinion) insurmountable hurdle, and you get tens of thousands of people to vote multiple times. Even in a state without voter ID, that requires you to actually impersonate multiple people. This is because, in addition to tracking each ballot, every voter is tracked. Once a given voter has voted – via a mail-in ballot or an in-person ballot – an additional vote will simply not be counted. If you’re going to impersonate, say, a dead person or a neighbor that you know will not vote, you need to know all of their information in order to impersonate them. You then need to be able to sign their name in a convincing way as well, as those signatures are cross-checked. 

Again, let’s suppose for the sake of argument that you successfully get tens of thousands of people to have the appropriate information and the appropriate signature to do this. Fraud sufficient enough to throw the results of an election would easily be discovered after the fact – and there simply hasn’t been any evidence of this. 

The most potentially damning evidence I saw was an accusation that 3000 people who don’t live in Arizona voted in Arizona. First, I would say this shows how easy it is to find things that look odd. Second, I will say that upon further examination these people were shown to be military people or students living out of state. If actual fraud of this type happened somewhere, it will be really easy to spot.

And not only is there no evidence of fraud to this degree in this election.  There’s no evidence of fraud to that degree in any US election in history.  Many studies have consistently found this to be true.  Here’s a report of a few of them

The down-ballot races

Finally, I just have to say this.  Why in the world would the Democrats go to the trouble of rigging an election to take out Donald Trump, but not take control of the Senate? Every Democrat I know wants Mitch McConnell and Lindsay Graham gone just as much as they want Donald Trump out of the White House. The idea that they would rig the presidential election but not the Senate election simply makes no sense.

I just don’t see it

All of the things above is why I simply don’t see how you could ever rig an election for congress or president in the US.  And I have seen nothing in the news for this particular election that would change that idea.  You are welcome to leave a comment below if you have facts to contradict what I’ve said here. 

----- Signature and Disclaimer -----

Written by W. Curtis Preston (@wcpreston). For those of you unfamiliar with my work, I've specialized in backup & recovery since 1993. I've written the O'Reilly books on backup and have worked with a number of native and commercial tools. I am now Chief Technical Evangelist at Druva, the leading provider of cloud-based data protection and data management tools for endpoints, infrastructure, and cloud applications. These posts reflect my own opinion and are not necessarily the opinion of my employer.

Dissecting two ransomware attacks on hospitals (Restore it All Podcast #68)

Prasanna and Curtis talk about two recent ransomware attacks on hospitals and what we can learn from them. They also discuss things you can do to protect yourself from such attacks, and how to prepare to respond if you get one. We especially talk about the 3-2-1 rule and the remote desktop protocol (RDP) and how these figure into protecting yourself from such things.

----- Signature and Disclaimer -----

Written by W. Curtis Preston (@wcpreston). For those of you unfamiliar with my work, I've specialized in backup & recovery since 1993. I've written the O'Reilly books on backup and have worked with a number of native and commercial tools. I am now Chief Technical Evangelist at Druva, the leading provider of cloud-based data protection and data management tools for endpoints, infrastructure, and cloud applications. These posts reflect my own opinion and are not necessarily the opinion of my employer.

Tape may be cheap, but disk is better for backups (Restore it All Podcast #67)

I do apologize for the delay in posting this.  I was a poll worker in San Diego for the last six days!

This podcast isn’t a rebuttal to last week’s podcast, but it might seem that way. Last week we talked about the advantages of tape for very long-term retention (e.g. 10 years), one of which is a significant cost advantage. This week we will discuss how backup, recovery, and disaster recovery are very different use cases, and why disk and cloud is a much more appropriate place for that use case. Joining us to discuss this topic is Druva’s CTO, Stephen Manley, who has spent many years at companies that use disk for this purpose.

----- Signature and Disclaimer -----

Written by W. Curtis Preston (@wcpreston). For those of you unfamiliar with my work, I've specialized in backup & recovery since 1993. I've written the O'Reilly books on backup and have worked with a number of native and commercial tools. I am now Chief Technical Evangelist at Druva, the leading provider of cloud-based data protection and data management tools for endpoints, infrastructure, and cloud applications. These posts reflect my own opinion and are not necessarily the opinion of my employer.

Why Tape is Cheaper than Glacier for Long Term Storage (Restore it All Podcast #67)

Matt Starr, CTO of Spectra, comes on the podcast to discuss the advantages of tape for long term storage. We talk about how tape is actually better at holding data long term than disk is – 10,000 times better if you compare it to SATA disk. We also talk about the advancements in tape in the last 10-20 years that have made libraries like Spectra’s even more reliable than they used to be. Finally, we talk about the Spectra T-Finity library that can now hold an Exabyte of data in a single unit! What started this whole idea of bringing Matt on was Spectra’s eBook that said that the T-Finity tape library was significantly cheaper than Glacier Deep Archive if you store your data for a long period of time (e.g. 10 years or more). Here’s the eBook that got the conversation started: https://bit.ly/37BtTkK

----- Signature and Disclaimer -----

Written by W. Curtis Preston (@wcpreston). For those of you unfamiliar with my work, I've specialized in backup & recovery since 1993. I've written the O'Reilly books on backup and have worked with a number of native and commercial tools. I am now Chief Technical Evangelist at Druva, the leading provider of cloud-based data protection and data management tools for endpoints, infrastructure, and cloud applications. These posts reflect my own opinion and are not necessarily the opinion of my employer.

Why is it so hard to backup consumer SaaS products? (Restore it All Podcast #66)

Daniel Rosehill, a self-described “backup anorak,” joins us to discuss how difficult it can be to backup consumer SaaS services, such as Evernote. (An anorak is a slightly prejorative term that refers to someone who is interested in a not-so-leading idea – like backups.) Daniel used to use EverNote on Linux, and sent a message to their support system on how to backup its data. They had no answer unless you were using it on Windows. We talk about Google Drive, Dropbox, and other consumer-grade cloud services, and how the challenges of backing them up should be a concern for any users of these platforms – not just backup anoraks.

----- Signature and Disclaimer -----

Written by W. Curtis Preston (@wcpreston). For those of you unfamiliar with my work, I've specialized in backup & recovery since 1993. I've written the O'Reilly books on backup and have worked with a number of native and commercial tools. I am now Chief Technical Evangelist at Druva, the leading provider of cloud-based data protection and data management tools for endpoints, infrastructure, and cloud applications. These posts reflect my own opinion and are not necessarily the opinion of my employer.

What it takes to make a successful SaaS product

What makes a successful data protection as a service (DPaaS) product? I watched another vendor announce a “SaaS” product that I suppose technically meets the definition – but looks like it will fall short in both metrics of success: customer satisfaction and overall revenue. It’s not going to offer an actual SaaS experience, and it’s going to compete with the company’s other products and therefore not be as successful as it could have been.

Fake SaaS is Vegan Sausage

I mean no disrespect to my vegan or vegetarian friends. I actually was a vegetarian for a time and I respect the choice.  But when I was a vegetarian, I was happiest when eating vegetables cooked as vegetables.  I did not need my vegetables to taste like meat.  There is nothing wrong with vegan sausage per se. If you eat it and like it, more power to you.  And if you’ve never had actual sausage, you might think you are eating something that tastes like sausage.  Trust me, you’re not.

That’s my point with these SaaS-like products that are popping up – they say they’re SaaS, but are they?  It starts with an on premises offering that doesn’t have a recurring revenue model outside of their support contracts.   They could refactor their existing solution to be cloud-friendly and truly SaaS, but that is a very expensive proposition for a company that still has an existing product to maintain. So they lift-and-shift the on premises solution into VMs in the cloud, put a SaaS interface in front of it and call it a day.

It’s vegan sausage.  It may look and feel like SaaS, but anyone who has truly experienced SaaS will tell you not so fast.  SaaS is more than a user interface. It’s a way of doing things that includes pay-as-you-go pricing, dynamic allocation of resources, and not having to ever worry about paying for something and not using it. It’s also about being less expensive than the alternatives. If you lift-and-shift and don’t refactor for the cloud, you’re going to have something that looks like SaaS, but will fall short in every one of those areas when compared with actual SaaS products.

Competing with your own products

I spoke today with three fellow industry people, each of which worked on cloud versions of three different hardware products.  Three different companies and three different products – and yet one story.  The products all started as lift-and-shift versions of an existing product. Salespeople have a bigger incentive to sell the on premises version of the product. The sales conversation uses the cloud version as a bargaining chip, but the ultimate goal is to sell the on premises version.  The result is no one ends up selling the cloud version of the product.  The ROI is smaller upfront and bigger over the long term.  But salespeople think short term – and they’re incentivized to do so.

The result in each story was that none of the three cloud versions of the three storage products were successful.  Salespeople killed it because it didn’t pay them as much. You could address this problem with sales incentives, but then you kill the bread and butter of the company for the new, experimental product.  Public companies – or companies trying to be public companies – can’t tolerate the dip in revenue this would create, and so they end up cannibalizing their own product. 

Judge for yourself

If you’re looking for a DPaaS product, ask questions to help you identify if the product is a true SaaS product or not. What does scaling the solution look like, both up and down?  Is it possible to pay for something you don’t use?  The product I saw today charged for front-end terabytes, meaning the size of the datacenter being backed up.  What happens if I buy a 100 TB license, but only deploy it 10 TB at a time?  I won’t actually be 100 TB until later in the year.  Does that create credits that rollover, or did I just buy 100 TB for a year?  What happens if we buy too much, such as what happens if you kill off a major project or sell off a portion of your company? What happens as I scale?  Do I need to warn the vendor so they can scale the back end for me, or does it scale automatically?

In the end, the only thing that really matters is what you get for your money. Fake SaaS solutions will cost more for less functionality. So make sure to take a look at the overall TCO of the solution before you buy it. 

I’ve had real SaaS, and I’ve now seen how it’s made.  (Sticking with my analogy to the bigger end.) Trust me, it’s better than fake SaaS any day, and anyone whose actually experienced a real SaaS product will agree.  Don’t settle for less.

----- Signature and Disclaimer -----

Written by W. Curtis Preston (@wcpreston). For those of you unfamiliar with my work, I've specialized in backup & recovery since 1993. I've written the O'Reilly books on backup and have worked with a number of native and commercial tools. I am now Chief Technical Evangelist at Druva, the leading provider of cloud-based data protection and data management tools for endpoints, infrastructure, and cloud applications. These posts reflect my own opinion and are not necessarily the opinion of my employer.

The Dangers of Improperly Secured Cloud Accounts (Restore it All Podcast #66)

The Palo Alto Networks’ Unit 42 threat hunting team found that a big customer of theirs had misconfigured two critical Amazon Web Services (AWS) services. If these misconfigurations were exploited by hackers, it could have created a data breach that could have cost the customers tens of millions of dollars. Prasanna Malaiyandi and W. Curtis Preston (Mr. Backup) discuss this misconfiguration, and what you can learn from it to protect backups you store in the cloud.

----- Signature and Disclaimer -----

Written by W. Curtis Preston (@wcpreston). For those of you unfamiliar with my work, I've specialized in backup & recovery since 1993. I've written the O'Reilly books on backup and have worked with a number of native and commercial tools. I am now Chief Technical Evangelist at Druva, the leading provider of cloud-based data protection and data management tools for endpoints, infrastructure, and cloud applications. These posts reflect my own opinion and are not necessarily the opinion of my employer.