I learned a lot from The Tiger King, currently the most popular show on Netflix. And like WestWorld, I saw a few lessons about backup and disaster recovery that I thought I’d share with you.
Spoiler Alert: If you haven’t seen the entire docuseries The Tiger King, stop reading this right now, as it contains some major spoilers from that series. How have you not watched this yet? It is the most incredible thing I’ve ever seen on Netflix, as in non-credible. As in, “I can’t believe what I’m watching!” (And yet it’s all just a little too real, and sadly there are deaths involved in the show. Just go watch it.)
Offsite backups matter
I can’t believe a guy responsible for producing a fully professional television series didn’t understand the importance of getting copies of his footage to multiple locations. I know it’s hard to get raw footage to two locations, as it is so large, but I also know that technology exists to help make it easier.
And look what happened. All that work literally went up in smoke. And that footage was gold, I tell you! Thankfully there was a documentary being filmed at the same time, so we go some idea of what that reality show would have been like. But that producer was sitting on a gold mine, and he just kept it under his mattress. If you learn nothing else from this show, it’s this.
Rogue admins exist
I get a lot of heat from some people online that I see rogue admins everywhere. Well, I don’t, but you have to prepare for them. Statistics show that most attacks from the inside, and this “fire” was no exception.
We all know that he did it, right? I mean, it was clearly arson. An arson that benefited him by erasing all the footage that could have been used to incriminate him. Also, it happened days after he suggested that it should happen. I mean… come on! It was an inside job!
This is why separation of powers is important in a good backup and DR system. No one person should be able to wipe out all your data. It’s also why we backup SaaS services like Office 365 and Salesforce. If you didn’t, a rogue admin could easily delete it all along with any snapshots of your data.
Explosions kill more than computers
This isn’t so much about backups, but as a man who was born and raised in Florida, I feel a connection with alligators. They’re nowhere near as scary as they look, and they’re more scared of you than you are of them – believe it or not. It was indeed sad to see the loss of so many alligators just to stop that production.
This also isn’t about backups. What I learned from her is that the way to protest private zoos is to have your own private zoo. Oh, I’m sorry… “refuge.” One where the biggest animals in the zoo are held in cages that aren’t even big enough for them to turn around in? Come on, Carole Baskins! How is PETA on your side?
Like a lot of companies, some of my employer’s plans for the year have been put on hold. For example, we had planned to do a video series where I fly up to Druva’s Sunnyvale headquarters and film a video discussing backups and related things with Stephen Manley. We would broadcast that video on LinkedIn Live. Given what’s going on right now with the Coronavirus, that’s not happening.
The big idea
Last week we asked the question, “what if we did it remote?” Would it be possible to use Zoom (which we already use) to create a video stream that we could somehow pass to LinkedIn Live? Many Google searches later, I felt the answer was a very tentative, “maybe.” Zoom does have an optional feature that can create a video stream from a meeting. The question is, can we send that stream to LinkedIn?
LinkedIn said “no,” but they gave us a list of vendors to try. I proceeded to contact almost all of them. Many simply did not respond in a timely manner. Others responded to the Zoom part of the question with answers that showed they were not the solution I needed.
A voice in the darkness
But one company, Wowza (a Streaming-as-a-Service platform), appeared to have web pages with instructions for both halves of this equation (streaming from Zoom and streaming to LinkedIn Live). The only problem was the LinkedIn part was in Beta, since it’s quite new. I signed up for a trial account and the wrote to them asking to enable the beta feature on my account.
Robert Vacante of Wowza wrote me and said he had enabled the feature and gave me links to the two pages I mentioned above. (I would later find out he is actually the Product Manager!) To make a long story short, within a matter of hours I was able to go from complete despair to being ready to test my first broadcast on LinkedIn Live. Then we did our first LinkedIn official Live broadcast the next morning!
SaaS wins the day
I didn’t have to download and configure anything. I didn’t have to buy a piece of hardware (as some of the solutions required). I just had to authenticate them with Zoom and LinkedIn, configure a few things, and we were off and running. Just like when my employer (Druva) backs up Office 365, G-Suite, and Salesforce, the whole process was managed in the cloud, and all the infrastructure needed to make it happen just magically appeared when I needed it.
Go back in time to before SaaS and this never would have happened. Imagine identifying an IT challenge, researching products for a few hours, then successfully deploying that product in your datacenter in a matter of hours. That simply wasn’t possible before the SaaS concept really took off. Maybe Druva’s onto something.
I want to give a big shout-out to the folks at Wowza, especially Robert Vacante who answered a few questions for me during my time crunch. I couldn’t have done it without them.
How did I do it?
Robert Vacante told me he thinks we may have been the first customer to connect both their Zoom streaming option to their LinkedIn Live option. I will say that, while the instructions were good, they were not streamlined for this particular workflow. This is especially true of the page on how to configure the LinkedIn Live option, as it is really built on our original plan of high-quality video from multiple “normal” sources. If you’re just going to use Zoom, things are actually much simpler. The rest of this blog post is the instructions I came up with for how to configure the system, and how to run it once it’s configured.
Request LinkedIn Live access
You need to be approved for LinkedIn Live. This is no small feat, as it took us four months to get approved.
On the Video Source and Transcoder Settings page, select OTHER RTMP.
Put a checkmark in the box next to Disable Source Security.
Leave all other settings on that page the same and click Next
Leave All settings on the page the same and click Next
On the Hosted Page Settings, give the hosted page a name, leave the other settings at their defaults. (e.g. “LinkedIn Live Test”) Click Next.
Setup the LinkedIn Live Target
Login to cloud.wowza.com
Go to Advanced > Transcoders > Click on the transcoder whose name starts with the name of the stream you just created. (e.g. “LinkedIn Live Test/Transcoder”)
Select Outputs and Targets Tab
Click Add Output
Leave the dropdown at Video + Audio
Click Video Passthrough
Click Audio Passthrough
Find the new Output in the list with the phrase “There are no stream targets assigned to this output.”
Click Add a Stream Target
Select LinkedIn Live from the list
Select whether you want to stream to a person or organization & click Next
Authenticate with LinkedIn and give permission for the app to interface with it. (The account you login to needs to match to an account that is authorized to use LinkedIn Live.)
Enter a Title for the LinkedIn Live event you will put in. What you put in the title and description here will go into the LinkedIn Live page when you go live.
Get ready to go live
This procedure assumes you called your stream config “LinkedIn Live” in Wowza.
Long before broadcasting
Your Zoom admin needs to enable live streaming in Zoom. They only need to enable live streaming via a “custom live streaming service.” You do not need to enable youtube, facebook, Workplace by Facebook etc.
Have a test zoom meeting where you test everyone’s setup, lighting, background, etc. Make sure the audio and video look good.
Use that same exact setup during the broadcast.
Wowza Setup (~30 mins or so prior to the live session)
Login in to cloud.wowza.com
Go to Advanced, Target Streams
Select LinkedIn Live Target
If it has a box that says to refresh the permissions, a LinkedIn admin w/the right permissions has to do that.
Go to Live Streams tab
Select LinkedIn Live Stream
Click Start Stream
Nothing will show up on LinkedIn at this point. Not until you tell Zoom to start the live stream. This is sort of “opening the door.”
Zoom setup (After the wowza setup, ~30 mins or so before the live session time)
Note: Even if you did this before, you have to follow this whole Zoom process each time of creating and saving a new meeting, because Zoom does not save the streaming settings you will be editing in this step.
Click Schedule a Meeting
Click Save (None of the other settings matter.)
Once you click save, there will be a Live Streaming section at the bottom. Click on “configure live streaming settings.” A box will pop up with three fields.
In another tab, go back to wowza.com
Select Live Streams
Select the stream you previously created (e.g. LinkedIn Live stream)
One at a time, copy and paste the following values from this page into the box that popped up on Zoom.
Copy/paste the “Primary Server” value into the “Stream URL” field (e.g. rtmp://2020s.entrypoint.cloud.wowza.com/app-942a)
You only need to copy the part that starts with rtmp: up to where there is a space (do not copy/paste the IP address)
If there are any leading or trailing spaces in the URL after you copy/paste it, delete them.
Copy the “Stream Name” value into the “Stream Key” field (e.g. 56a664d7)
Copy te “Hosted Page URL” value into the “Live Streaming Page URL” field (e.g. https://player.cloud.wowza.com/hosted/bkds7a/player.html)
Again, only copy the URL, without any spaces. If any leading or trailing spaces show up after the copy/paste, delete them.
Click Save on the Zoom popup box.
Note: This still doesn’t start the stream so you can do this early to make sure your audio/video looks good
Click Start Meeting
Click Invite and send the link to whomever else needs to be on the Zoom.
Make sure everyone’s video and audio look good before moving onto the next step.
Note: Do these steps only once you are ready to actually go live on LinkedIn Live.
Have someone other than you login to LinkedIn Live to monitor the stream
Setup a Slack chat (or similar) w/them so you can communicate during the stream
Click on the three dots in Zoom and click Start live Streaming Service
It will open a tab in your browser. If all is well, you will see a progress bar that eventually finishes, after which you will see a tab w/the live stream in it. DO NOT CLOSE THIS TAB. DOING SO WILL STOP THE STREAM. To mute its audio, right click on its tab select “Mute Site”
You are live on LinkedIn Live!
The person monitoring the stream should let you know all is OK
Thank people for coming, and kill time for a few minutes to allow people to get the notification and login.
Do whatever you came to do on video
Stop the stream
Stop talking. Monitor the muted tab to see when the stream shows that you have stopped talking.
Click Stop Stream in Wowza to Stop the Stream
Click the three dots in Zoom and click Stop Live Streaming Service
Enrico Signoretti (@esignoretti) joins us from Italy to discuss how the Coronavirus/Covid-19 quarantine is going. We also get a first-hand report about what it’s like for a company that didn’t have anyone working remotely to suddenly have hundreds doing so.
@wcpreston and @pmalaiyandi discuss Coronavirus and how it’s going to affect the IT community, as well as your backup and recovery system. We discuss remote employees, cloud data protection, and disaster recovery.
Westworld was really just a giant object lesson about proper backup and recovery design. The producers of the show may have felt differently; they seemed to think the show was more about free will or something. I will just say that if the creators behind Westworld (the park) followed proper backup and recovery design, the end of season two would have been very different.
Spoiler alert! Westworld season two spoilers below!
I’m writing this blog post immediately after re-watching season two of Westworld in preparation for season three coming out next week. I’m going to be revealing major plot points of season two in this blog post. You have been warned.
My tongue will definitely be planted firmly in my cheek for some of this post, but there are actually some really important lessons to be learned from what happens in season two of Westworld.
Solid State Storage
Solid-state storage apparently gets much better in the future. No more SSDs or SD cards. Everything will be stored inside a round sphere with no port or obvious electrical connection points like you see with an SD card. Instead, data-transfer appears to happen wirelessly in the same way that we charge wirelessly today. So we’ve got that to look forward to.
We already have water-cooled computers and racks, so this isn’t too far-fetched of an idea. Water cooling can be quite effective as long as it is properly managed, of course. But I don’t think water cooling will take the form that it does in Westworld. Loose water still does quite a bit of damage, as does condensation in a data center – something else that they make sure to display in the show. It looks cool, but that’s about it.
The number one threat to the two data centers in the show (besides poor design) is the hosts themselves. An android driving by AI runs amok and believes that the backups are actually what’s holding them back, because the park operators use backups to restore the hosts when they become damaged, which is what allows the guests to damage them with impunity. Dolores has a point, to be sure. From the perspective of the hosts, these backups are indeed the chain that keeps them enslaved. If there were no backups, the park operators wouldn’t be so quick to blow away their hosts just for fun.
The data protection lesson here is to never forget the danger of an insider threat. Many, if not most, of data center attacks have come from people inside the company. A disgruntled employee wants to harm the company that just fired them. Another employee feels that they are not properly compensated and chooses to solve that by letting ransomware loose in a data center. Make sure to protect against insider threats in your data protection system.
In addition to destroying data, Dolores also escapes the park with a bunch of IP as well. She steals her own copy, as well as what appears to be about five pearls containing copies of other hosts. This is very different from insiders seeking to damage data or encrypt data. This risk is someone (internal or not) stealing your company’s intellectual property. The modern equivalent to what happened in the show is someone creating a copy of important data and then walking out of the company with it. This is another risk that you really need to look into, and it is what data loss prevention software is all about.
The 3-2-1 rule still applies
Those familiar with my podcast will not be surprised to hear me mention the 3-2-1 rule. The designers of Westworld (the park) ignored almost every single aspect of this rule. They did not have different versions of the hosts over time; they only have the most recent copy of each host’s image. This is because they didn’t want the host to accidentally remember previous things. They also did not adhere to the “2” because they really only had one copy, and it was stored in one data center. (Okay, perhaps there were multiple copies within that data center but I don’t think so.) They definitely did not have one of the copies off-site, because if they did the season would’ve had a very different ending.
Hale: One of the hosts just blew up our data center.
Bernard: I’ll get the off-site backups
Hale: This is going to be expensive. Good thing we’re making billions of dollars from all these rich people paying $40,000 a day to come to our park.
The ultimate sin: non-consensual personal data
There was one scene towards the end of season two that really made me laugh. They first acknowledged that many customers had been killed by their product. Then they said that the much bigger PR problem was going to be when people figure out that they’ve been secretly recording everyone’s activities without their consent or knowledge.
Your product went berserk and killed everyone currently using it, and you think nonconsensual recording of personal data is going to be the bigger PR problem? Boy, the show producers really believe in the privacy of personal data, don’t they? So do I, and I don’t want to minimize how horrible it was that they were secretly recording everyone’s behaviors. I just think that once a product is shown to have killed everyone that used it, I don’t think anyone else is going to use the product ever again. But it is interesting that this very modern problem at the center of GDPR, CCPA, and other nascent regulations worked its way into the show.
Looking forward to season three
I’ll just say that I’m a big fan of the show and I’m looking forward to seeing what happens in season three. Now that we realized the man in black is also an android (you did see the scene at the end of the credits, right), how does that change the storyline? (Of course, the men in black is actually an homage to the original man in black played by Yule Brenner, who was also an android.) Since Dolores is Dolores again, whose brain is in Hale’s body? Since they can faithfully reproduce whatever host they want, but other hosts what we see return? What happens to the park now? I’m just full of questions.
You shouldn’t need an annual day to remind you to treat your sweetheart well. Every day should be Valentine’s Day, right? Reaching out and telling your sweetheart that you love them (or remember them if they are no longer with us) is something that you should do on a very regular basis.
Just like backups.
You shouldn’t have to remind yourself to do backups. In fact, backups should require no action on your part; they should just happen. Like love expressions for your sweetheart, they should happen at least on a daily basis – and possibly more often than that. Making sure backups “just happen” requires you to do a few things.
Your backup system should allow you to define a schedule for backups. The default setting for most people is once a night during a time when most people are not using the computing environment. For example, Time Machine defaults to once an hour.
Whatever the frequency, your backup system should be set up so that backups occur on a very regular basis without anyone having to make them happen. We’ll discuss how often that should be a little bit later in this blog post.
Storage available for backups
Backups are not going to happen if they have nowhere to store the data. Historically this meant you had a tape library full of tapes that were ready to be swapped in when necessary. Backup technology has evolved and most of us are using disk or cloud as the primary target. So the main challenge here is to ensure that the disk is available, online, and has sufficient capacity to hold your backups. (This is one of the great things about using the cloud as your backup target; it’s never out of space.)
This is one of the reasons why I do not like Time Machine for your regular laptop backups. It requires you to plug in a portable hard drive in order for the backups to work, then you have to unplug it in order to get your backups away from the thing that you’re backing up. (The 321 rule is always waiting.) So just make sure that whatever backup storage you have, it is always available and always has available capacity for your backups.
How often should you backup?
The more often you backup, the less data you will lose. In more technical terms, the more often you backup, the shorter recovery point objective (RPO) you can support. Let’s consider a few extremes.
If you only backup once a night and your off-site storage system requires swapping tapes, the best RPO you can support is 96 hours. Why is that, you say? Let’s say something bad happens on Monday before the iron Mountain truck comes. If you are sending backups offsite every day, what night was the most recent off-site backup taken? The answer is Thursday night.
Think about it. The last truck to leave your facility left Friday morning, which means it has Thursday night’s backups. That means you’re going to lose all of Friday’s work, any work done over the weekend, and any work that was done on Monday prior to the disaster. That’s 96 hours of lost work. This is why backup frequency and off-site frequency matters. However, backing up more often and sending backups offsite more often can be a costly endeavor, so this must be a decision based on business requirements.
This means this is a business discussion, not a technical one. Stakeholders in your company should decide what the RPO is for their environment because it should be based on the cost of lost data for that particular stakeholder. Business units with very high data loss costs will seek a much tighter RPO, perhaps an hour or even a few minutes. Such an RPO requires backing up more often. And that is the answer to the question, “How often should I back up?”
Give your backup admin some flowers
I’ll bring this back to Valentine’s Day by saying that your backup admin has a very tough job. No one remembers the hundreds of thousands of backups they got right; they only remember the one restore they got wrong. Get them some flowers, chocolate, or whatever it is that gives them a smile. Say thank you, have a nice Valentine’s Day – and then don’t wait another year before you do that again.