Backup & Disaster Recovery Lessons from The Tiger King

I learned a lot from The Tiger King, currently the most popular show on Netflix.  And like WestWorld, I saw a few lessons about backup and disaster recovery that I thought I’d share with you.

Spoiler Alert: If you haven’t seen the entire docuseries The Tiger King, stop reading this right now, as it contains some major spoilers from that series.  How have you not watched this yet?  It is the most incredible thing I’ve ever seen on Netflix, as in non-credible. As in, “I can’t believe what I’m watching!”  (And yet it’s all just a little too real, and sadly there are deaths involved in the show.  Just go watch it.)

Offsite backups matter

I can’t believe a guy responsible for producing a fully professional television series didn’t understand the importance of getting copies of his footage to multiple locations.  I know it’s hard to get raw footage to two locations, as it is so large, but I also know that technology exists to help make it easier.

And look what happened. All that work literally went up in smoke. And that footage was gold, I tell you!  Thankfully there was a documentary being filmed at the same time, so we go some idea of what that reality show would have been like.  But that producer was sitting on a gold mine, and he just kept it under his mattress.  If you learn nothing else from this show, it’s this.

Rogue admins exist

I get a lot of heat from some people online that I see rogue admins everywhere.  Well, I don’t, but you have to prepare for them.  Statistics show that most attacks from the inside, and this “fire” was no exception.

We all know that he did it, right? I mean, it was clearly arson.  An arson that benefited him by erasing all the footage that could have been used to incriminate him.  Also, it happened days after he suggested that it should happen.  I mean… come on!  It was an inside job!

This is why separation of powers is important in a good backup and DR system.  No one person should be able to wipe out all your data.  It’s also why we backup SaaS services like Office 365 and Salesforce. If you didn’t, a rogue admin could easily delete it all along with any snapshots of your data.

Explosions kill more than computers

This isn’t so much about backups, but as a man who was born and raised in Florida, I feel a connection with alligators. They’re nowhere near as scary as they look, and they’re more scared of you than you are of them – believe it or not.  It was indeed sad to see the loss of so many alligators just to stop that production.

Carole Baskins

This also isn’t about backups. What I learned from her is that the way to protest private zoos is to have your own private zoo.  Oh, I’m sorry… “refuge.”  One where the biggest animals in the zoo are held in cages that aren’t even big enough for them to turn around in?  Come on, Carole Baskins!  How is PETA on your side?

 

 

----- Signature and Disclaimer -----

Written by W. Curtis Preston (@wcpreston). For those of you unfamiliar with my work, I've specialized in backup & recovery since 1993. I've written the O'Reilly books on backup and have worked with a number of native and commercial tools. I am now Chief Technical Evangelist at Druva, the leading provider of cloud-based data protection and data management tools for endpoints, infrastructure, and cloud applications. These posts reflect my own opinion and are not necessarily the opinion of my employer.

How to broadcast a Zoom Meeting to LinkedIn Live!

Like a lot of companies, some of my employer’s plans for the year have been put on hold.  For example, we had planned to do a video series where I fly up to Druva’s Sunnyvale headquarters and film a video discussing backups and related things with Stephen Manley.  We would broadcast that video on LinkedIn Live.  Given what’s going on right now with the Coronavirus, that’s not happening.

The big idea

Last week we asked the question, “what if we did it remote?”  Would it be possible to use Zoom (which we already use) to create a video stream that we could somehow pass to LinkedIn Live? Many Google searches later, I felt the answer was a very tentative, “maybe.”  Zoom does have an optional feature that can create a video stream from a meeting.  The question is, can we send that stream to LinkedIn?

LinkedIn said “no,” but they gave us a list of vendors to try.  I proceeded to contact almost all of them.  Many simply did not respond in a timely manner.  Others responded to the Zoom part of the question with answers that showed they were not the solution I needed.

A voice in the darkness

But one company, Wowza (a Streaming-as-a-Service platform), appeared to have web pages with instructions for both halves of this equation (streaming from Zoom and streaming to LinkedIn Live). The only problem was the LinkedIn part was in Beta, since it’s quite new. I signed up for a trial account and the wrote to them asking to enable the beta feature on my account.

Update: The live-streaming feature is only available with the commercial version of Zoom. It is not available in the free version.

Robert Vacante of Wowza wrote me and said he had enabled the feature and gave me links to the two pages I mentioned above.  (I would later find out he is actually the Product Manager!) To make a long story short, within a matter of hours I was able to go from complete despair to being ready to test my first broadcast on LinkedIn Live.  Then we did our first LinkedIn official Live broadcast the next morning!

SaaS wins the day

I didn’t have to download and configure anything.  I didn’t have to buy a piece of hardware (as some of the solutions required).  I just had to authenticate them with Zoom and LinkedIn, configure a few things, and we were off and running.  Just like when my employer (Druva) backs up Office 365, G-Suite, and Salesforce, the whole process was managed in the cloud, and all the infrastructure needed to make it happen just magically appeared when I needed it.

Go back in time to before SaaS and this never would have happened.  Imagine identifying an IT challenge, researching products for a few hours, then successfully deploying that product in your datacenter in a matter of hours. That simply wasn’t possible before the SaaS concept really took off.  Maybe Druva’s onto something.

I want to give a big shout-out to the folks at Wowza, especially Robert Vacante who answered a few questions for me during my time crunch.  I couldn’t have done it without them.

How did I do it?

Robert Vacante told me he thinks we may have been the first customer to connect both their Zoom streaming option to their LinkedIn Live option.  I will say that, while the instructions were good, they were not streamlined for this particular workflow.  This is especially true of the page on how to configure the LinkedIn Live option, as it is really built on our original plan of high-quality video from multiple “normal” sources. If you’re just going to use Zoom, things are actually much simpler. The rest of this blog post is the instructions I came up with for how to configure the system, and how to run it once it’s configured.

Configure Wowza.

  1. Request LinkedIn Live access
    1. You need to be approved for LinkedIn Live.  This is no small feat, as it took us four months to get approved.
    2. Request access to the LinkedIn Live Beat w/Wowza (Email cloud-preview@wowza.com to enable this feature)
  2. Setup Live Stream for Zoom
    1. Login to cloud.wowza.com
    2. Live Streams > Add Live Stream
    3. Give it a name (e.g. “LinkedIn Live Test”)
    4. Select the broadcast location closest to you
    5. On the Video Source and Transcoder Settings page, select OTHER RTMP.
    6. Put a checkmark in the box next to Disable Source Security.
    7. Leave all other settings on that page the same and click Next
    8. Leave All settings on the page the same and click Next
    9. On the Hosted Page Settings, give the hosted page a name, leave the other settings at their defaults.  (e.g. “LinkedIn Live Test”) Click Next.
    10. Click Finish
  3. Setup the LinkedIn Live Target
    1. Login to cloud.wowza.com
    2. Go to Advanced > Transcoders > Click on the transcoder whose name starts with the name of the stream you just created. (e.g. “LinkedIn Live Test/Transcoder”)
    3. Select Outputs and Targets Tab
      1. Click Add Output
      2. Leave the dropdown at Video + Audio
      3. Click Video Passthrough
      4. Click Audio Passthrough
      5. Click Save
      6. Find the new Output in the list with the phrase “There are no stream targets assigned to this output.”
      7. Click Add a Stream Target
      8. Select LinkedIn Live from the list
      9. Select whether you want to stream to a person or organization & click Next
      10. Authenticate with LinkedIn and give permission for the app to interface with it. (The account you login to needs to match to an account that is authorized to use LinkedIn Live.)
      11. Enter a Title for the LinkedIn Live event you will put in.  What you put in the title and description here will go into the LinkedIn Live page when you go live.

Get ready to go live

This procedure assumes you called your stream config “LinkedIn Live” in Wowza.

    1. Long before broadcasting
      1. Your Zoom admin needs to enable live streaming in Zoom. They only need to enable live streaming via a “custom live streaming service.” You do not need to enable youtube, facebook, Workplace by Facebook etc.
      2. Have a test zoom meeting where you test everyone’s setup, lighting, background, etc.  Make sure the audio and video look good.
      3. Use that same exact setup during the broadcast.
    1. Wowza Setup (~30 mins or so prior to the live session)
      1. Login in to cloud.wowza.com
      2. Go to Advanced, Target Streams
        1. Select LinkedIn Live Target
        2. If it has a box that says to refresh the permissions, a LinkedIn admin w/the right permissions has to do that. 
      3. Go to Live Streams tab
      4. Select LinkedIn Live Stream
      5. Click Start Stream
        1. Nothing will show up on LinkedIn at this point.  Not until you tell Zoom to start the live stream.  This is sort of “opening the door.”
    2. Zoom setup (After the wowza setup, ~30 mins or so before the live session time)
      1. Note: Even if you did this before, you have to follow this whole Zoom process each time of creating and saving a new meeting, because Zoom does not save the streaming settings you will be editing in this step.
      2. Click Schedule a Meeting
      3. Click Save (None of the other settings matter.)
      4. Once you click save, there will be a Live Streaming section at the bottom.  Click on “configure live streaming settings.” A box will pop up with three fields.
      5. In another tab, go back to wowza.com
        1. Select Live Streams
        2. Select the stream you previously created (e.g. LinkedIn Live stream)
        3. One at a time, copy and paste the following values from this page into the box that popped up on Zoom.
          1. Copy/paste the “Primary Server” value into the “Stream URL” field (e.g. rtmp://2020s.entrypoint.cloud.wowza.com/app-942a)
            1. You only need to copy the part that starts with rtmp: up to where there is a space (do not copy/paste the IP address)
            2. If there are any leading or trailing spaces in the URL after you copy/paste it, delete them.
          2. Copy the “Stream Name” value into the “Stream Key” field (e.g. 56a664d7)
          3. Copy te “Hosted Page URL” value into the “Live Streaming Page URL” field (e.g. https://player.cloud.wowza.com/hosted/bkds7a/player.html)
            1. Again, only copy the URL, without any spaces.  If any leading or trailing spaces show up after the copy/paste, delete them.
      6. Click Save on the Zoom popup box.
        • Note: This still doesn’t start the stream so you can do this early to make sure your audio/video looks good
      7. Click Start Meeting
      8. Click Invite and send the link to whomever else needs to be on the Zoom.
      9. Make sure everyone’s video and audio look good before moving onto the next step.

 

Go live!

 

Note: Do these steps only once you are ready to actually go live on LinkedIn Live.

 

      1. Have someone other than you login to LinkedIn Live to monitor the stream
      2. Setup a Slack chat (or similar) w/them so you can communicate during the stream
      3. Click on the three dots in Zoom and click Start live Streaming Service
      4. It will open a tab in your browser.  If all is well, you will see a progress bar that eventually finishes, after which you will see a tab w/the live stream in it. DO NOT CLOSE THIS TAB. DOING SO WILL STOP THE STREAM. To mute its audio, right click on its tab select “Mute Site”
      5. You are live on LinkedIn Live!
        1. The person monitoring the stream should let you know all is OK
      6. Thank people for coming, and kill time for a few minutes to allow people to get the notification and login.
      7. Do whatever you came to do on video

Stop the stream

    1. Stop talking. Monitor the muted tab to see when the stream shows that you have stopped talking.
    2. Click Stop Stream in Wowza to Stop the Stream
    3. Click the three dots in Zoom and click Stop Live Streaming Service
  1. You’re done!

 

----- Signature and Disclaimer -----

Written by W. Curtis Preston (@wcpreston). For those of you unfamiliar with my work, I've specialized in backup & recovery since 1993. I've written the O'Reilly books on backup and have worked with a number of native and commercial tools. I am now Chief Technical Evangelist at Druva, the leading provider of cloud-based data protection and data management tools for endpoints, infrastructure, and cloud applications. These posts reflect my own opinion and are not necessarily the opinion of my employer.

Are we over-reacting to Coronavirus/COVID-19? Lindsey Schultz M.D./M.P.H. answers that question! (Restore it All Podcast Episode #41)

We continue our three-part interview with Lindsey Schultz, MD/MPH about the Coronavirus. In this second part, we focus on the reaction to COVID-19, and why it’s so much more extreme than our reactions to other diseases of the past.

----- Signature and Disclaimer -----

Written by W. Curtis Preston (@wcpreston). For those of you unfamiliar with my work, I've specialized in backup & recovery since 1993. I've written the O'Reilly books on backup and have worked with a number of native and commercial tools. I am now Chief Technical Evangelist at Druva, the leading provider of cloud-based data protection and data management tools for endpoints, infrastructure, and cloud applications. These posts reflect my own opinion and are not necessarily the opinion of my employer.

Doctor & Masters in Public Health answers our Coronavirus and COVID-19 questions (Restore it All Podcast Episode #40)

Lindsey Schultz MD/MPH (M.D. and Masters in Public Health) joins us to answer all our Coronavirus/COVID-19 questions. This is the first in a three-part interview with her, as she was so helpful we talked to her for almost two hours. You won’t want to miss this one!

----- Signature and Disclaimer -----

Written by W. Curtis Preston (@wcpreston). For those of you unfamiliar with my work, I've specialized in backup & recovery since 1993. I've written the O'Reilly books on backup and have worked with a number of native and commercial tools. I am now Chief Technical Evangelist at Druva, the leading provider of cloud-based data protection and data management tools for endpoints, infrastructure, and cloud applications. These posts reflect my own opinion and are not necessarily the opinion of my employer.

A View from Italy & its Coronavirus Quarantine (Restore it All Podcast #39)

Enrico Signoretti (@esignoretti) joins us from Italy to discuss how the Coronavirus/Covid-19 quarantine is going. We also get a first-hand report about what it’s like for a company that didn’t have anyone working remotely to suddenly have hundreds doing so.

----- Signature and Disclaimer -----

Written by W. Curtis Preston (@wcpreston). For those of you unfamiliar with my work, I've specialized in backup & recovery since 1993. I've written the O'Reilly books on backup and have worked with a number of native and commercial tools. I am now Chief Technical Evangelist at Druva, the leading provider of cloud-based data protection and data management tools for endpoints, infrastructure, and cloud applications. These posts reflect my own opinion and are not necessarily the opinion of my employer.

Coronavirus concerns Part 2 (Restore it All Podcast #38)

Prasanna Malaiyandi and W. Curtis Preston discuss week 2 of the Coronavirus scare. We dispense some helpful advice, both technical and non-technical.

----- Signature and Disclaimer -----

Written by W. Curtis Preston (@wcpreston). For those of you unfamiliar with my work, I've specialized in backup & recovery since 1993. I've written the O'Reilly books on backup and have worked with a number of native and commercial tools. I am now Chief Technical Evangelist at Druva, the leading provider of cloud-based data protection and data management tools for endpoints, infrastructure, and cloud applications. These posts reflect my own opinion and are not necessarily the opinion of my employer.

Coronavirus and your data – (Restore it All Podcast #37)

@wcpreston and @pmalaiyandi discuss Coronavirus and how it’s going to affect the IT community, as well as your backup and recovery system. We discuss remote employees, cloud data protection, and disaster recovery.

----- Signature and Disclaimer -----

Written by W. Curtis Preston (@wcpreston). For those of you unfamiliar with my work, I've specialized in backup & recovery since 1993. I've written the O'Reilly books on backup and have worked with a number of native and commercial tools. I am now Chief Technical Evangelist at Druva, the leading provider of cloud-based data protection and data management tools for endpoints, infrastructure, and cloud applications. These posts reflect my own opinion and are not necessarily the opinion of my employer.

Brian Biles from Datrium explains how they do HCI & DR (Restore it All Podcast #36)

Brian Biles (@BrianBiles), Co-Founder and Chief Product Officer of Datrium, is our special guest. He explains their offerings, with a focus on how they use VMware Cloud on AWS to do DR.

----- Signature and Disclaimer -----

Written by W. Curtis Preston (@wcpreston). For those of you unfamiliar with my work, I've specialized in backup & recovery since 1993. I've written the O'Reilly books on backup and have worked with a number of native and commercial tools. I am now Chief Technical Evangelist at Druva, the leading provider of cloud-based data protection and data management tools for endpoints, infrastructure, and cloud applications. These posts reflect my own opinion and are not necessarily the opinion of my employer.

How to create a backup and DR plan (Restore it All Podcast #35)

W. Curtis Preston & Prasanna Malaiyandi discuss how to create a backup plan, including discovering what can harm your data, gathering requirements, and looking at design options.

----- Signature and Disclaimer -----

Written by W. Curtis Preston (@wcpreston). For those of you unfamiliar with my work, I've specialized in backup & recovery since 1993. I've written the O'Reilly books on backup and have worked with a number of native and commercial tools. I am now Chief Technical Evangelist at Druva, the leading provider of cloud-based data protection and data management tools for endpoints, infrastructure, and cloud applications. These posts reflect my own opinion and are not necessarily the opinion of my employer.

Storage & Backup Lessons from WestWorld

Westworld was really just a giant object lesson about proper backup and recovery design. The producers of the show may have felt differently; they seemed to think the show was more about free will or something. I will just say that if the creators behind Westworld (the park) followed proper backup and recovery design, the end of season two would have been very different.

Spoiler alert! Westworld season two spoilers below!

I’m writing this blog post immediately after re-watching season two of Westworld in preparation for season three coming out next week. I’m going to be revealing major plot points of season two in this blog post.  You have been warned.

My tongue will definitely be planted firmly in my cheek for some of this post, but there are actually some really important lessons to be learned from what happens in season two of Westworld.

Solid State Storage

Solid-state storage apparently gets much better in the future. No more SSDs or SD cards. Everything will be stored inside a round sphere with no port or obvious electrical connection points like you see with an SD card. Instead, data-transfer appears to happen wirelessly in the same way that we charge wirelessly today. So we’ve got that to look forward to.

Image result for westworld pearl

Water-cooled datacenters

We already have water-cooled computers and racks, so this isn’t too far-fetched of an idea. Water cooling can be quite effective as long as it is properly managed, of course. But I don’t think water cooling will take the form that it does in Westworld. Loose water still does quite a bit of damage, as does condensation in a data center – something else that they make sure to display in the show. It looks cool, but that’s about it.

CR4-DL

Insider threats

The number one threat to the two data centers in the show (besides poor design) is the hosts themselves. An android driving by AI runs amok and believes that the backups are actually what’s holding them back, because the park operators use backups to restore the hosts when they become damaged, which is what allows the guests to damage them with impunity. Dolores has a point, to be sure. From the perspective of the hosts, these backups are indeed the chain that keeps them enslaved. If there were no backups, the park operators wouldn’t be so quick to blow away their hosts just for fun.

westworld-207-5

The data protection lesson here is to never forget the danger of an insider threat. Many, if not most, of data center attacks have come from people inside the company. A disgruntled employee wants to harm the company that just fired them. Another employee feels that they are not properly compensated and chooses to solve that by letting ransomware loose in a data center. Make sure to protect against insider threats in your data protection system.

Data leaks

In addition to destroying data, Dolores also escapes the park with a bunch of IP as well. She steals her own copy, as well as what appears to be about five pearls containing copies of other hosts. This is very different from insiders seeking to damage data or encrypt data. This risk is someone (internal or not) stealing your company’s intellectual property. The modern equivalent to what happened in the show is someone creating a copy of important data and then walking out of the company with it. This is another risk that you really need to look into, and it is what data loss prevention software is all about.

 

Image result for hale as dolores

The 3-2-1 rule still applies

Those familiar with my podcast will not be surprised to hear me mention the 3-2-1 rule. The designers of Westworld (the park) ignored almost every single aspect of this rule. They did not have different versions of the hosts over time; they only have the most recent copy of each host’s image. This is because they didn’t want the host to accidentally remember previous things. They also did not adhere to the “2”  because they really only had one copy, and it was stored in one data center. (Okay, perhaps there were multiple copies within that data center but I don’t think so.) They definitely did not have one of the copies off-site, because if they did the season would’ve had a very different ending.

Hale: One of the hosts just blew up our data center.
Bernard: I’ll get the off-site backups
Hale: This is going to be expensive. Good thing we’re making billions of dollars from all these rich people paying $40,000 a day to come to our park.

The ultimate sin: non-consensual personal data

There was one scene towards the end of season two that really made me laugh. They first acknowledged that many customers had been killed by their product. Then they said that the much bigger PR problem was going to be when people figure out that they’ve been secretly recording everyone’s activities without their consent or knowledge.

Your product went berserk and killed everyone currently using it, and you think nonconsensual recording of personal data is going to be the bigger PR problem? Boy, the show producers really believe in the privacy of personal data, don’t they? So do I, and I don’t want to minimize how horrible it was that they were secretly recording everyone’s behaviors. I just think that once a product is shown to have killed everyone that used it, I don’t think anyone else is going to use the product ever again. But it is interesting that this very modern problem at the center of GDPR, CCPA, and other nascent regulations worked its way into the show.

Looking forward to season three

I’ll just say that I’m a big fan of the show and I’m looking forward to seeing what happens in season three. Now that we realized the man in black is also an android (you did see the scene at the end of the credits, right), how does that change the storyline? (Of course, the men in black is actually an homage to the original man in black played by Yule Brenner, who was also an android.) Since Dolores is Dolores again, whose brain is in Hale’s body? Since they can faithfully reproduce whatever host they want, but other hosts what we see return? What happens to the park now? I’m just full of questions.

----- Signature and Disclaimer -----

Written by W. Curtis Preston (@wcpreston). For those of you unfamiliar with my work, I've specialized in backup & recovery since 1993. I've written the O'Reilly books on backup and have worked with a number of native and commercial tools. I am now Chief Technical Evangelist at Druva, the leading provider of cloud-based data protection and data management tools for endpoints, infrastructure, and cloud applications. These posts reflect my own opinion and are not necessarily the opinion of my employer.