Mr. Backup Forced to Restore! (Restore it All Podcast #143)

Start listening

On this week’s episode of Backup Central’s Restore it All, Mr. Backup himself becomes the guest, while Prasanna Malaiyandi takes over as host. W. Curtis Preston explains the backup configuration of the website behind the Restore it All podcast, and how bit rot caused him to have to restore part of it. We talk about bit rot, the 3-2-1 rule, off-site backups, backups stored in S3 and Google Drive, and what it’s like to restore part of a MySQL database. Luckily, the folks at LiquidWeb were very helpful. Watch Curtis explain how practices what he preaches over at

You can listen to the podcast via any of the above links, or you can watch it here:

The following is a transcript automatically created by Youtube.  It will contain many inaccuracies and is being used primarily for SEO to help people find the podcast.

backup central is back up
and if you can’t find a link to this
podcast you will know why
[Laughter] exactly
and rescue me from
[Music] hi and welcome to backup central’s
restored all podcast i’m your host w
curtis preston aka mr backup and i have
with me my herbal tea consultant
Prasanna Malaiyandi
good curtis so
herbal teas are good but what herbal tea
are you drinking
uh today
it’s peach detox and you know and as as
you know listeners don’t know
basically i gave up coffee
uh about a month or so ago
you know it’s a long it’s it’s a
you know it’s a medical thing and
et cetera and you haven’t wanted to jump
off a bridge which is good and i haven’t
wanted to jump off a bridge i did have
to do this slow
decrease the amount of coffee that i was
doing i think we should tell the
listeners what happened to you when you
tried this at the beginning yeah i tried
i tried a d i tried a rapid detox right
i tried to quit cold turkey and i got
i wasn’t functional the next day i was
just like i mean
i think actually one of the podcasts
might have been with that yeah
yeah i don’t know but but like 24 hours
later and i was not a happy camper and
so then i did a little research and
basically it was like don’t do that and
capping withdrawal symptoms yeah so i
lowered it down to one cup a day and
then i got rid of the cup a day and now
i’ve had to
get in and it’s about the caffeine it’s
both about the acid of the coffee as
well as the caffeine and so
i’m not just switching to t i’m
switching to herbal teas which are
caffeine free and today
the the flavor
du jour is peach detox
so i know i see you’re drinking yogi
brand ginger yogi brand tea which i love
yogi i actually really like their ginger
herbal tea it’s good i like ginger and
my tea even normal black tea i drink
with ginger and milk don’t ask me but
that’s how i do it
i can never get into like the fruit
flavored teas i just find it too subtle
some of them are super subtle right they
put the bee in subtle
and i tried another peach tea there was
actually a curtis brand
of course
for the record it was not me that bought
it was still in my wife
she uh she was like oh she you know
you know so she bought it and it was a
peach one and it was
yeah it put the bee in subtle it was it
it felt like i was drinking hot water
but this is actually you know
but again i’ve had to adjust my palate
right i have to this is the new world i
live in i think my favorite is turmeric
something or other oh yeah those are
good too yeah yeah so
um yeah but it you know it’s a
it’s a new world and and honestly the t
it’s kind of a crutch it’s it’s it’s to
deal with that
the what’s something the memory
yeah was a hot drink and it’s the muscle
memory of of sipping on something right
um although the other day i found myself
walking into the kitchen and starting to
pour a cup of coffee i was like wait
i walked all the way from here into the
kitchen took out the pot and i had it in
my hand i was about to pour it i was
like wait
i’m not supposed to do this anymore
[Music] you’re like stupid muscle memory yeah
that’s stupid no i think it’s
interesting i think there are a lot of
herbal teas out there which are good and
i think people just aren’t exposed to
all of those that are out there
right everyone’s like oh yeah herbal
teas no caffeine it’s not going to be
good but some of them taste good some of
them are really tasty yeah i like the
ones that tend to be spicy you know yeah
you know i like
i like spice and um
yeah we’ll see as i
i will keep our listeners updated as i
well as i work my way through the
well the other thing i’ve also found is
sometimes just adding a touch of honey
to herbal yeah i’m also yeah the the
only problem with that is i had switched
off of sweeteners in my coffee
so i don’t want to add calories it’d be
very easy
in in the dieting world uh
of which i have been apart wait for way
too long uh you know we use the phrase
we don’t want to drink our calories if
i’m gonna i’m gonna get calories i’m
gonna eat something you know that’s true
but but yeah um
well first off i’ll throw out our usual
disclaimer persona and i work for
different companies i work for druva he
works for zoom this is not a podcast of
either company
and the opinions that you hear are there
for our own
and uh please rate this podcast at
and if you
are interested in the topics that we’re
interested in we would love to have you
on and uh we you know we need a whole
new batch of guests for for 2022. and uh
we’d love to have you on it’s it’s a
it’s a warm and pleasant
environment it is pleasant i know that’s
what i’ve been told by our guests is
that we’re we’re nice people
well one of us
the other of us is told to behave
i’ll let the audience members guess as
to whom i’m referring there
but um
and but i i had an event
i had
one of these things where
so you’re like a guest on the show today
i’m like a guest on the show yeah
the thing that ended up happening to me
really um
what do you call it it
sort of incorporated
so much of what we talk about on the
show we’re going to talk about bitrat
we’re going to talk about the three two
one rule right we’re going to talk about
actually the fact that
i wasn’t an expert
in restoring the thing that i needed to
which is probably common for a lot of
people out there right which is probably
common for a lot of people i may be mr
backup but i you know i
that this is a backup that i don’t i
don’t do very very very often
so um i thought today
we would let you sort of
interview me
as a guest
as a guest like what
happened to me oh
just only
regarding this incident or just in
general i can interview you
so when you were five what happened to
you that changed your life
yeah so so let’s fast forward up until
event if i recall from our past
conversations right it happened
a couple of weeks ago right within the
last couple weeks ago
yeah and maybe provide some background
as people know right this is the
backup centrals restore all podcasts so
a lot of people may not be familiar with
what is backup central so maybe start
from there
so backup central
is my website that
you know it originally started as a blog
back in 1999 if you can imagine that
back before we even really had the term
i did a lot of blogging over the years
we had some forums on there those forums
at this point have been turned into um
read-only forums because
they’re just
they just weren’t generating the kind of
attention that i that i that they used
to and they were more hassle than they
were worth so i just put them in read
only but today
it is its primary purpose is to placed
from which you can go to anywhere to
hear this podcast right so if you if you
follow this podcast on
each podcast has a blog post and in that
blog post is a
as a link to all of the different places
where you could potentially listen
to the podcast and also you can comment
on episodes right
and a lot of the people who would be uh
visiting this website right are probably
like either vendors or backup admins or
people who just care about backups or
want to know learn more right about data
yeah it like exactly right it is
definitely aimed at
aimed at people that are interested in
the same sorts of things that this
podcast is about
gotcha so
and you still run this and this is where
you can find the latest episodes of
backup central so what happened
right that
i got a notice
the uh oh wait actually before sorry
before we talk about what happened
tell us how is this actually set up like
okay it’s a backup central website but
what’s actually what is it running on
it is a cpanel based
you know it’s a linux
web host using cpanel for those of you
that are familiar with
web hosting companies
and it runs in a company called liquid
which it’s been there for quite some
time i i had another provider at one
point but i switched over to liquid web
it wasn’t over here was it
it was
it’s not a vh
over i don’t think ovh was in existence
back then
and and the thing that pushed me over to
liquid web was the superior amount of
technical support i remember the my very
first interaction with them
was that i
i just click the
you know contact us button
at like one in the morning
and i got
really good answers like technical
about how this
company worked how the product worked et
cetera what it was like to migrate from
the other company over to this thing i
had all these technical questions and i
was amazed
you know the expertise and basically
what i found out was that they had 24×7
tech support and i
they and at nighttime they just connect
the sales
you know button to the tech support
and that which is crazy because you’re
just like a single user right who hasn’t
even bought their service yet right yeah
and back then it was like you know 20 a
month or something yeah uh it’s a lot
more than that now but back then it was
a lot smaller and they answered my
questions and boom i became a customer
you know it is definitely a
fully managed
uh site that also happens to use
my sequel in the back end
and and by the way i mentioned the those
the forums are going to rear their ugly
head during the story
and so i use cpanel to back it up and
the way cpanel works is it first
uh takes how would you back it up
well you know
i don’t even want i don’t even want to
go there
because something like what ended up
happening might happen
[Music] so the first thing it does is it takes a
mysql dump
of any
mysql databases that you have in there
and then it creates a tar image of the
home directory that includes those mysql
and stores it in a directory
uh called slash backups and
if you
want to pay for a second disk then you
can store those backups on a second disk
i chose not to pay for a second disk
which at first sounds uh not good yeah
but what i chose instead was to copy
my backups up to two count them two
cloud hosting providers
right um talk about all three one rule
right yeah so
i used um
the um
what was the was it our clone was yeah
what was the one yeah
so basically i’m able to fuse mount you
know do it do a fuse mount of google
drive onto my system
and then
yeah see our uh cpanel copies to that
as a second you know after the backup’s
done it copies that directory and it
also supports backing up directly to
amazon s3
and so i’ve chosen to use s3 as my you
know my long-term storage and by long
term i mean you know weeks not
five years yeah but
and just uh just on that specifically
right so
that amazon s3 copy is
air gapped
right because it is only accessible to
this program it’s not constantly being
connected to the cpanel instance correct
right and first
well the google one too because it gets
disconnected okay that’s what i was
gonna ask like i do i do the fuse mount
at the time of the backup gotcha and
then disconnect so it’s not available
right right
good job
it’s like you know the backup space it’s
like it’s like i enormous space yeah
and you know and some people go hey
don’t you work for druva you should have
druva back it up it’s like i’m not
really druva’s target audience yeah
right or target customer um
you know we we tend to work with larger
companies you know what this reminds me
uh daniel
rose hill rosenhill with
the backup anorak right back
yeah you’re basically piecing together a
backup solution for what you need
because you have a very specific purpose
and there are no great tools well there
are tools but
you don’t want to pay for them right
like you could pay your hosting provider
for that second disk but right you’d
rather use a different solution
yeah and i do pay
um you know now that i think about it i
don’t think i ever put any expiration on
those backups
i don’t know i could have five years of
backups in s3 i don’t even know i should
probably go take a look um but my but my
bill is like three dollars yeah
from amazon so it’s not a it doesn’t
rise up into my you know
and these backups you’re doing daily
yes it automatically does the backup at
something like midnight something like
okay so this is your environment and
so you have so we met the three two one
right now
for these backups here’s a question
because you always tell everyone they
should be doing this
do you ever verify your backups
i have um i’ve verified them i don’t i
i don’t do a full verif verify which you
know we talk about
and that would require me actually
killing the host you know doing the
doing what our friend from what alaska
did yeah i didn’t do that
yeah paul van dyke i have used a backup
within cpanel
to create another virtual host
you know and use that and restore that
right so i have verified the process
kind of end to end i don’t
do it on a very regular basis but on a
semi-regular basis i have i mean the
process that i’m using
with the exception of the s3 copy is
kind of old as time it’s a mysql dump
followed by a tar it’s not
it’s not and and and it’s a script that
like everyone who’s using cpanel is
using i’m not i’m not
rolling my own here no customization
yeah yeah
yeah with the exception of
the thing that’s happening outside of
cpanel is i’m running a cron job to do
the fuse mount of the google drive and
then to unmount it afterwards yeah
because you’re doing this and since we
always talk about this
how are you ensuring that
like if there’s an issue with that
google drive mount the fuse mount that
get flagged
well two things one is i would say
that if the fuse mount copy failed
the the cpanel script would complain to
me okay number one and number two
i i you know
i i’ve written a few shell scripts in my
time and i and if this thing didn’t work
i would just you know i would email the
crap out of myself saying hey there’s
something there’s something you need to
go take a look at right
and and then third i would say that
the google part is the belt in the belt
and suspenders approach right um
in this story you will notice for
example that i did not go to google
so your amazon copy would still happen
even if there was an issue with the
google google correct and that’s all
completely automated within the cpanel
and then by the way at the end of the
cpanel piece i do get an email
uh which i usually at least glance at
saying that everything’s beautiful yeah
gotcha okay so it’s good so you have
your backup system set up fairly well
and it’s automated and it notifies you
if there are any issues as well as
you’re doing these backups
then what happened in the last couple
so i’m i’m gonna say so
i i get an email that says
something’s not right right the backup
and the complaint was that the my sequel
dump didn’t work
there was a problem with the integrity
of my my sql database
imagine imagine my joy yeah when i heard
yeah no it’s not good
and it got worse
i was like well i wonder how long this
has been going on because again i try to
read this email i don’t always read it
saw that it been going on for a little
like i don’t know like over a week did
this this error have been happening over
a week
which which and would be expected as
well in like other customer cases where
you’re running it back sometimes you
don’t catch these failures or you’re
just like yeah it’ll eventually fix
itself it might be something transient
or something else right and
if i had noticed the problem right away
i could have gone immediately to the one
backup that is stored on the host
right there’s always one backup the most
recent backup is stored on the host and
and i could i could use that one to
restore to
you know
the last if you had caught it in time if
i caught it in time but i didn’t catch
it in time
and so it was in other places and so it
i was not going to be able to restore
this this very important database from
the backup that was on-prem as it were
gotcha so what did you do you’re
probably like oh no what can i do like
isn’t there other mechanisms like
wouldn’t your website have gone down if
there was a mysql
inconsistency database and consistency
like what why was that not caught on the
actual website itself
was it just
yeah that’s a great question
and the answer is i don’t know um i i
know that
the database continue the function first
off the table in question
what was a table within
the forum so the forums
the read-only forums which are based on
the the no i’m sorry they were based on
that they are based on forums spelled
with a ph now
the they continued to function
but that there was this integrity check
that was trying to run and every time it
tried to run it was failing but again it
was only failing on one table
and which in the my sequel world
essentially means one file yeah there’s
there was one file each table and my
sequel is represented by a file in the
file system and one of those files had
become corrupt in some random way
so you weren’t able to do a backup but
the website function for now
and so but you didn’t have any local
copies either but luckily for you
because of the three two one rule you
happen to have copies also in google
drive and exactly so how did you decide
where to go to grab your copy
well i
you know just being honest i know aws
better than i know google drive
so i was like i’m gonna i’m gonna i’m
gonna go to aws right
and um
you know it
and and and by the way the the the
forums while they
while they um
[Music] what do you call it um
they are read only and have been now for
well over a year
there’s still a significant amount of my
traffic right so it’s not like and by
the way the table in question
was like forum underscore messages
so it was like you know important the
actual content the main content that was
important for the website
um and so yeah i mean that that decision
was pretty easy i was like yeah i’ll go
i’ll go to aws i’ve been there before i
know how to get stuff easily out of s3
so what was your recovery process like
first i’m sure you had to restore the
data and then recover the database so
what did that look like yeah if i was a
little bit bigger nerd or like a little
current on some of my tech i would have
been able to
the um
the file
[Music] directly from
aws to
the the host
i wasn’t or am not and so what i did was
i went to the aws s3
web interface yep
i found i i went back to my file system
structure well well they went back to my
email to figure out when did this error
start happening so i went to the backup
just before that and by the way the
the good news is my my website
especially that portion of my website as
we’ve already discussed
is the same it’s static so i just needed
to find one that was good so i found a
date when the backup succeeded
so i then um i went to that date in the
you know
uh browse interface yeah
and downloaded that
it’s a gzip tar file so that meant i had
to download it to my computer unzip the
tar file
and then
my plan was to
edit the because my sql dump is just a
text zip file my plan was to edit that
and take out all the tables that weren’t
the table that i needed to repair
and then upload that to phpmyadmin
right and um
and as i was
so i was
i was i was doing that
and that’s when i was like um
i don’t remember what the actual numbers
were but it was like
phpmyadmin you have a maximum
upload capability of let’s say two
gigabyte that would be a number that
would be that would make sense and my
file was like
well this isn’t going to work whatever
the number was it probably wasn’t 75
gigabyte whatever the number was
way over the limit
so i was like okay this isn’t going to
so now what i did was i i scp’d secure
the um
the just the sequel
portion of the corrupted table yeah and
i scp that up to the temp directory on
my host
and then
um i called liquid web
i was like hey
i’m not i could do this if i did enough
googling i could also possibly screw
this up i don’t i don’t do table level
restores of my sequel on the command
line like i haven’t i haven’t done that
in a while yep so how about you guys
take care of this for me and they go
yeah no problem
and then like five minutes later they’re
like it’s done
like thank you
um and so i was able to restore this um
database and this worked because that
was kind of a read-only instance
that didn’t have a lot of changes a step
right so you could just sort of put that
table back and you would be good to go
right there was no dependency yeah other
tables which might have right that’s
really important because if it was a
constantly changing database i wouldn’t
be able to restore just one table
uh i i maybe i could maybe i couldn’t it
would depend on the table
and because if you restore one table to
one point of time and another table to
another point in time you create what’s
called a referential referential
integrity problem
and that is that you have
data you have data in one table that’s
referencing data in another table that
now no longer exists yep but luckily
these day these tables as we were
already as we have already discussed
static and so i could restore any table
at any time and everything would be fine
do you also
for your other tables
um do you
back up the logs in case you needed to
roll forward or roll backwards like say
you had a table that was changing when
you’re doing your mysql dump are you
also capturing those other
not really because again
the database that i have and the in the
system that i have it has
a couple of changes a week
it’s not like a bank where you’re going
to get billion dollar trans billions
right right right right
i mean it’s guaranteed to get at least
one change with mila uploading the most
recent episode of this podcast
and it might get a change if somebody
made a comment or something on a blog
post or i get a
follow back
you know ping back they call it
um because somebody else commented on my
post or whatever but yeah it’s it’s a
really low
change level gotcha so you were able to
get this back up now
how did you verify that the restore was
actually successful or the recovery was
actually successful
well i uh good question i did check
the two checks that i was able to do
was that i was able to see that the
number of rows was the same before and
after but the most important thing was i
ran the integrity check
uh that failed before and it succeeded
and then i immediately ran the backup
process i was just going to ask you i
was like did you do a backup after that
i did i i couldn’t like i did it at like
three o’clock in the afternoon and i
couldn’t wait till midnight i wanted to
get it back up so that was off-site
here’s a question for you so you knew
that you had seven days worth of bad
backups did you go delete those backups
you know
andrew if
that would have been a good thing to do
i didn’t do that you you make a point
you make a good point there
right because just in case like in the
future if you’re like hey i need to go
restore right you probably don’t want to
be restoring bad known bad bad
that that’s actually a really good point
i honestly i was just so happy to have
my table back
i was like you know i really hope i
don’t have to do a podcast episode about
how i lost data on backup central
uh that would that would just be
too embarrassing hey kolfi how’s it
no but that yeah that’s a really good
point that honestly i should go back and
uh get rid of those i mean
having said that
i don’t
really go back in time on backup central
backups right in fact i was like i said
i was just thinking
that i should go check and see
if there’s a snapshot deletion process i
may have set one up in amazon like when
i set this up the first time
but um if i didn’t i now have like
three years worth of
backups of backup central yeah which
which isn’t bad but i think yeah it’s
just that cleanup process just because
if like things go wrong in the future
and you’re under stress right the last
thing you want to do is pull a bad
and i think exactly it’s interesting
because i don’t think we’ve ever talked
about that aspect before
right if you have bad backups to get rid
of them yeah
because here’s here’s a counterpoint
right is
those backups still might be useful for
you even though that one table is bad
the rest of the data might still be
right so here’s a question should you
delete that backup or not or should you
at least market that it’s bad again you
need to understand your process and you
need to understand your computing
environment in my case i would say yes
because there is nothing in there of
unique value
does that make sense even if it contains
your latest blog post
but if but if it but if it did because i
restored it and then i took a backup i
have that okay right there’s nothing
there is nothing that is only in those
right yep but but i think i think the
important thing here is to talk about
sort of
how this process
drove home so many things that we talk
about in the podcast
right so yeah we’ve talked about bit rot
before or applications just go
just break right not in your control
right you didn’t do anything right this
is static data that hasn’t been touched
but things just go bad right sometimes
so so let’s talk about that we’ve talked
about bit rot
and that is the only
explanation i have
for why this file
suddenly became corrupt it is a
read-only database
no there have been no updates to this
database in at least a year
and so
my sequel wouldn’t have corrupted it’s
not writing anything to it yeah but then
it just magically got
randomly corrupted in some unknown way
not like hey all your tables went away
yeah it just said we can’t the integrity
check fails yeah
and that’s why databases are powerful
because if it didn’t have that integrity
check you would never know curtis
exactly exactly
and so and so you know number one
you know i ha
these are things that we talk about a
lot we talk about bit route all the time
and there are people that are out there
in the blogosphere or in the the reddit
sphere if i will right you say that bit
rot never happens
well they’re just like it’s so rare it’s
like a unicorn you know
you know you’re never gonna
you know have it and here i have on this
tiny little host yep um i i had bit
go ahead well no i was just going to say
it’s funny that you bring up bit rot
because i was recently invest or
researching um
so i have a bunch of data at home and
sits on a hard drive and i was thinking
of getting a new hard drive like a
smaller hard drive
and i was like oh ssds are so cheap but
this is more like data i rarely ever
access and so i started digging into it
based on our previous conversations i
was like
okay so what does bit rot look like on
hard drives versus ssds
if you don’t power up an ssd like at
least once a year there’s no guarantee
you’re getting the data and it’s
actually probably much less ssds are a
really bad long-term archive solution
unless you’re gonna empower them all the
time exactly so i looked and i was like
no maybe i should just get a spinning
disc and that’s probably the best option
for me
for especially for this particular data
that i don’t access often and
people don’t realize that right because
they’re like oh ssd super fast it must
be better in all ways than spinning disk
and it’s like nope make sure you pick
the right media for what you need
yeah and the degree to which bit rot
will affect you is the degree to which a
single change in a single file will
a big thing so in my case it corrupted a
single file
which happened to contain the most
important table
of the database that i was running which
essentially if that had been come if
that had become corrupted
i had been unable to recover it i would
have lost
all that data the bulk of the content on
backup central
so a similar thing would be if you’re
storing long-term backups or archives on
a disk that suffers bit rot
and what you have is a giant
like tarball yeah
so a big file like that as opposed to
like a timer
yeah right yeah
um or as opposed to the way druva writes
to the cloud right where we we do lots
of little pieces
um if you have one big tarball and a
piece of that gets corrupted it can
it can make that tar ball unreadable yes
it’s rare no it will probably not happen
to you
but if it does is that a risk you’re
willing to take
right because in the end it all comes
down to risk right you could have paid
uh your provider your service provider
that extra copy on a separate disk
right but instead you chose to have a
more resilient architecture and store
that data off site in two separate
right for resiliency purposes which i
think covers both the three two one rule
and the air gap rule that we always talk
about yes absolutely right and
um and then
having the data a long-term copy of the
data because we talk about that too
where sometimes
and in this case it was due to human
i don’t know human error is the right
word but
due to my own
i didn’t notice the problem happen and
so i had to go back to a backup a lot
longer than i would think that i would
normally need to go back to
like because it was like over two weeks
ago i wouldn’t think that i would need
to restore data from my website from
over two weeks ago but the data got
corrupted two weeks ago and i didn’t
notice yeah so that’s another concept
right and i know we’ve talked about this
with uh
uh celeste when she came on talking
about ransomware how it could sit in the
system for a while and right you might
have to go back a while with the dwell
time i think though this is something
that people who rely on storage
don’t understand right sometimes you’ll
say oh i have seven days of snapshots i
should be able to quickly get back my
data but like you said you may not
notice that something is wrong within
those seven days and now what are you
going to do
i remember a customer of mine back when
i used to you know actually configure
backup systems in the wild
for other people
i remember a
customer which for the record
this was the customer where i’m at the
world famous
stuart little
hello stuart
the boss back then
he he had a very adamant
policy that the backup the retention
policy for all backups was two
so imagine in this case if my retention
policy for backup central had been two
weeks and i didn’t notice this thing it
happened for two weeks then um
it would have been
would have been very very sad
would have been very very sad
um and of course the 3-2-1 rule i mean
now in this case
it wasn’t
the 3-2-1 rule wasn’t necessarily the
the thing that made me recover because
if i had had
the local copy because it wasn’t that
fire didn’t take up my server or
what this does
it it does reiterate the fact that i was
able to go to another place and and pull
down that object but it’s important
though to remember that that was a
painful process for you right versus
having to recover it locally right
bringing it back unzipping it right
trying to figure out how to upload it
all the rest of that version right right
local copy which is sitting right there
that would allow you to quickly access
it quickly restore
and by the way you want to know what’s
funny if i you know this is this is the
if i’d have known what a you know
yeah if i’d uh
if i’d if i’d had a moment and thought
about it you know what what would have
been a whole lot easier
to fuse mount the google drive
yeah and unzip it in place yep
that’s you know what’s funny is it
didn’t hit me until literally just now
that if i had fused mounted the google
i could have unzipped it in place and i
wouldn’t have had to because i
downloaded it everything edited it i
uploaded it you know and you wouldn’t
have had to do any of that that’s why i
was asking you i was like why are you
using s3 but
i guess now we know
well again you know i mean
i specialize in these things but it’s
not like i know everything and it’s not
like i’m
you know well and i think whatever to a
point right
you had a crisis situation
that you were trying to deal with and we
tend to go back to what we know best
right and for you
you said i know s3 better than
google drive and that’s what i’m most
comfortable using right so yes the steps
were more painful but you went with what
you thought would work
right and i think that’s why it becomes
important to have
a playbook if you will
right and having tested out some of
these steps ahead of time before you
actually need it
you know what
and yes and so i reached out to liquid
web my hosting provider and i and i’ve
always loved the
the um
the support that i’ve gotten from them
and when i’ve needed to do something
that again they do it all the time i’m
not a day-to-day system i don’t want to
mess up my own system
and so i i would reach out to them i was
like you know what i got i got to do the
thing with the thing i got it i got to
you know upgrade
php or i gotta
you know increase the buffer size of
something and i’m like i just i just
fire it’s just easier and safer for me
at this point to fire off a ticket to
somebody some stuff i’ll do myself but
but if it’s something that i that i’m
not super confident in i will reach out
to the liquid web folks and they will
make it happen yeah you’re like if i
have to google this i probably should
not be doing this
exactly even if i do have a backup i
could take the host out for for a while
right exactly uh and so i you know i
fired off it’s like hey i got this thing
i’ve uploaded the sql file it’s sitting
there and i just need someone to restore
it to the database and i got an email
back pretty quickly from a sarah satyan
at liquid web and she emailed me
at 10 34 a.m
and yeah this was january 10th by the
way so this was like two weeks ago
she emailed me at 10 34 a.m said let me
take a look at it give me some time to
investigate and i’ll let you know
and then at 1101 she said i have now
restored the database from the backup
you have provided please do verify the
same from your side and let us know if
you need any further assistance that’s
awesome customer service
yeah so 20 less than 20 minutes later
my database back up and running
you know and no liquid web is not
whatever’s not sponsoring i just i just
really like these guys
um um and excu excuse the sexist term i
like these folks sarah is is i’m
assuming not a guy so thank you very
much sarah for your assistance
um and and i hope you you actually hear
this episode
so yeah so i i i i think there’s a lot
that we can learn from from this and
the other thing i want to also point out
even though you’ve been in this space
for a while curtis right we all sort of
make mistakes or we all sort of
go through these learnings right it’s
one of those things that like practice
makes better and hopefully
you don’t have to practice doing
restores a lot but
in case it’s there right you just need
to be able to
get it done
yeah unzipping easy peasy untarring easy
command line restore of a single table
to my sequel
a c panel hosted my sequel which means
that the database name and the password
is weird if you’ve never done cpanel you
don’t know what i’m talking about but
the database i know the database as
let’s say persona
and the but the real database name
is not that it’s something else so i
need to know that
so yeah so that was complicated enough
for me to go i think i’m gonna
think i’m gonna put in a ticket and
sarah came to the rescue so shout out to
so everything’s back up and running i
think we’ve learned a bunch
right not only applying the backup
central is back up
and if you can’t find a link to this
podcast you will know why
[Laughter] exactly
cool any other takeaways curtis
things you would want to tell our
listener yeah i mean
you know i i think even though even
though i had to reach out to somebody
else for help i think it shows that we
practice what we preach here right
that i believe in the 3-2-1 rule i
believe in periodic backups that use the
tools that are provided
i i i minimize as much as possible
customizing and scripting
the the google thing to me is is an
add-on right it’s just a it’s just a
like i did it just because i could do it
but but and also i i did it in a
parallel way in such a way that i i
could have actually edited
the cpanel provides a script that they
update over time
to do all the backups i could have
actually hacked that script
and and done it it’s just a born shell
script i could have done that but i
chose to write a separate script that
just works outside and by the way if
that script fails in addition to you
know in addition to the fact that it’s
no big deal because it’s an extra copy
and the fact that i would get that
script should yell at me when it doesn’t
work but number three you know what else
will happen if that script fails
the mount point where the fuse mounted
file system comes in will get filled up
with the copy
because the script you know the script
the cpanel script it doesn’t know what
i’m doing it’s i told it to copy it into
slash google
and uh
yep it it reminds me
of a hundred years ago uh
we used to have and i think i’ve told
this story on the podcast but we used to
have my i cut my teeth on what what were
called att system 5 3b2s
and they had a single
uh in their internal tape drive was a
quick 80 tape drive which was 80
megabytes and not quick
and we
it was qic
we had a sing i had seven of these three
b2s we had a single
eight millimeter exabyte 8200 drive and
that was the only thing that we could do
a full backup on the quick the internal
drives could do incremental backups
because the full backups were too big so
we had one tape drive and we used
something called rfs which was the
predecessor to nfs
but and and it had a feature that nfs
doesn’t have and that was you could
remote file you could remote mount a
right and so
we had a crown job we had seven systems
seven days a week right so each night a
different host would remote mount that
and sometimes the remote mount wouldn’t
work maybe it didn’t get unmounted from
the last one whatever right right
so what would happen is
the dev the dev tape whatever whatever
the file was that’s a local mount point
it it would just be a local well it
would be a file and it would it would
back it up to that file
and uh the root file system would fill
and so we wrote a script that we were
like okay do a dash f on this thing and
try try to do a we issued some some tape
tape commands to it and see if it works
or not yeah and see if it works and i
think i even like i remember massaging
the script over time and i remember
actually i would do a tar
to the device
and then um and then i would say and
then i would do a
a i would do a du on the file and was
the file any bigger
if the file was bigger then it didn’t
work yep
the things that i did exactly yeah yeah
well i just want to say thank you for
being my interviewer well i also want to
say thank you for sharing your story
curtis i know
it’s hard sometimes to get people to
come on this podcast and
tell our listeners what works what
doesn’t work the mistakes they might
have made
you know you did anything
if my pain can help anyone else
no but it was a good story thank you
curtis for joining the podcast
you know what as i like to say i live to
no i think i think i think the better
thing is you practice what you preach
i practice what i preach i eat my own
dog food i drink the kool-aid although
that is a really bad history behind that
term uh so so thanks thanks persona for
being my uh my interviewer this year
this was fun curtis maybe next time i’ll
do it again all right all right well
for all the listeners
yep and remember to subscribe so that
you can restore it
me all
your head


%d bloggers like this: