Backup Central

A while back where was some discussion about validating rsnapshot backups

In my travels I came across hashdeep
http://md5deep.sourceforge.net/start-hashdeep.html which looks perfect for
this sort of checking

I think you could run

hashdeep -r /mnt/snapshots/Daily.0 > /mnt/snapshots/Daily.0/hashdeep.txt
after taking the snapshot

then

hashdeep -x -k /mnt/snapshots/Daily.1/hashdeep.txt -r /mnt/snapshots/Daily.1
hashdeep -x -k /mnt/snapshots/Daily.2/hashdeep.txt -r /mnt/snapshots/Daily.2
.
.

Should display anything thats changed ie files suffering from bitrot.

or have I missed something?

--
Michael
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Michael John Lush PhD Tel:44-1223 492626
Bioinformatician
HUGO Gene Nomenclature Committee Email: hgnc < at > genenames.org
European Bioinformatics Institute
Hinxton, Cambridge
URL: http://www.genenames.org
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure contains a
definitive record of customers, application performance, security
threats, fraudulent activity and more. Splunk takes this data and makes
sense of it. Business sense. IT sense. Common sense..
http://p.sf.net/sfu/splunk-d2d-c1
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

2011/6/24 Michael Lush <mjlush < at > ebi.ac.uk>:
hashdeep -r /mnt/snapshots/Daily.0 > /mnt/snapshots/Daily.0/hashdeep.txt
after taking the snapshot

then

hashdeep -x -k /mnt/snapshots/Daily.1/hashdeep.txt -r /mnt/snapshots/Daily.1
hashdeep -x -k /mnt/snapshots/Daily.2/hashdeep.txt -r /mnt/snapshots/Daily.2

Since most files in your backup will be hardlinked to earlier versions
of the same files,
this will be useless, as you will then compute the hash of the same file twice.

You could instead try to find all new changed files, i.e. those with only
one link that are also present in the older backups.

Best
Martin

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure contains a
definitive record of customers, application performance, security
threats, fraudulent activity and more. Splunk takes this data and makes
sense of it. Business sense. IT sense. Common sense..
http://p.sf.net/sfu/splunk-d2d-c1
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

On Fri, Jun 24, 2011 at 09:02:26AM +0100, Michael Lush wrote:

In my travels I came across hashdeep
http://md5deep.sourceforge.net/start-hashdeep.html which looks perfect for
this sort of checking

I think you could run

hashdeep -r /mnt/snapshots/Daily.0 > /mnt/snapshots/Daily.0/hashdeep.txt
after taking the snapshot

then

hashdeep -x -k /mnt/snapshots/Daily.1/hashdeep.txt -r /mnt/snapshots/Daily.1
hashdeep -x -k /mnt/snapshots/Daily.2/hashdeep.txt -r /mnt/snapshots/Daily.2
.
.

Should display anything thats changed ie files suffering from bitrot.
or have I missed something?

You do, of course, *expect* files to change from one backup to another,
and often for quite a lot of them to change. It would be more useful, I
think, to run that on the machine you're backing up, rely on rsnapshot
to magickally backup the database it creates, and then check the backup
against that database.

Don't forget to also check owner, group, timestamps and mode!

--
David Cantrell | Reality Engineer, Ministry of Information

I hate baby seals. They get asked to all the best clubs.

------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

On Tue, 28 Jun 2011, David Cantrell wrote:

On Fri, Jun 24, 2011 at 09:02:26AM +0100, Michael Lush wrote:

In my travels I came across hashdeep
http://md5deep.sourceforge.net/start-hashdeep.html which looks perfect for
this sort of checking

I think you could run

hashdeep -r /mnt/snapshots/Daily.0 > /mnt/snapshots/Daily.0/hashdeep.txt
after taking the snapshot

then

hashdeep -x -k /mnt/snapshots/Daily.1/hashdeep.txt -r /mnt/snapshots/Daily.1
hashdeep -x -k /mnt/snapshots/Daily.2/hashdeep.txt -r /mnt/snapshots/Daily.2
.
.

Should display anything thats changed ie files suffering from bitrot.
or have I missed something?

You do, of course, *expect* files to change from one backup to another,
and often for quite a lot of them to change.

But I don't expect files to change /within/ a backup. what I'm trying to
verify is that a file that has been sitting on a disk for the last n
years has not changed since I put it there, if it does change that's a
pretty good sign that I want to swap that disk out sharpish.

It would be more useful, I
think, to run that on the machine you're backing up, rely on rsnapshot
to magickally backup the database it creates, and then check the backup
against that database.

That's a pretty good idea, assuming I don't update the archive while I'm
taking the snapshot it would also demonstrate that the transfer was sound.

--
Michael

------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

On Tue, Jun 28, 2011 at 03:09:28PM +0100, Michael Lush wrote:
On Tue, 28 Jun 2011, David Cantrell wrote:
It would be more useful, I
think, to run that on the machine you're backing up, rely on rsnapshot
to magickally backup the database it creates, and then check the backup
against that database.

That's a pretty good idea, assuming I don't update the archive while I'm
taking the snapshot it would also demonstrate that the transfer was sound.

and you could use it to periodically check your older backups'
integrity.

The problem with this, at least for me, is that merely to read one of my
backups takes about a day.

--
David Cantrell | Enforcer, South London Linguistic Massive

Compromise: n: lowering my standards so you can meet them

------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

Awesome idea.

It can be wise, however, to simply test your DR plan.

i.e If you are backing up SQL... try to restore it!!

Michael.

On Wed, Jun 29, 2011 at 12:25 AM, David Cantrell <david < at > cantrell.org.uk ([email]david < at > cantrell.org.uk[/email])> wrote:
On Tue, Jun 28, 2011 at 03:09:28PM +0100, Michael Lush wrote:
On Tue, 28 Jun 2011, David Cantrell wrote:

                                             It would be more useful, I
think, to run that on the machine you're backing up, rely on rsnapshot
to magickally backup the database it creates, and then check the backup
against that database.

That's a pretty good idea, assuming I don't update the archive while I'm
taking the snapshot it would also demonstrate that the transfer was sound.


and you could use it to periodically check your older backups'
integrity.

The problem with this, at least for me, is that merely to read one of my
backups takes about a day.

--
David Cantrell | Enforcer, South London Linguistic Massive

Compromise: n: lowering my standards so you can meet them


------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net ([email]rsnapshot-discuss < at > lists.sourceforge.net[/email])
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

On Wed, Jun 29, 2011 at 06:14:07PM +1000, Michael Lynch wrote:

It can be wise, however, to simply test your DR plan.
i.e If you are backing up SQL... try to restore it!!

Indeed. As it happens, I don't back up my RDBMSes, but if I did I'd
certainly do that too.

If you haven't tested your backups, you don't *have* backups, you have
hopes.

--
David Cantrell | A machine for turning tea into grumpiness

------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

Ha ha

That reminds me.. I need to test my backups

On Thu, Jun 30, 2011 at 9:12 PM, David Cantrell <david < at > cantrell.org.uk ([email]david < at > cantrell.org.uk[/email])> wrote:
On Wed, Jun 29, 2011 at 06:14:07PM +1000, Michael Lynch wrote:

It can be wise, however, to simply test your DR plan.
i.e If you are backing up SQL... try to restore it!!


Indeed.  As it happens, I don't back up my RDBMSes, but if I did I'd
certainly do that too.

If you haven't tested your backups, you don't *have* backups, you have
hopes.

--
David Cantrell | A machine for turning tea into grumpiness


------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net ([email]rsnapshot-discuss < at > lists.sourceforge.net[/email])
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

Hallo, Michael,

Du meintest am 01.07.11:

If you haven't tested your backups, you don't *have* backups, you
have hopes.

That reminds me.. I need to test my backups

Just to make you tremble:
about 2 years ago my colleague and I had seen that (on one machine) the
"master" backup was damaged since more than 2 months ...

Viele Gruesse!
Helmut

------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

Lol.

We had a customer and the backup software had been backing up fine...
But what was on disk was not the same as what was in memory.

When a reboot was done, the ntfs filesyatem was stuffed... But it was
also stuffed in the backup!!!

The supposed 'succesful' backups for the last six mo tha were useless!

Regards

On 01/07/2011, at 2:28 AM, "Helmut Hullen" <Hullen < at > t-online.de> wrote:


------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss