Backup Central

Hi,

I'm cuurently evaluating BackupPC for our firm and for now,
I like it a lot, so it's nice to hear it's so successful for you.

We already have a tape backup system in place, with a 100GB
(uncompressed) tape drive, so currently I'm playing around with the
tape backup function of 2.1beta.

I have a ~300GB tape changer, and ATL L200 to be exact, and I have connected
it to the BackupPC and did get it to archive a small system to tape.
However if I tried to use the compression it failed, saying "2004-06-02
16:58:02 Archive failed (Error: /usr/local//bin/BackupPC_tarCreate, compress
or split failed)" None gets me to tape, but I have to be careful not to
exceed one DLT-8000's capacity or I hang things....

I tried putting the archives to disk first, as for now the BackupPC
machine doesn't have a tape drive.
So I set
$Conf{ArchiveDest}='/srv/data/archives'
Strangely, setting this in the pc/'archivehost'/config.pl didn't work.

From reading the comments in BackupPC_archiveHost, it seems it doesn't
use split when archiving to a device (tape).
It uses BackupPC_tarCreate to extract the data from a certain host and
backup number and convert it into a tar archive. That is piped to
stdout, and into the tape device, or if you specified a directory
as $Conf{ArchiveDest}, pipes it through split. It seems split doesn't
like sizes over 2GB, then the archive fails. Don't know why it fails
with gzip, though...


I will have to figure this out first, so Question One: How does the "split"
work? Is it a tar of N for each host and then cat'ed into split in the
given size? This would suggest that it gets tmp'd somewhere and spooled out
to tape; From what I can interpret of that code, this seems to be the case.
I have a 500 GB RAID on this box but, well, it is getting full, so I want to
be careful to leave room for the backups...

Question Two: Is there a way of using the -M (--multiple volumes) switch in
tar to accomplish this in the existing code? If it is not too difficult to
do, then it helps me greatly in question three.

It seems the archiveHost script doesn't use the "real" GNU tar, but
BackupPC_tarCreate, so probably it doesn't have an -M switch.
Would be a useful feature, though...


Question Three: If the answer to two is yes, then could we also use the tar
switch "--new-volume-script file" ( Run a script at the end of each archive
volume)? This would allow me to automatically have the tape changer move to
the next tape and tar happily continues... I have been using the perl
module juke.pm with the program mtx to do this with some rather kludgy
scripts of my own, but it isn't a prime time solution.

So you have some solution that splits up the archive and changes tapes?
I always wonder when you split up tar archives, can you extract the data
of the parts independently? normally not, I suspect, so to restore
an archive you need all the tapes.

In my tests it did one tar archive per host, so if these are smaller
than one tape, you wouldn't need to split archives, but fill each tape
with several tar's, then change tapes, etc.


Question four: If the above will work, then might there be a way of setting
this to go on a sensible schedule, with a message to the backup operator
when the Data Storage Elements are full and need to be changed? My boss
wants this in case I get hit by a truck tomorrow on the way to work.

That's one of my questions, too. I was wondering if one could call
BackupPC_archive from the command line, instead of using the web
interface. Thus, one could use cron to automate it.

Another thing I was thinking about is some kind of tape management, like
amanda does. Like giving labels to each tape, checking if the right tape
is in the drive before starting to write (prevent erasing the wrong
tape), storing some catalogue of the tapes.

Anyway, some comments from someone with more insight about BackupPC
would be helpful. I don't know too much perl, I know more about C (++)
and Python...

Best regards,

Oliver Freyd



-------------------------------------------------------
This SF.Net email is sponsored by the new InstallShield X.
From Windows to Linux, servers to mobile, InstallShield X is the one
installation-authoring solution that does it all. Learn more and
evaluate today! http://www.installshield.com/Dev2Dev/0504
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

Hi,
Yes Craig has done a bang up job; My hat is off to you, man! BackupPC is,
in my twenty years experience in this game, as good as any of the commercial
packages available. I have tried most of them over the years, and only
Legato works as well; it has a bunch of great features too, but it costs
Mega$$. God forbid if something doesn't work on it though, as support is
non existent....

Yes, from what I can glean from the exceptionally well documented scripts,
Archive::Tar is the basis of the module. I don't see any -multi option when
I checked it on cpan. It appears to be a in memory archive that resembles
tar. Jos Boumans, it's writer, suggests tar if you were to need speed over
the functionality of the module.

So, hmmm, I need to calculate the size of the archives going to tape per
backup run: measure remaining space, trigger the autoloader to switch tapes
when the next archive is larger than the remainder (Some tape wastage there)
; recommence the archive. This is okay, but I have several systems which
have +50GB archives. This means, without tape spanning (--multi-volume) I
can't conform to the Disaster Recovery Plan (DRP) from my Boss. At least
not from within BackupPC, anyway. This is where I am at right now, as I
have unreliable or split archives of the big machines. I can't see my
machines having less data on them in the future either. Funny that; as
drives get bigger, the researchers want even stuff crammed on 'em....

Yes, I basically sit in front of the console and trigger the tape sequence
by hand. If things go wrong, I can massage things, or reboot and try again
if necessary. All this manual intervention eats up a considerable amount
of time.

Incidentally, Tar will create indexes of tapes, and as you rightly suppose,
you need to have the entire backup set to get things figured out during a
restore. Tar will label the tape and the --multi-volume will increment the
label if you set it up right.

So, we both think a scriptable invocation of the archive process would be
good. I am looking for: a way of getting the archive to tape, preferably
without needing another Three Hundred GB of disk space as a buffer; hooks
within BackupPC to control my tape changer so that I can do my snapshots for
offsite retention; a scheduler for running this in a prescribed manner; a
notification system for errors; perhaps even a indexing scheme for the
various backup sets.

I think the indexing and tracking of the tape sets might be a bit of a
bigger project, although that would be great if it could be had. I imagine
that would require database integration that currently doesn't exist. That
would be way over my head to implement, but I would certainly participate.

So, to sum up, I suppose I'm kinda volunteering to try of different
suggestions on how I might overcome this tape spanning difficulty.
Unfortunately, I'm not much of a coder, so I must turn to the group at large
to help me work this one through.

Thanks,
Dave






-----Original Message-----
From: Oliver Freyd [mailto:Oliver.Freyd < at > ion-tof.com]
Sent: 3-Jun-04 7:20 AM
To: IIT, Ottawa, Helpdesk 2nd Floor
Cc: 'backuppc-users < at > lists.sourceforge.net'
Subject: Re: [BackupPC-users] RE: Tape Backup


Hi,

I'm cuurently evaluating BackupPC for our firm and for now,
I like it a lot, so it's nice to hear it's so successful for you.

We already have a tape backup system in place, with a 100GB
(uncompressed) tape drive, so currently I'm playing around with the
tape backup function of 2.1beta.

I have a ~300GB tape changer, and ATL L200 to be exact, and I have
connected
it to the BackupPC and did get it to archive a small system to tape.
However if I tried to use the compression it failed, saying "2004-06-02
16:58:02 Archive failed (Error: /usr/local//bin/BackupPC_tarCreate,
compress
or split failed)" None gets me to tape, but I have to be careful not to
exceed one DLT-8000's capacity or I hang things....

I tried putting the archives to disk first, as for now the BackupPC
machine doesn't have a tape drive.
So I set
$Conf{ArchiveDest}='/srv/data/archives'
Strangely, setting this in the pc/'archivehost'/config.pl didn't work.

From reading the comments in BackupPC_archiveHost, it seems it doesn't
use split when archiving to a device (tape).
It uses BackupPC_tarCreate to extract the data from a certain host and
backup number and convert it into a tar archive. That is piped to
stdout, and into the tape device, or if you specified a directory
as $Conf{ArchiveDest}, pipes it through split. It seems split doesn't
like sizes over 2GB, then the archive fails. Don't know why it fails
with gzip, though...


I will have to figure this out first, so Question One: How does the
"split"
work? Is it a tar of N for each host and then cat'ed into split in the
given size? This would suggest that it gets tmp'd somewhere and spooled
out
to tape; From what I can interpret of that code, this seems to be the
case.
I have a 500 GB RAID on this box but, well, it is getting full, so I want
to
be careful to leave room for the backups...

Question Two: Is there a way of using the -M (--multiple volumes) switch
in
tar to accomplish this in the existing code? If it is not too difficult
to
do, then it helps me greatly in question three.

It seems the archiveHost script doesn't use the "real" GNU tar, but
BackupPC_tarCreate, so probably it doesn't have an -M switch.
Would be a useful feature, though...


Question Three: If the answer to two is yes, then could we also use the
tar
switch "--new-volume-script file" ( Run a script at the end of each
archive
volume)? This would allow me to automatically have the tape changer move
to
the next tape and tar happily continues... I have been using the perl
module juke.pm with the program mtx to do this with some rather kludgy
scripts of my own, but it isn't a prime time solution.

So you have some solution that splits up the archive and changes tapes?
I always wonder when you split up tar archives, can you extract the data
of the parts independently? normally not, I suspect, so to restore
an archive you need all the tapes.

In my tests it did one tar archive per host, so if these are smaller
than one tape, you wouldn't need to split archives, but fill each tape
with several tar's, then change tapes, etc.


Question four: If the above will work, then might there be a way of
setting
this to go on a sensible schedule, with a message to the backup operator
when the Data Storage Elements are full and need to be changed? My boss
wants this in case I get hit by a truck tomorrow on the way to work.

That's one of my questions, too. I was wondering if one could call
BackupPC_archive from the command line, instead of using the web
interface. Thus, one could use cron to automate it.

Another thing I was thinking about is some kind of tape management, like
amanda does. Like giving labels to each tape, checking if the right tape
is in the drive before starting to write (prevent erasing the wrong
tape), storing some catalogue of the tapes.

Anyway, some comments from someone with more insight about BackupPC
would be helpful. I don't know too much perl, I know more about C (++)
and Python...

Best regards,

Oliver Freyd


-------------------------------------------------------
This SF.Net email is sponsored by the new InstallShield X.
From Windows to Linux, servers to mobile, InstallShield X is the one
installation-authoring solution that does it all. Learn more and
evaluate today! http://www.installshield.com/Dev2Dev/0504
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

"IIT, Ottawa, Helpdesk 2nd Floor" writes:

I have a ~300GB tape changer, and ATL L200 to be exact, and I have connected
it to the BackupPC and did get it to archive a small system to tape.
However if I tried to use the compression it failed, saying "2004-06-02
16:58:02 Archive failed (Error: /usr/local//bin/BackupPC_tarCreate, compress
or split failed)" None gets me to tape, but I have to be careful not to
exceed one DLT-8000's capacity or I hang things....

The error is a little crytpic. What it means is that BackupPC_archiveHost
tried to run a shell command that includes BackupPC_tarCreate, optionally
piped into compress and split, and output to the output device or directory.

One likely cause is that the backuppc user doesn't have permission to
write to the output device.

I need to improve the error reporting...

I will have to figure this out first, so Question One: How does the "split"
work? Is it a tar of N for each host and then cat'ed into split in the
given size? This would suggest that it gets tmp'd somewhere and spooled out
to tape; From what I can interpret of that code, this seems to be the case.
I have a 500 GB RAID on this box but, well, it is getting full, so I want to
be careful to leave room for the backups...

The split is per host, so that means per tar archive. That isn't the
right way, since what you really want is to split after all the host
archives are combined, but it doesn't work that way currently.

Also, if the output is a device (eg: tape), then split is not run.
Split is only enabled when the output is a directory, and that
directory is used to write the multiple files output by split.

So currently there isn't any support for handling limits on media
size for archives. BackupPC_archiveHost is intended to be a
starting point that can be locally customized, and the default
version will likely improve over time.

Question Two: Is there a way of using the -M (--multiple volumes) switch in
tar to accomplish this in the existing code? If it is not too difficult to
do, then it helps me greatly in question three.

Question Three: If the answer to two is yes, then could we also use the tar
switch "--new-volume-script file" ( Run a script at the end of each archive
volume)? This would allow me to automatically have the tape changer move to
the next tape and tar happily continues... I have been using the perl
module juke.pm with the program mtx to do this with some rather kludgy
scripts of my own, but it isn't a prime time solution.

As Oliver Freyd responded, these options are not supported, since
it is really BackupPC_tarCreate that builds the tar archive from
the backup, not GNU tar.

Question four: If the above will work, then might there be a way of setting
this to go on a sensible schedule, with a message to the backup operator
when the Data Storage Elements are full and need to be changed? My boss
wants this in case I get hit by a truck tomorrow on the way to work.

This is currently beyond the scope of BackupPC. I don't want to
reinvent the good work already in Amanda, Bacula, afbackup etc.
However, the heavy use of hardlinks in the pool makes tape backup
difficult, so a good, integrated solution would be a nice addition.

Craig


-------------------------------------------------------
This SF.Net email is sponsored by the new InstallShield X.
From Windows to Linux, servers to mobile, InstallShield X is the one
installation-authoring solution that does it all. Learn more and
evaluate today! http://www.installshield.com/Dev2Dev/0504
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

Oliver Freyd writes:

I'm cuurently evaluating BackupPC for our firm and for now,
I like it a lot, so it's nice to hear it's so successful for you.

We already have a tape backup system in place, with a 100GB
(uncompressed) tape drive, so currently I'm playing around with the
tape backup function of 2.1beta.

I have a ~300GB tape changer, and ATL L200 to be exact, and I have connected
it to the BackupPC and did get it to archive a small system to tape.
However if I tried to use the compression it failed, saying "2004-06-02
16:58:02 Archive failed (Error: /usr/local//bin/BackupPC_tarCreate, compress
or split failed)" None gets me to tape, but I have to be careful not to
exceed one DLT-8000's capacity or I hang things....

I tried putting the archives to disk first, as for now the BackupPC
machine doesn't have a tape drive.
So I set
$Conf{ArchiveDest}='/srv/data/archives'
Strangely, setting this in the pc/'archivehost'/config.pl didn't work.

This is a bug that Pete Wenzel sent a patch for. The fix will be
in the next version.

From reading the comments in BackupPC_archiveHost, it seems it doesn't
use split when archiving to a device (tape).
It uses BackupPC_tarCreate to extract the data from a certain host and
backup number and convert it into a tar archive. That is piped to
stdout, and into the tape device, or if you specified a directory
as $Conf{ArchiveDest}, pipes it through split. It seems split doesn't
like sizes over 2GB, then the archive fails. Don't know why it fails
with gzip, though...

Yes, split is only used when the target is a directory (not a tape
or other device).

I haven't tested split for files bigger than 2GB. I would hope it
would work. Is it a problem with the command-line argument overflowing
or split itself?

That's one of my questions, too. I was wondering if one could call
BackupPC_archive from the command line, instead of using the web
interface. Thus, one could use cron to automate it.

You can schedule things using cron by sending BackupPC a message.
This is better than running the underlying commands like BackupPC_dump
and BackupPC_archive directly.

To run an archive automatically you need to create a request file that
contains all the information about what to archive (eg: hosts, output
directory etc). The ArchiveInfo.NNN file saved in the archive client is
an example of an archive request file. The archive request file should
be stored in the /data/BackupPC/pc/ARCHIVE_HOST directory.

You can start the archive by doing:

/path/to/bin/BackupPC_serverMesg archive USER ARCHIVE_HOST REQ_FILENAME

So a script with a few lines of code run by cron would allow you
to automate archives.

Another thing I was thinking about is some kind of tape management, like
amanda does. Like giving labels to each tape, checking if the right tape
is in the drive before starting to write (prevent erasing the wrong
tape), storing some catalogue of the tapes.

Contributions are welcome .

Craig


-------------------------------------------------------
This SF.Net email is sponsored by the new InstallShield X.
From Windows to Linux, servers to mobile, InstallShield X is the one
installation-authoring solution that does it all. Learn more and
evaluate today! http://www.installshield.com/Dev2Dev/0504
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

Hello,

first of all, thanks for your reply and keep up the good work,
BackupPC is great!

That's one of my questions, too. I was wondering if one could call
BackupPC_archive from the command line, instead of using the web
interface. Thus, one could use cron to automate it.


You can schedule things using cron by sending BackupPC a message.
This is better than running the underlying commands like BackupPC_dump
and BackupPC_archive directly.

To run an archive automatically you need to create a request file that
contains all the information about what to archive (eg: hosts, output
directory etc). The ArchiveInfo.NNN file saved in the archive client is
an example of an archive request file. The archive request file should
be stored in the /data/BackupPC/pc/ARCHIVE_HOST directory.

You can start the archive by doing:

/path/to/bin/BackupPC_serverMesg archive USER ARCHIVE_HOST REQ_FILENAME

So a script with a few lines of code run by cron would allow you
to automate archives.

I'm just trying this out, saved a RequestFile created with the web
interface, and called the serverMesg command line. It's currently
running the backup. So that's ok.

But in the request file, in "BackupList" you have to put the number of
the backup you want to archive. What do I do if I always want the last
available backup? Reading tarCreate, I found I could use number -1, as
it says in the comment "count from the end" (reminds me of python
arrays). So that seems to work. Only drawback I can think of:
If this '-1' gets stored in the ArchiveInfo file, later we don't know
which archive is on the tape, only that it was the 'last' at that time.
Maybe the '-1' in the RequestFile should be parsed and replaced be the
'real' number of the latest backup before calling the tarCreate.

Well, of course I could re-create the requestfile everytime, but
a static requestfile would be more practical...

So thats all for now,

I'll try some other things,

regards,

Oliver




Another thing I was thinking about is some kind of tape management, like
amanda does. Like giving labels to each tape, checking if the right tape
is in the drive before starting to write (prevent erasing the wrong
tape), storing some catalogue of the tapes.


Contributions are welcome .

Craig