Backup Central

Hi,

there is a regular discussion on how to backup/move/copy the backuppc
pool. Did anyone try to backup the pool with bacula?

I need to expand the raid volume where the pool is stored (Arecac RAID
controller). Doing this without backup is a bit frightening (I didn't
use LVM for the filesystem).

I also use bacula to backup parts of the data. Quoting the manual:

hardlinks=yes--no When enabled (default), this directive will cause hard
links to be backed up. However, the File daemon keeps track of hard
linked files and will backup the data only once. The process of keeping
track of the hard links can be quite expensive if you have lots of them
(tens of thousands or more). This doesn't occur on normal Unix sys-
tems, but if you use a program like BackupPC, it can create hundreds
of thousands, or even millions of hard links. Backups become very
long and the File daemon will consume a lot of CPU power checking
hard links. In such a case, set hardlinks=no and hard links will not
be backed up. Note, using this option will most likely backup more
data and on a restore the file system will not be restored identically
to the original.


Has anyone tried to backup the pool with bacula?

Ralf

------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables
unlimited royalty-free distribution of the report engine
for externally facing server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Hi,

there is a regular discussion on how to backup/move/copy the
backuppc pool. Did anyone try to backup the pool with bacula?

Hello there...

I don't know about bacula, but would like myself also to get a backup of
the BackupPC server: anybody got some suggestions and practical
examples?

Thanks,
F.

------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables
unlimited royalty-free distribution of the report engine
for externally facing server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Boniforti Flavio wrote:
Hi,

there is a regular discussion on how to backup/move/copy the
backuppc pool. Did anyone try to backup the pool with bacula?


Hello there...

I don't know about bacula, but would like myself also to get a backup of
the BackupPC server: anybody got some suggestions and practical
examples?

Thanks,
F.


How often do you want to backup the server? What about using rsync to
backup the pool? I want to mirror my primary backup server to another
system daily so I can switch to my backup server quickly. I was going
to use rsync for this.
------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables
unlimited royalty-free distribution of the report engine
for externally facing server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

------------------------------------------------------------------------




/
/








------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables
unlimited royalty-free distribution of the report engine
for externally facing server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Boniforti Flavio wrote:
Hi,

there is a regular discussion on how to backup/move/copy the
backuppc pool. Did anyone try to backup the pool with bacula?

Hello there...

I don't know about bacula, but would like myself also to get a backup of
the BackupPC server: anybody got some suggestions and practical
examples?

I think the only way to do it at a reasonable speed is to unmount the
partition where the archive is stored and image-copy it to an
equal-sized partition. Or, if you created it as a RAID1 with a missing
mirror, you can add/sync a mirror while mounted - but you still won't
have any performance if you try to do anything else for the duration of
the copy and you need to unmount momentarily to get a clean filesystem
as you fail/remove the mirror.

Some people are copying smaller archives with rsync (-aH), and the
newest version of rsync is supposed to handle the hardlinks more
efficiently. You can always try that and see how long it takes.

--
Les Mikesell
lesmikesell < at > gmail.com

------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables
unlimited royalty-free distribution of the report engine
for externally facing server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On 05/19 05:51 , Boniforti Flavio wrote:
I don't know about bacula, but would like myself also to get a backup of
the BackupPC server: anybody got some suggestions and practical
examples?

I have one system where I do backups of backuppc to tape for disaster
recover. Here's the system I use:
- Stop backuppc to quiesce the filesystem. LVM snapshots are not sufficient
for this, because the disk load gets too high, data flow rate gets too low,
and the tape starts to 'shoeshine'.
- Dump /var/lib/backuppc to tape with 'tar'. I have a 500GB tape drive,
which is sufficient to store the backuppc data from that machine.
- In case you're curious, the tar command I use is:
tar -cv --totals --exclude-from=$EXCLUDEFILE -f /dev/st0 /var/lib/backuppc/
2>&1 > $LOGFILE
- Restart backuppc.

--
Carl Soderstrom
Systems Administrator
Real-Time Enterprises
www.real-time.com

------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables
unlimited royalty-free distribution of the report engine
for externally facing server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On 05/19 12:04 , Tim Cole wrote:
How often do you want to backup the server? What about using rsync to
backup the pool? I want to mirror my primary backup server to another
system daily so I can switch to my backup server quickly. I was going
to use rsync for this.

Rsync memory requirements have historically been too high for this.
I tried rsync'ing 100GB on a machine with 512MB RAM and about the same swap;
and it crushed the box to the point where I had to power-cycle it.

Best thing to do is set up a duplicate backuppc server that independently
backs up the hosts you want to have a redundant backup for.

--
Carl Soderstrom
Systems Administrator
Real-Time Enterprises
www.real-time.com

------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables
unlimited royalty-free distribution of the report engine
for externally facing server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Carl Wilhelm Soderstrom wrote:
On 05/19 05:51 , Boniforti Flavio wrote:
I don't know about bacula, but would like myself also to get a backup of
the BackupPC server: anybody got some suggestions and practical
examples?

I have one system where I do backups of backuppc to tape for disaster
recover. Here's the system I use:
- Stop backuppc to quiesce the filesystem. LVM snapshots are not sufficient
for this, because the disk load gets too high, data flow rate gets too low,
and the tape starts to 'shoeshine'.
- Dump /var/lib/backuppc to tape with 'tar'. I have a 500GB tape drive,
which is sufficient to store the backuppc data from that machine.
- In case you're curious, the tar command I use is:
tar -cv --totals --exclude-from=$EXCLUDEFILE -f /dev/st0 /var/lib/backuppc/
2>&1 > $LOGFILE
- Restart backuppc.

Have you ever restored one of these tapes, and if so, how long did it
take? As a wild guess, I'd expect a couple of days where an image copy
would be an hour or two.

--
Les Mikesell
lesmikesell < at > gmail.com

------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables
unlimited royalty-free distribution of the report engine
for externally facing server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Carl Wilhelm Soderstrom wrote:
On 05/19 12:04 , Tim Cole wrote:
How often do you want to backup the server? What about using rsync to
backup the pool? I want to mirror my primary backup server to another
system daily so I can switch to my backup server quickly. I was going
to use rsync for this.

Rsync memory requirements have historically been too high for this.
I tried rsync'ing 100GB on a machine with 512MB RAM and about the same swap;
and it crushed the box to the point where I had to power-cycle it.

Best thing to do is set up a duplicate backuppc server that independently
backs up the hosts you want to have a redundant backup for.

These days, I'm not sure 512MB RAM and 'server' belong in the same
sentence, but it is still hard to deal with the disk head motion needed
to traverse and recreate all those hardlinks splattered more or less
randomly across the disk.

--
Les Mikesell
lesmikesell < at > gmail.com

------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables
unlimited royalty-free distribution of the report engine
for externally facing server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On 05/19 11:35 , Les Mikesell wrote:
Have you ever restored one of these tapes, and if so, how long did it
take? As a wild guess, I'd expect a couple of days where an image copy
would be an hour or two.

I think it's about 8 hours to create the tape (whereas when using LVM
snapshots it took almost 2 days). I've scanned and recovered files from the
tape, and it takes a few hours, depending on where on the tape the file
is. I've never convinced the company to do a full test restore (we're
contractors, and doing that test would cost more money); but I have
scanned the tapes and checked their integrity, and that takes a healthy long time.

An image copy would indeed be much faster. I prefer a file-level backup
rather that filesystem-level backup simply because recovery in case of
corruption is much better.

--
Carl Soderstrom
Systems Administrator
Real-Time Enterprises
www.real-time.com

------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables
unlimited royalty-free distribution of the report engine
for externally facing server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Hi,

Carl Wilhelm Soderstrom wrote on 2009-05-19 11:54:16 -0500 [Re: [BackupPC-users] backup the backuppc pool with bacula]:
On 05/19 11:35 , Les Mikesell wrote:
Have you ever restored one of these tapes, and if so, how long did it
take? As a wild guess, I'd expect a couple of days where an image copy
would be an hour or two.

I think it's about 8 hours to create the tape (whereas when using LVM
snapshots it took almost 2 days). I've scanned and recovered files from the
tape, and it takes a few hours, depending on where on the tape the file
is. I've never convinced the company to do a full test restore (we're
contractors, and doing that test would cost more money); but I have
scanned the tapes and checked their integrity, and that takes a healthy long time.

An image copy would indeed be much faster. I prefer a file-level backup
rather that filesystem-level backup simply because recovery in case of
corruption is much better.

it really depends on what you want to do in case of disaster.

1.) Restore the pool
Forget it.

2.) Restore files from previous backups or even single whole backups
No problem.
[Though I have no idea how tar handles restoring a subset of the archive
that is made up of hardlinks to various files outside this subset. This
may well prove not to work or require insane amounts of tape seeks - if
tar seeks on its input at all.]

With an image level copy it's the other way around, but I agree that I
wouldn't trust tape media very far. Has anyone invented tape-RAID-6 yet?

Regards,
Holger

------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables
unlimited royalty-free distribution of the report engine
for externally facing server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Hi,

Les Mikesell wrote on 2009-05-19 11:12:25 -0500 [Re: [BackupPC-users] backup the backuppc pool with bacula]:
[...] the newest version of rsync is supposed to handle the hardlinks more
efficiently.

reason suggests that this is an urban myth. The newest version of rsync
handles *large file lists* better, not *hardlinks*. To handle hardlinks
better, you would almost certainly need to create a temporary file (which
can easily be several *GB* in size in our cases). I somehow doubt any general
purpose tool would dare do that (let alone find a spot where it can - my /tmp
simply isn't large enough). The issue is that the temporary file will either
be very large or unneeded, and an algorithm able to handle extreme numbers of
hardlinks will probably be slow in the overwhelming majority of cases with
very few hardlinks.

You can always try that and see how long it takes.

That you can. Just remember that your space usage (and hardlink counts) will
grow over time. How does the time it takes grow in proportion to space and/or
hardlink counts? At some random point it will stop working. You may never
reach that point. But what if you do? Can you simply find another solution
then? How long can you keep your pool offline for the copy process? At what
point do you abort the copy process? How can you monitor its progress?

Just some things to think about ...

Regards,
Holger

------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables
unlimited royalty-free distribution of the report engine
for externally facing server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On Tue, May 19, 2009 at 11:35:36AM -0500, Les Mikesell wrote:
Have you ever restored one of these tapes, and if so, how long did it
take? As a wild guess, I'd expect a couple of days where an image copy
would be an hour or two.

I did this with solaris ufsdump/ufsrestore once.

Making the tape took one or two hours.

I gave up and cancelled the restore after 24 hours.

danno

------------------------------------------------------------------------------
Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, &
iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian
Group, R/GA, & Big Spaceship. http://www.creativitycat.com
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On Tue, May 19, 2009 at 12:27 PM, Carl Wilhelm Soderstrom
<chrome < at > real-time.com> wrote:
On 05/19 12:04 , Tim Cole wrote:
How often do you want to backup the server?  What about using rsync to
backup the pool?  I want to mirror my primary backup server to another
system daily so I can switch to my backup server quickly.  I was going
to use rsync for this.

Rsync memory requirements have historically been too high for this.
I tried rsync'ing 100GB on a machine with 512MB RAM and about the same swap;
and it crushed the box to the point where I had to power-cycle it.

Best thing to do is set up a duplicate backuppc server that independently
backs up the hosts you want to have a redundant backup for.


Try Rsync v3, it has much lower memory requirements since it builds
the file list incrementally. I used it at one company to do nightly
syncs of their ~4TB backuppc pool offsite.

------------------------------------------------------------------------------
Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, &
iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian
Group, R/GA, & Big Spaceship. http://www.creativitycat.com
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Hi,

Rob Terhaar wrote on 2009-05-21 13:01:58 -0400 [Re: [BackupPC-users] backup the backuppc pool with bacula]:
[...]
Try Rsync v3, it has much lower memory requirements since it builds
the file list incrementally.

by all means, try it. But it's not the file list that is the specific problem
of BackupPC.

I used it at one company to do nightly
syncs of their ~4TB backuppc pool offsite.

It's still a matter of file count (used inodes, to be exact), not pool storage
size. rsync V3 may perform significantly better if you have many links to
comparatively few inodes, but if you have many inodes (for some unknown value
of "many"), I am still convinced that you will hit a problem. Feel free to
convince me otherwise, but "works for me" is unlikely to succeed .

Regards,
Holger

------------------------------------------------------------------------------
Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, &
iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian
Group, R/GA, & Big Spaceship. http://www.creativitycat.com
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Good point Holger- I don't have hardlink counts or stats, but it
should only take about 10mns of work to download/compile rsync3, and
run a benchmark on your pool

On 5/21/09, Holger Parplies <wbppc < at > parplies.de> wrote:
Hi,

Rob Terhaar wrote on 2009-05-21 13:01:58 -0400 [Re: [BackupPC-users] backup
the backuppc pool with bacula]:
[...]
Try Rsync v3, it has much lower memory requirements since it builds
the file list incrementally.

by all means, try it. But it's not the file list that is the specific
problem
of BackupPC.

I used it at one company to do nightly
syncs of their ~4TB backuppc pool offsite.

It's still a matter of file count (used inodes, to be exact), not pool
storage
size. rsync V3 may perform significantly better if you have many links to
comparatively few inodes, but if you have many inodes (for some unknown
value
of "many"), I am still convinced that you will hit a problem. Feel free to
convince me otherwise, but "works for me" is unlikely to succeed .

Regards,
Holger


--
Sent from my mobile device

------------------------------------------------------------------------------
Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, &
iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian
Group, R/GA, & Big Spaceship. http://www.creativitycat.com
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/