Am Montag 26 April 2004 11:59 schrieb Geoffrey McRae:
Hi,
Hi,
First I would like to thank everyone that has contributed to BackupPC for
such a good program.
Thank Craig... He has done all the amazing code...
Ok, now the situation is that I am trying to implement a large system
backup solution using external USB2.0 caddys, the system I have devised
will currently works as follows:
=A0 2x External USB caddys with 160GB HDDs
=A0 Internal 160GB HDD mounted to /usr/share/backuppc/pc
=A0 BackupPC set up to backup, there is around 20 machines
This works great, but what I need is that on completion of all backups I
need to execute a script to mount the USB caddy, copy the data to it and
unmount it, so it can be taken offsite. I can write the script to do this
but I dont know how to make it run when the whole backup process has
finished.=20
I think it is not very lucky to try to simply "copy" the files... Not becau=
se=20
it would miss files, just because it's damn slow. On a 80GB HDD I needed 2=
=20
days on a 500Mhz System!
Even a "rm" isn't really faster. That has nothing to do with the data itsel=
f,=20
it is the filesystem that takes so long to create millions of files and=20
links.
On my System it also crashed my pc, because the cp process needs to generat=
e=20
filelistings and these took all my ram, so that linux started to kill=20
processes (even very important ones!). So that could also get a problem...=
=20
Because I dont investigate that a lot I don't really know if my analysis th=
at=20
the filelisting took all ram is correct. But it is very likely.
I would think about implementing that as a raid1 setup... So that the=20
stationary hdd is always part of the raid1 and one of the two other usb dis=
ks=20
will be hot-added and synced in background.
If you need to change the disks, simply remove it from the raid and plug in=
=20
the second usb disk and the resync starts again (one line in the console to=
=20
start a resync)...
If you always have one of the two usb disks in your system, then you have m=
ore=20
or less always a raid1 running (just in the syncing time your raid is=20
incomplete), so a hard disk failuire is uncritical for you... even if there=
=20
wasn't running the seconds raid disk at the moment of the crash you will ha=
ve=20
it running again very soon. just plugin a new stationary disk and one of t=
he=20
two usb disks an then the new stationary disk gets sync...
Another advantage is that it is quite unimportant when you change the usb=20
disks, because the resync will also work if you do that when backuppc is=20
storing new backups. (raid1 sync is clever enough to handle a sync even whe=
n=20
you write new data to your raid1 in that time). Even when you remove a disk=
=20
when backuppc runs, it is not a real problem, becuase the pool should be=20
allright even on the removed usb disk, even with uncompleted backups in it.=
=20
Just the linking process could be critical, but I don't think so... Please=
=20
somebody correct my If I am wrong with that...
A raid1 sync on a 160GB Harddisc should not take longer than 1h. (On a 120G=
B=20
IBM IDE Drive it takes 45 Min). The raid1 sync doesn't need to create every=
=20
file, instead it copies data like "dd" and so is even very fast on a filled=
=20
backuppc pool... Compare that to a full "cp" for the whole disk! You could=
=20
save a lot of time and even have the advantages of a raid1!
After having a look at the BackupPC source I have come to the=20
conclusion that it is not easy to hack in because it forks a backup
process for every pc (I could be wrong).
If you do that not with a raid1 and you need that copy commands in a time w=
hen=20
backuppc does nothing, than it would be ideal to do that in the:=20
"BackuPC_nightly" command.
There is no possibility to do that in a easy way via a config file (please=
=20
correct me, if I am wrong)... But that could be a very useful config=20
parameter for the upcoming backuppc 2.1... But I think it is not implemente=
d=20
yet. So for now you would need to hack that in.
If you really need that, than maybe the end of the BackupPC_nightly command=
=20
would be ideal for adding your stuff. Because when BackupPC runs the=20
backuppc_nightly stuff, than no backup is running. So when the nightly run=
=20
itself is completed, you "cp" stuff would fit here. With this method you ca=
n=20
be shure, that no backup is running while you disk cp process runs.
But remember, on a filled pool with about 160GB data it will takes very lon=
g=20
(I think even on a 1.8Ghz System over one day!)... So I wouldn't use this=20
method!
FYI:
Also I have setup a loop back device to encrypt the data on the USB caddy
with Blowfish encryption so that if the device is stolen the data is not
accessable, this works great (had to move to 2.6.5 kernel for this).
fstab entry: =A0/dev/sda1 =A0 =A0/mnt/hdd_usbcaddy =A0 =A0reiserfs =A0
defaults,noauto,loop,encryption=3Dblowfish =A0 =A00 =A0 =A00
Nice... You can also add this feature to the stationary disk and then use t=
he=20
raid1 setup I discribed above... So you USB Drives have only encrypted data=
,=20
as well as the stationary harddrive (because it's a 1:1 copy)...
My perl experiance is very limited and I am unable to figure this out for
myself, if you could provide any help it would be much appreciated,
Thankyou.
Hope I could give you a good alternative to cp... Have you ever messured th=
e=20
time you need to copy a full 160GB pool filled with typical backuppc data?
If you are interessted in the raid1 solution, please take a look at the=20
raid/mdadm howto's for linux. The could help a lot in the beginning.
I have never build a raid1-setup with removeable storage (like described=20
above) myself, but I planned a new backuppc system for out customers and=20
that's why I was thinking of an alternative to cp... We also want to use=20
removeable disks, so that we have a disk for disaster recovery. So I was=20
thinking of an RAID1 + LVM Setup on a stationary and remove disk.
I had once the situation that I needed to move a pool from on two another d=
isk=20
(because the old disk was to small) and that takes so long that I sweared t=
o=20
myself only to use LVM. With a LVM you can add and remove additional disks =
to=20
you "backuppc disk". If you combine that with a RAID1 it is a very nice=20
setup... Easy to expand and also has a one-drive fault-tolerance. So I woul=
d=20
suggest also take a lot at LVMs.
=20
Geoffrey McRae
gnif < at > spacevs.com
BackupPC Server Specs:
=A0 Celeron 1.8 (A bit overkill don't you think
)
not really... Maybe you need to backup more systems via backuppc in the nea=
r=20
future... than you need a good system. With a fast system like this you can=
=20
think of setting the compress parameter higher than the default 3 (in=20
backuppcs config file). This would save some percent of your disk storage.
=A0 Gigabyte MB with USB2.0
=A0 256MB Ram
=A0 512MB Swap
=A0 Debian 3.0r2 - Testing sources
Just for my information? You mean you use Debian 3.0r2 (Woody) and took=20
BackupPC from the testing/sarge sources?
=A0 Kernel 2.6.5 - Optomised for the machine and usb-storage
nice... because a 2.6 should have a faster I/O throughput than a 2.4 kernel.
=2D-Ralph
-------------------------------------------------------
This SF.net email is sponsored by: The Robotic Monkeys at ThinkGeek
For a limited time only, get FREE Ground shipping on all orders of $35
or more. Hurry up and shop folks, this offer expires April 30th!
http://www.thinkgeek.com/freeshipping/?cpg=12297
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/
