Backup Central

I'm running two machines, each doing separate backups. One machine is a P3,
700Mz with 1gig of RAM, and it's sailing through 130 client machines very
nicely backing up small sets. Basically just machine configs, password
files, sql DB's, etc.

My second machine is doing my client data backups. This is on a 1.8Gz AMD
with 1.5gig of RAM and a 1.2TB raid 5 array. This one isn't performing very
well at all. The files it's backing up can total up to 100 gig per client
(Range from 5gig to 100gig) and I'm seeing a lot of failures due to ALRM or
PIPE. When it doesn't fail, the BackupPC_Link script does terrible things
to my memory. I am swapping to disk and dragging the machine down to a
crawl. What can I do to help this problem? Would setting compression to
'0' help me out at all, or is there a way to turn pooling off? I could get
away without pooling files if it would help.

Thanks in advance for your advice.

Corey Baldwin
---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.708 / Virus Database: 464 - Release Date: 6/18/2004



-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 -
digital self defense, top technical experts, no vendor pitches,
unmatched networking opportunities. Visit www.blackhat.com
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

On 07/08 01:32 , Corey Baldwin wrote:
My second machine is doing my client data backups. This is on a 1.8Gz AMD
with 1.5gig of RAM and a 1.2TB raid 5 array. This one isn't performing very
well at all. The files it's backing up can total up to 100 gig per client
(Range from 5gig to 100gig) and I'm seeing a lot of failures due to ALRM or
PIPE.

do you have your timeout set high enough, such that it doesn't time out when
backing up this machine?

I had some machines being backed up over a WAN, which required >2day
timeouts, before the backups would succeed.

--
Carl Soderstrom
Systems Administrator
Real-Time Enterprises
www.real-time.com


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 -
digital self defense, top technical experts, no vendor pitches,
unmatched networking opportunities. Visit www.blackhat.com
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



Corey Baldwin wrote:
| well at all. The files it's backing up can total up to 100 gig per client

How many clients at a time? Try running one at a time?

| crawl. What can I do to help this problem? Would setting compression to
| '0' help me out at all, or is there a way to turn pooling off? I
could get
| away without pooling files if it would help.

(asbestos underpants in place) Compression and pooling, done properly,
shouldn't use significantly more RAM than without. That said, why not
try turning off compression first? I think it only applies to new
backups, so you can run one new one without compression to test. Also,
you can manually delete that backup later if it's wasting a lot of space.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQO4aHngHNmZLgCUhAQJT4g//QzNPKHGaHuNH4O7RfbbcsrEkZYTwnhBy
uMUeA54oHijp8f7o8bXnH1OSxHlx4JogGtxnYqKrmsBQ8S/5K6ikaH0l8zHMDiIB
juQ8hM2RpR0HLoatA2P+rlT8wjtxgFKSoR4OJsQEwFO3JvQCHfLr1N1WtEylIEaU
5AcCwgJK5QYQDlKODrO31N6ljLtRnnjFXq7a80cbapvrV6SYt1liP0CgjX+XG+1x
lpM5wMym9J1iPoNmdn1qu/KuZgU2Tc4ZqgEX6MrrGF13v6sKXBBpztD7vqnelghY
Ja4CndL3U2Sc5M0086cgsSp3V2a9+YmeNZsjLEplvsdSZFW5TWJpY9z6u2F/8kbK
1LKnYPP9mH+Sgl2biNorb4LUO4ux1uITjD/No66dRZWBmCU+zCFOy+UprrNVzsV8
oz3zQZXEMgfNe9aAkEjcUq1lEsUkktGEjK9NCKOqfUY8KWdKwSpwg9Jpoc3m5gAa
+yxcOrtRE/fw0X4k7fVHlzgpZNxyl2ctAZ2EXnLgAb2OTmsC2jDdpsBgrZUoa6ry
ibAbxcK6ljiXsOHUabg0U6OT0VIU9VorUXBIHmlF/7OQXjkQb9YEG7mPG6MnEEVQ
tQpdSb6rtBKRPe3LaAVcOzhy6U4gNl6D4QF7Wd0HV4z+NWsyI07BpO9KZYG1VhLh
N0JsvYiPcqs=
=xZjd
-----END PGP SIGNATURE-----


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 -
digital self defense, top technical experts, no vendor pitches,
unmatched networking opportunities. Visit www.blackhat.com
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

Well, I think I've got it fixed. Reading through the mailling list
archives, I found an SSH switch -x and compared it to my rsyncclientcmd. I
didn't have this, so added it. I also broke the raid on my system drives, I
think this was slowing things down when BackupPC_link was running my memory
into swap (swap drive was raid1.) I also raide my timeout as per Carl's
suggestion.

All or any of these changes has my backup machine humming right along.

I do have one issue though, I was going to wait until this backup finished
to see the outcome, but since I'm mailing now, I'll bring it up now. I have
an exclude in a per-pc config.pl file. It is not excluding at the moment.
The full machine backup is 250gig, and I excluded two large directories (All
linux, all rsync over SSH) so I could run seperate backups on them using an
alias. The backup is still running right now, but I can see the directory I
excluded in the 'new' directory under the machines backup dir. rsync should
not be transferring this directory as far as I know. I have it listed as
'/path/to/directory' in the exclude line, should I have
'/path/to/directory/*'?

Thanks in advance,
Corey Baldwin

-----Original Message-----
From: Carl Wilhelm Soderstrom [mailto:chrome < at > real-time.com]
Sent: Friday, July 09, 2004 12:51 PM
To: Corey Baldwin
Subject: Re: [BackupPC-users] Any ideas on how to save memory usage?


On 07/08 05:51 , Corey Baldwin wrote:
I'm getting off track here. I can tell how fast I'm transferring data.
That's not necessarily the problem. I've got all day to wait on a
transfer.
My problem is that the backup is erroring out, and I'm not entirely sure
why.

one of the things that causes the errors that you see; is Backuppc timing
out on the connection. the data might be flowing fine; but the timeout
period allocated for that connection might be exceeded.

This is more of a memory issue. I'm swapping to the drive, and I don't
want
to have to add more memory.

unfortunately, rsync can be a memory hog.
my advice is, just bite the bullet and buy more memory. I'm planning 1.5GB+
when I get my new backup server built here. (probably 2GB).

sorry I don't have better news for you; tho some of the advice on the
mailing list (split up your backups) might be worthwhile).

--
Carl Soderstrom
Systems Administrator
Real-Time Enterprises
www.real-time.com

---
Incoming mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.516 / Virus Database: 313 - Release Date: 9/1/2003

---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.516 / Virus Database: 313 - Release Date: 9/1/2003



-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 -
digital self defense, top technical experts, no vendor pitches,
unmatched networking opportunities. Visit www.blackhat.com
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

Corey Baldwin writes:

I do have one issue though, I was going to wait until this backup finished
to see the outcome, but since I'm mailing now, I'll bring it up now. I have
an exclude in a per-pc config.pl file. It is not excluding at the moment.
The full machine backup is 250gig, and I excluded two large directories (All
linux, all rsync over SSH) so I could run seperate backups on them using an
alias. The backup is still running right now, but I can see the directory I
excluded in the 'new' directory under the machines backup dir. rsync should
not be transferring this directory as far as I know. I have it listed as
'/path/to/directory' in the exclude line, should I have
'/path/to/directory/*'?

Hmmm. The exclude path is relative to the share. So if your share
is / then /path/to/directory should work.

Could you email the rsync command-line from the top of the XferLOG
file?

Craig


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 -
digital self defense, top technical experts, no vendor pitches,
unmatched networking opportunities. Visit www.blackhat.com
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/