Backup Central

Hi,

I have backuppc running for a client, and been facing constant problem lately on it.

The server running the backuppc would go under high load (19 - 25 load on a quad core xeon, 8gb ram) and i can see multiple /usr/local/BackupPC/bin/BackupPC_dump processes. When i try to kill the processes, it doesn't work. Trying to reboot via Ctrl+Alt+Del doesn't work, the only thing which works is hard reboot.

In the above setup, i am using backuppc with DeltaCopy. I am assuming that if the clients shutdown in between of a backup, backuppc would automatically after a while disconnect those sessions. Please correct me if i am wrong.

Whenever there are a few clients being backed up, i see a lot of cpu load (it cud probably be due to disk io as well)

Is there anyone else facing similar issues ? Any help on this would be highly appreciated.

--
Thanks and Regards,

Anand Gupta

On 3/4/2010 4:28 PM, Anand Gupta wrote:
Hi,

I have backuppc running for a client, and been facing constant problem
lately on it.

The server running the backuppc would go under high load (19 - 25 load
on a quad core xeon, 8gb ram) and i can see multiple
/usr/local/BackupPC/bin/BackupPC_dump processes. When i try to kill the
processes, it doesn't work. Trying to reboot via Ctrl+Alt+Del doesn't
work, the only thing which works is hard reboot.

In the above setup, i am using backuppc with DeltaCopy. I am assuming
that if the clients shutdown in between of a backup, backuppc would
automatically after a while disconnect those sessions. Please correct me
if i am wrong.

Whenever there are a few clients being backed up, i see a lot of cpu
load (it cud probably be due to disk io as well)

Is there anyone else facing similar issues ? Any help on this would be
highly appreciated.

How much ram does the server have? It sounds like the rsync directory
trees might be filling memory and making the server swap to the point
that you give up before it is able to respond. It might help to take
$Conf{MaxBackups} down to 1 or 2.

--
Les Mikesell
lesmikesell < at > gmail.com

------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Hi Les,

Thanks for the reply.

As i mentioned in my mail, the server has 8gb ram. It has a adaptec 3405 8 port raid card with sata drives ( 8 x 750gb - raid 5).

Right now the maxbackups is set to 8 (i did this thinking the server was powerful enough to handle that many number of clients at one shot). Another reason being, there are like 100+ clients in the network, so if the number of too low, then all clients might not get backed up. Will try the setting and see if anything changes.
--
Thanks and Regards,

Anand Gupta


On Fri, Mar 5, 2010 at 4:17 AM, Les Mikesell <lesmikesell < at > gmail.com ([email]lesmikesell < at > gmail.com[/email])> wrote:

On 3/4/2010 4:28 PM, Anand Gupta wrote:
Hi,

I have backuppc running for a client, and been facing constant problem
lately on it.

The server running the backuppc would go under high load (19 - 25 load
on a quad core xeon, 8gb ram) and i can see multiple
/usr/local/BackupPC/bin/BackupPC_dump processes. When i try to kill the
processes, it doesn't work. Trying to reboot via Ctrl+Alt+Del doesn't
work, the only thing which works is hard reboot.

In the above setup, i am using backuppc with DeltaCopy. I am assuming
that if the clients shutdown in between of a backup, backuppc would
automatically after a while disconnect those sessions. Please correct me
if i am wrong.

Whenever there are a few clients being backed up, i see a lot of cpu
load (it cud probably be due to disk io as well)

Is there anyone else facing similar issues ? Any help on this would be
highly appreciated.



How much ram does the server have?  It sounds like the rsync directory
trees might be filling memory and making the server swap to the point
that you give up before it is able to respond.  It might help to take
$Conf{MaxBackups} down to 1 or 2.

--
  Les Mikesell
   lesmikesell < at > gmail.com ([email]lesmikesell < at > gmail.com[/email])

------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net ([email]BackupPC-users < at > lists.sourceforge.net[/email])
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On 03/05 04:25 , Anand Gupta wrote:
Right now the maxbackups is set to 8 (i did this thinking the server was
powerful enough to handle that many number of clients at one shot). Another
reason being, there are like 100+ clients in the network, so if the number
of too low, then all clients might not get backed up. Will try the setting
and see if anything changes.

Disk contention is the killer; and even on a powerful machine I've not been
able to efficiently run more than 3-4 backups at a time (presuming they're local
machines & fast on their end as well).

It's hard to judge; but basically if there are a lot of processes waiting
for I/O (a 'D' state in 'top'); try cutting down the number of concurrent
backups. You'll have to judge for yourself what the best number for you is.
It may be that things work fastest when there's a certain amount of disk
contention; but no more and no less.

--
Carl Soderstrom
Systems Administrator
Real-Time Enterprises
www.real-time.com

------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

It's hard to judge; but basically if there are a lot of processes
waiting
for I/O (a 'D' state in 'top'); try cutting down the number of
concurrent
backups. You'll have to judge for yourself what the best number for you
is.
It may be that things work fastest when there's a certain amount of disk
contention; but no more and no less.

Also - you need a good filesystem to handle lots (or even not so many) of
backups. I reently switched from EXT3 to EXT4 and saw on order of magnitude
(I kid you not, 10+ hours to 1) reduction in the backup time and system
load. Unfortunately, I think this introduced some problems in the RHEL5
ext4 code so I also switched from 32-bit RHEL5 to 64-bit -- that seems to
have cleared up the problems.

-Josh


--
--------------------------------------------------------
Joshua Malone Systems Administrator
(jmalone < at > nrao.edu) NRAO Charlottesville
434-296-0263 www.cv.nrao.edu
434-249-5699 (mobile)
BOFH excuse #202:

kernel panic: write-only-memory (/dev/wom0) capacity
exceeded.
--------------------------------------------------------

------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

If you are using EXT3 or XFS then I suggest you use an external journal.
get yourself a small SSD or a small 15RPM disk.  You could use a regular disk if you like but the faster the better. 

(EXT3)Destroy the journal and re-create it on the extra disk.
#unmount the backuppc disk
#on the journal device
mke2fs -O journal_dev -L name_label /dev/journal_disk
#drop old journal
tune2fs -O ^has_journal /dev/current_disk
#recreate the journal
tune2fs -o journal_data -j -J device=LABEL=name_label /dev/current_disk
-or-
tune2fs -o journal_data -j -J device=/dev/journal_disk /dev/current_disk
#remount the disk

#You can add a directory index to the filesystem for a small gain
tune2fs -O dir_index /dev/current_disk

#also, mount with noatime. (/etc/fstab)
/dev/sdb1 /share ext3 defaults,noatime,errors=remount-ro 0 1
The external journal will cut your I/O load on the disk/disk set in half because the filesystem no longer writes the journal on each transaction to that drive.  It's a small amount of data but it still requires a disk seek which is what hits the most for many small files (backuppc)




On Fri, Mar 5, 2010 at 7:01 AM, Josh Malone <jmalone < at > nrao.edu ([email]jmalone < at > nrao.edu[/email])> wrote:

It's hard to judge; but basically if there are a lot of processes
waiting
for I/O (a 'D' state in 'top'); try cutting down the number of
concurrent
backups. You'll have to judge for yourself what the best number for you
is.
It may be that things work fastest when there's a certain amount of disk
contention; but no more and no less.


Also - you need a good filesystem to handle lots (or even not so many) of
backups. I reently switched from EXT3 to EXT4 and saw on order of magnitude
(I kid you not, 10+ hours to 1) reduction in the backup time and system
load. Unfortunately, I think this introduced some problems in the RHEL5
ext4 code so I also switched from 32-bit RHEL5 to 64-bit -- that seems to
have cleared up the problems.

-Josh


--
--------------------------------------------------------
      Joshua Malone       Systems Administrator
    (jmalone < at > nrao.edu ([email]jmalone < at > nrao.edu[/email]))    NRAO Charlottesville
       434-296-0263         www.cv.nrao.edu
       434-249-5699 (mobile)
BOFH excuse #202:

kernel panic: write-only-memory (/dev/wom0) capacity
exceeded.
--------------------------------------------------------


------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net ([email]BackupPC-users < at > lists.sourceforge.net[/email])
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On 03/05/2010 09:01 AM, Josh Malone wrote:
Also - you need a good filesystem to handle lots (or even not so many) of
backups. I reently switched from EXT3 to EXT4 and saw on order of magnitude
(I kid you not, 10+ hours to 1) reduction in the backup time and system
load. Unfortunately, I think this introduced some problems in the RHEL5
ext4 code so I also switched from 32-bit RHEL5 to 64-bit -- that seems to
have cleared up the problems.

When you switched to ext4 and got this performance improvement, did
you simply upgrade your existing ext3 volumes via tune2fs, or did you
rebuild the entire filesystem so that the existing on-disk structures
were migrated as well?

Cheers,
Raman

------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On Fri, 05 Mar 2010 13:38:16 -0500, Raman Gupta <rocketraman < at > fastmail.fm>
wrote:
When you switched to ext4 and got this performance improvement, did
you simply upgrade your existing ext3 volumes via tune2fs, or did you
rebuild the entire filesystem so that the existing on-disk structures
were migrated as well?

Cheers,
Raman

The command I ran was:

umount /dev/foo
tune4fs -O extents,uninit_bg,dir_index /dev/foo
fsck -pDf /dev/foo

I can't remember if I had already mounted noatime before I made the switch
or not, but I also set noatime on the mount options. This is in a md-raid5
configuration across 4 7k spindles. It was choking on about 9 gigs of mysql
data per night and it's now fine. Transfer speed went from .5 to 1
MBytes/sec to 20+. I suspect that once of the mysql tables, which is 2GB+
in size, was choking the filesystem pretty badly (but I can't prove it)

-Josh

--
--------------------------------------------------------
Joshua Malone Systems Administrator
(jmalone < at > nrao.edu) NRAO Charlottesville
434-296-0263 www.cv.nrao.edu
434-249-5699 (mobile)
BOFH excuse #426:

internet is needed to catch the etherbunny
--------------------------------------------------------

------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Anyone have any info on JFS?


----- "Josh Malone" <jmalone < at > nrao.edu> wrote:

It's hard to judge; but basically if there are a lot of processes
waiting
for I/O (a 'D' state in 'top'); try cutting down the number of
concurrent
backups. You'll have to judge for yourself what the best number for you
is.
It may be that things work fastest when there's a certain amount of disk
contention; but no more and no less.

Also - you need a good filesystem to handle lots (or even not so many) of
backups. I reently switched from EXT3 to EXT4 and saw on order of magnitude
(I kid you not, 10+ hours to 1) reduction in the backup time and system
load. Unfortunately, I think this introduced some problems in the RHEL5
ext4 code so I also switched from 32-bit RHEL5 to 64-bit -- that seems to
have cleared up the problems.

-Josh


--
--------------------------------------------------------
Joshua Malone Systems Administrator
(jmalone < at > nrao.edu) NRAO Charlottesville
434-296-0263 www.cv.nrao.edu
434-249-5699 (mobile)
BOFH excuse #202:

kernel panic: write-only-memory (/dev/wom0) capacity
exceeded.
--------------------------------------------------------

------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Thanks for the pointers. I use JFS, with noatime. Would xfs perform better with backuppc ?


--
Thanks and Regards,

Anand Gupta

-----Original Message-----
From: dan <dandenson < at > gmail.com>
Date: Fri, 5 Mar 2010 10:12:11
To: General list for user discussion,questions and support<backuppc-users < at > lists.sourceforge.net>
Subject: Re: [BackupPC-users] high load and stuck processes

------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev


------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Gerald Brandt <gbr < at > majentis.com> wrote on 03/05/2010 02:32:10 PM:

Anyone have any info on JFS?

Not hard information with BackupPC, but just general JFS-type info.

As a long-time OS/2 user, I followed JFS quite actively for *years*. It
seems back in the day that there were quite a few proprietary solutions
that needed lots of storage (TiVo like things, DVD libraries, etc., back
when 1TB was a *lot* of storage...) that used JFS quite successfully.

The biggest problem for me was lack of visibility. For a backup server,
drop-dead reliability in *every* was was most important to me, and the
idea of JFS corruption always worried me. There were just way fewer
people with extensive experience with it, and seeing as Linux's JFS is
incompatible with AIX's JFS, the expertise available for AIX did not
really have anything to do with Linux, so I have stayed with EXT2 on my
BackupPC servers.

The performance of JFS, though, is appealing. I know a lot of people have
turned to XFS to deal with EXT2 limitations; JFS could certainly also be
an alternative in that situation, too. The other advantage that XFS has
is a seemingly more active (and visible) community. There was really just
a *single* guy at IBM behind JFS. He did incredible work in supporting
the community, but this lack of support was always chilling for me.

Eventually, I quit following JFS closely. I don't remember exactly, but
it was probably about 2-3 years ago when I did so: some time after every
major distro dropped support for it, especially in the installer...

Tim Massey


------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

I use BackupPC on a JFS mount, and it works quite well.

I haven't compared it to xfs or ext4 directly.

Anyone have any info on JFS?


------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

dan wrote:
If you are using EXT3 or XFS then I suggest you use an external journal.
get yourself a small SSD or a small 15RPM disk. You could use a regular
disk if you like but the faster the better.

This would work with a fast usbstick as well? With quite good results I
expect, and not much problems if it wears out, and perhaps cheaper than
a ssd. Or could I put in 4 usbsticks and create a raidset from it, and
store the journal on there? Perhaps stupid, but worth a shot. Depends
on what you're using the backupserver for i guess. SSD is probably more
reliable for a bigger shop.

/eric

------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On Sat, Mar 6, 2010 at 1:07 AM, Eric Persson <eric < at > persson.tm ([email]eric < at > persson.tm[/email])> wrote:
dan wrote:
If you are using EXT3 or XFS then I suggest you use an external journal.
get yourself a small SSD or a small 15RPM disk.  You could use a regular
disk if you like but the faster the better.


This would work with a fast usbstick as well? With quite good results I
expect, and not much problems if it wears out, and perhaps cheaper than
a ssd. Or could I put in 4 usbsticks and create a raidset from it, and
store the journal on there? Perhaps stupid, but worth a shot. Depends
on what you're using the backupserver for i guess. SSD is probably more
reliable for a bigger shop.



USB is the weak spot here.  You want something on SATA (or IDE, SCSI).  You could certainly try it but USB is pretty weak on IO performance so I wouldn't know if it would help performance or not.   Try to raid up a few sticks BUT make use you put them on different controllers.  A modern USB flash drive will see a USB bus speedlimit if you put two devices on 1 bus.

Oh, just as a reminder, you can do an external journal on ext3 and ext4 as well as xfs.

On Sat, Mar 6, 2010 at 9:37 AM, dan <dandenson < at > gmail.com ([email]dandenson < at > gmail.com[/email])> wrote:


On Sat, Mar 6, 2010 at 1:07 AM, Eric Persson <eric < at > persson.tm ([email]eric < at > persson.tm[/email])> wrote:
dan wrote:
If you are using EXT3 or XFS then I suggest you use an external journal.
get yourself a small SSD or a small 15RPM disk.  You could use a regular
disk if you like but the faster the better.


This would work with a fast usbstick as well? With quite good results I
expect, and not much problems if it wears out, and perhaps cheaper than
a ssd. Or could I put in 4 usbsticks and create a raidset from it, and
store the journal on there? Perhaps stupid, but worth a shot. Depends
on what you're using the backupserver for i guess. SSD is probably more
reliable for a bigger shop.




USB is the weak spot here.  You want something on SATA (or IDE, SCSI).  You could certainly try it but USB is pretty weak on IO performance so I wouldn't know if it would help performance or not.   Try to raid up a few sticks BUT make use you put them on different controllers.  A modern USB flash drive will see a USB bus speedlimit if you put two devices on 1 bus.