Backup Central

On Jan 31, 2012, at 4:25 PM, Kimball Larsen wrote:

We are a small office (6 employees) with a mixture of windows and mac machines sitting on desks. I have set up a server (Ubuntu linux) that has been happily running backuppc for several years handling backups for all the machines in the office with grace AND style. We love it.

However, in the last few months some of the users have noticed that when backuppc is running a backup (incremental or full - does not seem to matter which) it can have a serious impact to the performance of their local machine. Stuff comes to a crawl and they are nearly unable to work because simple things like switching from one application to another starts to take several seconds, etc. The machine behaves like it is hammering swap space and thrashing for memory. At least one user reports this goes on for several hours (and I confirmed that his latest incremental took 119 minutes to complete).

All the machines affected in this way are wired to the gigabit network (not wireless), and I'm using rsync for the transfer method. The users with the complaints are all using OS X on late model high-end MacBook Pro laptops.

Is there anything I can to to have the backups run in a more transparent manner? We are not all that concerned with speed of backup process - we're all here all day anyway, so as long as everyone gets a backup at least once a day we're happy.

I have set up backuppc to only run 1 concurrent backup - should I change this to a larger number, making the server work harder and hopefully easing up on my clients a bit?

Thanks!

-- Kimball

I just wanted to follow up with a description of what I changed to solve this:

First off, the users with performance problems on their machines during backups all had a copy of Parallels (Windows emulation software) that was either running or had been run in the last day. Parallels stores a virtual hard drive in a single file that is quite large - 14GB in one case and nearly 30 in another. These files were being included in the backups, and I suspect are the main culprit of my problem.

They are also backed up in Time Machine, so I simply removed them from backuppc as they are less critical than other data.

Further, following the suggestions of several of the folks who responded, I made 2 other changes to my RsyncClientCmd:
I changed the cipher to use arcfour, (which is a bit less secure, but we don't care because this all happens on our LAN) because it is faster.
Secondly, I set the nice level of the rsync command to make it play ... well... nicer.

Here is my current RsyncClientCmd:
$sshPath -q -x -c arcfour -l username $host nice -n 19 $rsyncPath $argList+


A heartfelt thank you to all who chimed in with suggestions.
Hopefully someone will find this useful in the future.

-- Kimball



------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On Wed, Feb 1, 2012 at 10:30 AM, Kimball Larsen <quangdog < at > gmail.com> wrote:
Here is my current RsyncClientCmd:
$sshPath -q -x -c arcfour -l username $host nice -n 19 $rsyncPath $argList+

I also added ionice to my archive command since disk activity can gave
the same effect even if you're not maxing out the processor.

I'm not sure if you'll run into the same problem but I think you can
only execute one thing, so in the above case you're running nice which
is in turn running your rsync. Someone on the list may have a hint on
how to successfully run both. I ended up with this for my archive
command:

$Conf{ArchiveClientCmd} = '/bin/nice -n19 /usr/bin/ionice -c3
$Installdir/bin/BackupPC_archiveHost $tarCreatePath $splitpath
$parpath $host $backupnumber $compression $compext $splitsize
$archiveloc $parfile *';

Richard

------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On 2012-02-01 16:30, Kimball Larsen wrote:
First off, the users with performance problems on their machines during backups all had a copy of Parallels (Windows emulation software) that was either running or had been run in the last day. Parallels stores a virtual hard drive in a single file that is quite large - 14GB in one case and nearly 30 in another. These files were being included in the backups, and I suspect are the main culprit of my problem.

Definitely exclude any VM disk images from any backup process. Unless you
can instruct the emulator to snapshot the disk (VMware ESX), there's no
point in copying it. It won't be usable. You'd have the same issue with
Virtualbox.

You can backup the contents of the VM itself, though. I use BackupPC to
backup both hosts and VM clients using rsync. It's easy to make a new VM
disk image, and restore into it.

Regards,
Tyler

--
"Religion is the opiate of the masses, so long as the masses are straight.
However, amass a bunch of lesbians and you're going to need actual drugs."
-- OKCupid Blog, with apologies to Karl Marx
http://blog.okcupid.com/index.php/gay-sex-vs-straight-sex/

------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Kimball Larsen wrote at about 09:30:38 -0700 on Wednesday, February 1, 2012:
I just wanted to follow up with a description of what I changed to solve this:

First off, the users with performance problems on their machines
during backups all had a copy of Parallels (Windows emulation
software) that was either running or had been run in the last day.
Parallels stores a virtual hard drive in a single file that is
quite large - 14GB in one case and nearly 30 in another. These
files were being included in the backups, and I suspect are the
main culprit of my problem.

This would certainly explain the behavior. Since those files likely
change each day, Rsync needs to read through them and calculate
rolling md4 checksums for each several kilobyte block along with a
full file md4 checksum. The checksums then need to be aligned with
(and possibly also calculated on) the server side. When the checksums
don't match and can't be aligned the differing blocks need to be
transferred. Finally the new file is reconstructed on the server side
and compressed. Clearly, this is not a trivial task for a 30GB file.

------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/