Backup Central

We are a small office (6 employees) with a mixture of windows and mac machines sitting on desks. I have set up a server (Ubuntu linux) that has been happily running backuppc for several years handling backups for all the machines in the office with grace AND style. We love it.

However, in the last few months some of the users have noticed that when backuppc is running a backup (incremental or full - does not seem to matter which) it can have a serious impact to the performance of their local machine. Stuff comes to a crawl and they are nearly unable to work because simple things like switching from one application to another starts to take several seconds, etc. The machine behaves like it is hammering swap space and thrashing for memory. At least one user reports this goes on for several hours (and I confirmed that his latest incremental took 119 minutes to complete).

All the machines affected in this way are wired to the gigabit network (not wireless), and I'm using rsync for the transfer method. The users with the complaints are all using OS X on late model high-end MacBook Pro laptops.

Is there anything I can to to have the backups run in a more transparent manner? We are not all that concerned with speed of backup process - we're all here all day anyway, so as long as everyone gets a backup at least once a day we're happy.

I have set up backuppc to only run 1 concurrent backup - should I change this to a larger number, making the server work harder and hopefully easing up on my clients a bit?

Thanks!

-- Kimball
------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On Tue, Jan 31, 2012 at 5:25 PM, Kimball Larsen <quangdog < at > gmail.com> wrote:
We are a small office (6 employees) with a mixture of windows and mac machines sitting on desks.  I have set up a server (Ubuntu linux) that has been happily running backuppc for several years handling backups for all the machines in the office with grace AND style.  We love it.

However, in the last few months some of the users have noticed that when backuppc is running a backup (incremental or full - does not seem to matter which) it can have a serious impact to the performance of their local machine.  Stuff comes to a crawl and they are nearly unable to work because simple things like switching from one application to another starts to take several seconds, etc.  The machine behaves like it is hammering swap space and thrashing for memory.  At least one user reports this goes on for several hours (and I confirmed that his latest incremental took 119 minutes to complete).

All the machines affected in this way are wired to the gigabit network (not wireless), and I'm using rsync for the transfer method.  The users with the complaints are all using OS X on late model high-end MacBook Pro laptops.

Is there anything I can to to have the backups run in a more transparent manner?  We are not all that concerned with speed of backup process - we're all here all day anyway, so as long as everyone gets a backup at least once a day we're happy.

Are you using any 'scan on access' type of virus protection? That
would be odd for a Mac, but I think there are such things. Do any
have local time machine backups that might be included? Or
directories with very large numbers of files? I think the rsync at
each end will keep a copy of the whole directory tree in memory while
both ends walk and compare contents. Normally this would be very fast
on incrementals where it doesn't do more than the directory check for
files that match but the list might be big enough to swap to disk.

--
Les Mikesell
lesmikesell < at > gmail.com

------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On Tue, Jan 31, 2012 at 5:25 PM, Kimball Larsen <quangdog < at > gmail.com> wrote:
Is there anything I can to to have the backups run in a more transparent manner?  We are not all that concerned with speed of backup process - we're all here all day anyway, so as long as everyone gets a backup at least once a day we're happy.

check the "nice" level during the backup (on the client) and see where
rsync is running. If it's the same as the other user processes, maybe
change the $Conf{RsyncClientCmd} to include a nice level? I know that
is suggested here:
http://backuppc.sourceforge.net/faq/ssh.html

A.

--
"The universe is probably littered with the one-planet graves of
cultures which made the sensible economic decision that there's no
good reason to go into space--each discovered, studied, and remembered
by the ones who made the irrational decision." - Randall Munroe

------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Two things :osX systemusers started to complain. Logs check of the backup
server.
Do parts of server fail? Disks perhaps?
Why not skip the dailyy routine of one a week randomly during? Prolongs
life of server (parts)
steve has you looking up the faq has not he? Try adding bitsamplerate
(oops! Other programme!)
but seriously do a wear and tear check of the server. Logs should indicate
something around the time slow became an issue

------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On Jan 31, 2012, at 5:10 PM, Les Mikesell wrote:

On Tue, Jan 31, 2012 at 5:25 PM, Kimball Larsen <quangdog < at > gmail.com> wrote:
We are a small office (6 employees) with a mixture of windows and mac machines sitting on desks. I have set up a server (Ubuntu linux) that has been happily running backuppc for several years handling backups for all the machines in the office with grace AND style. We love it.

However, in the last few months some of the users have noticed that when backuppc is running a backup (incremental or full - does not seem to matter which) it can have a serious impact to the performance of their local machine. Stuff comes to a crawl and they are nearly unable to work because simple things like switching from one application to another starts to take several seconds, etc. The machine behaves like it is hammering swap space and thrashing for memory. At least one user reports this goes on for several hours (and I confirmed that his latest incremental took 119 minutes to complete).

All the machines affected in this way are wired to the gigabit network (not wireless), and I'm using rsync for the transfer method. The users with the complaints are all using OS X on late model high-end MacBook Pro laptops.

Is there anything I can to to have the backups run in a more transparent manner? We are not all that concerned with speed of backup process - we're all here all day anyway, so as long as everyone gets a backup at least once a day we're happy.

Are you using any 'scan on access' type of virus protection? That
would be odd for a Mac, but I think there are such things.
Nope, no realtime scanning stuff at all.
Do any
have local time machine backups that might be included?

No, time machine is on external drives, specifically excluded from backups.

Or
directories with very large numbers of files?

This I can check on. What is considered "very large numbers of files"? More than 1024? More than 102400?

I think the rsync at
each end will keep a copy of the whole directory tree in memory while
both ends walk and compare contents. Normally this would be very fast
on incrementals where it doesn't do more than the directory check for
files that match but the list might be big enough to swap to disk.

Hmm.. Does it produce a copy of the whole directory tree for each backup location? If so, would it be beneficial to split up the backups such that instead of telling it to backup
/Users/myusername/ I explicitly list each of the directories in my home:
/Users/myusername/Documents/
/Users/myusername/Library
...etc?

Thanks!

-- Kimball



------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On Jan 31, 2012, at 5:47 PM, Steve wrote:

On Tue, Jan 31, 2012 at 5:25 PM, Kimball Larsen <quangdog < at > gmail.com> wrote:
Is there anything I can to to have the backups run in a more transparent manner? We are not all that concerned with speed of backup process - we're all here all day anyway, so as long as everyone gets a backup at least once a day we're happy.

check the "nice" level during the backup (on the client) and see where
rsync is running. If it's the same as the other user processes, maybe
change the $Conf{RsyncClientCmd} to include a nice level? I know that
is suggested here:
http://backuppc.sourceforge.net/faq/ssh.html

Ah - this looks to be exactly what I'm looking for. I figured I could probably nice the rsync command, but didn't know exactly where to find the correct syntax.

I'll give this a whirl.

Thanks!

-- Kimball



------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

-----Original Message-----
From: Steve [mailto:leperas < at > gmail.com]
Sent: den 1 februari 2012 01:48
To: General list for user discussion, questions and support
Subject: Re: [BackupPC-users] Backuppc is Hammering My Clients

check the "nice" level during the backup (on the client) and see where
rsync is running. If it's the same as the other user processes, maybe
change the $Conf{RsyncClientCmd} to include a nice level? I know that
is suggested here:
http://backuppc.sourceforge.net/faq/ssh.html

Good call, thanks. I'll try that on a machine over here. Backups take
forever on this ancient computer...
--
/Sorin

------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On 2012-02-01 08:32, Kimball Larsen wrote:
On Jan 31, 2012, at 5:47 PM, Steve wrote:
check the "nice" level during the backup (on the client) and see where
rsync is running. If it's the same as the other user processes, maybe
change the $Conf{RsyncClientCmd} to include a nice level? I know that
is suggested here:
http://backuppc.sourceforge.net/faq/ssh.html

Ah - this looks to be exactly what I'm looking for. I figured I could probably nice the rsync command, but didn't know exactly where to find the correct syntax.

Damn, others beat me to it. For Mac OS clients, I use:

$Conf{RsyncClientCmd} = '$sshPath -q -x -l root $host /usr/bin/nice -n 19
$rsyncPath $argList+';

Nice helps CPU priority, but it does nothing for disk IO. So for Linux
clients, I use:

$Conf{RsyncClientCmd} = '$sshPath -q -x -l root $host /usr/bin/nice -n 19
/usr/bin/ionice -c 3 $rsyncPath $argList+';

Regards,
Tyler

--
"I am patient with stupidity but not with those who are proud of it."
-- Edith Sitwell

------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On Tue, 2012-01-31 at 19:47 -0500, Steve wrote:
On Tue, Jan 31, 2012 at 5:25 PM, Kimball Larsen <quangdog < at > gmail.com> wrote:
Is there anything I can to to have the backups run in a more transparent manner? We are not all that concerned with speed of backup process - we're all here all day anyway, so as long as everyone gets a backup at least once a day we're happy.

check the "nice" level during the backup (on the client) and see where
rsync is running. If it's the same as the other user processes, maybe
change the $Conf{RsyncClientCmd} to include a nice level? I know that
is suggested here:
http://backuppc.sourceforge.net/faq/ssh.html

If you add the flags "-c arcfour" to the ssh command line that will use
the arcfour cypher which is less less secure but much cpu intensive, I
use it on CPU bound clients such as my iPhone.

--
Tim Fletcher <tim < at > night-shade.org.uk>


------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On Wed, Feb 1, 2012 at 2:30 AM, Kimball Larsen <quangdog < at > gmail.com> wrote:

 Do any
have local time machine backups that might be included?

No, time machine is on external drives, specifically excluded from backups.

It might be worth checking that the excludes work and the links that
make it show on the desktop aren't being followed.

Or
directories with very large numbers of files?

This I can check on.  What is considered "very large numbers of files"?  More than 1024?  More than 102400?

It would be relative to the amount of RAM available - probably millions.

I think the rsync at
each end will keep a copy of the whole directory tree in memory while
both ends walk and compare contents. Normally this would be very fast
on incrementals where it doesn't do more than the directory check for
files that match but the list might be big enough to swap to disk.

Hmm.. Does it produce a copy of the whole directory tree for each backup location?  If so, would it be beneficial to split up the backups such that instead of telling it to backup
/Users/myusername/  I explicitly list each of the directories in my home:
/Users/myusername/Documents/
/Users/myusername/Library
...etc?

Each 'share' is processed in a separate run. It will only help if the
number of files really is a problem. I always add --one-file-system
to the rsync args to avoid accidentally walking into nfs mounts or
external drives or DVDs that might be mounted. If you do, be sure to
add shares for everything you need.

--
Les Mikesell
lesmikesell < at > gmail.com

------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Les Mikesell wrote at about 07:40:20 -0600 on Wednesday, February 1, 2012:
On Wed, Feb 1, 2012 at 2:30 AM, Kimball Larsen <quangdog < at > gmail.com> wrote:

 Do any
have local time machine backups that might be included?

No, time machine is on external drives, specifically excluded from backups.

It might be worth checking that the excludes work and the links that
make it show on the desktop aren't being followed.

Or
directories with very large numbers of files?

This I can check on.  What is considered "very large numbers of files"?  More than 1024?  More than 102400?

It would be relative to the amount of RAM available - probably millions.

I have no trouble backing up half a million files on a system with
just 512MB. Surely, if you have high-powered relatively new PCs you
will have many 4+ Gigabytes so it is unlikely that RAM swapping will
be the problem. Plus any >3.0 version of rsync uses RAM quite
efficiently.

------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On Wed, 2012-02-01 at 20:24 -0500, Jeffrey J. Kosowsky wrote:
Les Mikesell wrote at about 07:40:20 -0600 on Wednesday, February 1, 2012:
On Wed, Feb 1, 2012 at 2:30 AM, Kimball Larsen <quangdog < at > gmail.com> wrote:

Do any
have local time machine backups that might be included?

No, time machine is on external drives, specifically excluded from backups.

It might be worth checking that the excludes work and the links that
make it show on the desktop aren't being followed.

Or
directories with very large numbers of files?

This I can check on. What is considered "very large numbers of files"? More than 1024? More than 102400?

It would be relative to the amount of RAM available - probably millions.

I have no trouble backing up half a million files on a system with
just 512MB. Surely, if you have high-powered relatively new PCs you
will have many 4+ Gigabytes so it is unlikely that RAM swapping will
be the problem. Plus any >3.0 version of rsync uses RAM quite
efficiently.

I find that the speed of the underlying storage is more of a factor than
the ram or cpu limits

--
Tim Fletcher <tim < at > night-shade.org.uk>


------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/