Backup Central

I've got a user running Win7-64 with 3 large drives: C (original boot), D
(data), and G (Win7 boot). I've configured rsyncd on it to serve them as 3
modules named for the drive letters. A "full" backup to my CentOS 5 box
takes about 40 hours, and an "incremental" takes about 8, so I've
configured his system to only back up on weekends. But even then, he often
works weekends and will reboot his machine because it's running slow from
the backup load, killing the backup.

What can I do to optimize the backup to reduce the time taken so it's more
likely to run to completion? I was thinking I could reconfigure it to be
three hosts with one drive each, so that there's more chance of running to
completion.

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

I've got a user running Win7-64 with 3 large drives: C (original boot), D
(data), and G (Win7 boot). I've configured rsyncd on it to serve them as 3
modules named for the drive letters. A "full" backup to my CentOS 5 box
takes about 40 hours, and an "incremental" takes about 8, so I've
configured his system to only back up on weekends. But even then, he often
works weekends and will reboot his machine because it's running slow from
the backup load, killing the backup.

What can I do to optimize the backup to reduce the time taken so it's more
likely to run to completion? I was thinking I could reconfigure it to be
three hosts with one drive each, so that there's more chance of running to
completion.

You don't say how big the drives are, but 40 hours says "terabytes" to me.
If that's not the case, I'd seriously consider [re]examining your rsync
version on the client. (Or perhaps beefing up your server.)

It's also possible that the client is underpowered in terms of CPU or RAM
for the vast numbers of files involved; it may be worth examining the
client during a backup to see what kind of resources it's using. (I
generally don't see a lot of slowness, even on active machines, here, but
we're not backing up terabytes from workstations, either.)

The suggestion you already made (namely, breaking the three modules into
three hosts) is probably the next thing I'd try.

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On Sun, Jul 8, 2012 at 9:59 PM, Kenneth Porter <shiva < at > sewingwitch.com> wrote:
I've got a user running Win7-64 with 3 large drives: C (original boot), D
(data), and G (Win7 boot). I've configured rsyncd on it to serve them as 3
modules named for the drive letters. A "full" backup to my CentOS 5 box
takes about 40 hours, and an "incremental" takes about 8, so I've
configured his system to only back up on weekends. But even then, he often
works weekends and will reboot his machine because it's running slow from
the backup load, killing the backup.

What can I do to optimize the backup to reduce the time taken so it's more
likely to run to completion? I was thinking I could reconfigure it to be
three hosts with one drive each, so that there's more chance of running to
completion.

Splitting the runs should help. You might even want to look at the
directory contents and split at the subdirectory level. Other than
sheer voulme, the things that can make rsync slow are very large
numbers of files where handling the directory listing becomes
cumbersone, and very large files with changes where the server has to
reconstruct a full copy with bits from the old and bits from the
transfer. An incremental against a fairly recent full should run
about as fast as both systems can read the directory contents, though.

--
Les Mikesell
lesmikesell < at > gmail.com

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On Monday, July 09, 2012 9:24 AM -0500 Michael Stowe
<mstowe < at > chicago.us.mensa.org> wrote:

You don't say how big the drives are, but 40 hours says "terabytes" to me.
If that's not the case, I'd seriously consider [re]examining your rsync
version on the client. (Or perhaps beefing up your server.)

The total size is 0.5 TB according to the Host Summary page. The network is
100 Mbit. The server is a Dell PE2900 with 4 GB. Probably the slowest link
is the external USB drive used to hold the backup on the server, formatted
as ext3. The client is mostly used for solid modeling.

I believe I'm using the default rsync args and don't know if there's
anything I can do to improve things there:

'--numeric-ids',
'--perms',
'--owner',
'--group',
'-D',
'--links',
'--hard-links',
'--times',
'--block-size=2048',
'--recursive',

The rsyncd service running on the Windows client is from cwrsync, version
3.0.8.

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On Mon, Jul 9, 2012 at 12:56 PM, Kenneth Porter <shiva < at > sewingwitch.com> wrote:

The total size is 0.5 TB according to the Host Summary page. The network is
100 Mbit. The server is a Dell PE2900 with 4 GB. Probably the slowest link
is the external USB drive used to hold the backup on the server, formatted
as ext3. The client is mostly used for solid modeling.

I believe I'm using the default rsync args and don't know if there's
anything I can do to improve things there:

'--numeric-ids',
'--perms',
'--owner',
'--group',
'-D',
'--links',
'--hard-links',
'--times',
'--block-size=2048',
'--recursive',

The rsyncd service running on the Windows client is from cwrsync, version
3.0.8.

See the docs about adding '--checksum-seed=32761'. It will save
reading/uncompressing the server-side copy for the next comparison,
but note that it doesn't start working until after the 2nd full run
with an unchanged file.

--
Les Mikesell
lesmikesell < at > gmail.com

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Probably the slowest link
is the external USB drive used to hold the backup on the server, formatted
as ext3.

In that case, let me add "don't do this, it's insanely slow" to my list of
suggestions.

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On Monday, July 09, 2012 2:19 PM -0500 Les Mikesell <lesmikesell < at > gmail.com>
wrote:

See the docs about adding '--checksum-seed=32761'. It will save
reading/uncompressing the server-side copy for the next comparison,
but note that it doesn't start working until after the 2nd full run
with an unchanged file.

Thanks. I just saw that in the config comments and that looked promising.



------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/