Backup Central

Hi maarten,

Some questions to help myself (and the list) helping:

1) Which file systems on source and target did you use before and after the server change? Which other ones did you try?
2) What kind of data do you have? Millions of dirs with millions of small files or millions of files in just a few dirs or several very large files (gigabytes) or all of them?

Still, what is observable is that any initial backup run (with --force)
runs significantly faster on the new server.

Think so, too.

Any differential run
afterwards is slower than on the original server. I feel this proves
there are no performance bottlenecks in the network, disks, filesystems
etc of the server.


The new server runs rdiff-backup 1.2.8, the old one 1.0.5.

Did you upgrade the remote server too?

Downgrading
the new server to 1.0.5 makes things a bit interesting: that speeds it
up a bit, but still a fair bit slower than the original.

As of Rdiff-backup 1.1.1, hashing is done with sha-1, maybe this slows down a bit. Source: http://rdiff-backup.nongnu.org/CHANGELOG-stable

During investigation we experimented with different filesystems, testing
local versus remote backups, looking at compile flags and versions of
librsync and python, but we have had no success there.

Did you try reiserfs (nice with lots and lots of small files)?




_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

Probably something above 5.

Yeah, please give it a 8 ... maybe then make it a --terminal-verbosity 8 rather than -v8 so that you don't let the backup-log in the target explode ...


_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

D. Kriesel wrote:
Hi maarten,

Some questions to help myself (and the list) helping:

1) Which file systems on source and target did you use before and after the server change? Which other ones did you try?
2) What kind of data do you have? Millions of dirs with millions of small files or millions of files in just a few dirs or several very large files (gigabytes) or all of them?

Okay, sorry, I should clarify a bit. By remote I meant it is coming over
the network, but it is not a client-server rdiff-backup setup. The
source comes from a Windows server 2008 share, mounted with cifs on the
two backup servers, the old one and the new one.
File systems on both backupserver are reiserfs. We had XFS as a test,
went for reiserfs in the end, to make both environments as equal as
possible.
Data is mostly 'office stuff' so anything ranging for spreadsheets,
photos, docs to your typical windows (profile) files. No source code, no
movies, no big drive snapshots. So basically both huge files and tiny
files are more or less scarce in this source tree. total number of files
runs in the high 6 digits, but definitely not more than a million.

Still, what is observable is that any initial backup run (with --force)
runs significantly faster on the new server.

Think so, too.

Any differential run
afterwards is slower than on the original server. I feel this proves
there are no performance bottlenecks in the network, disks, filesystems
etc of the server.


The new server runs rdiff-backup 1.2.8, the old one 1.0.5.

Did you upgrade the remote server too?

Sorry was not clear: no remote rdiff-backup, just a network-mounted share.

Downgrading
the new server to 1.0.5 makes things a bit interesting: that speeds it
up a bit, but still a fair bit slower than the original.

As of Rdiff-backup 1.1.1, hashing is done with sha-1, maybe this slows down a bit. Source: http://rdiff-backup.nongnu.org/CHANGELOG-stable

That's interesting.

During investigation we experimented with different filesystems, testing
local versus remote backups, looking at compile flags and versions of
librsync and python, but we have had no success there.

Did you try reiserfs (nice with lots and lots of small files)?

Certainly. It was our first choice, because of that.

Regards,
Maarten



_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki