Backup Central

Hello,

recently we were forced to re-setup a large BackupPC-server with up to
150 clients and 5.0TB (12 disks in RAID10). This was several weeks ago.

Unfortunately the server has problems keeping up with the backups since
then. (see
http://sourceforge.net/mailarchive/message.php?msg_id=29433950) Even
after enabling rsync checksum caching and playing around with raising or
lowering MaxBackups, MaxPendingCmds, the server doesn't scale as
expected. And just for the record: before the re-setup, it managed the
same amount of backups and the same hosts.

Now while comparing BackupPC logs to another big BackupPC-server on our
side, I recognized that BackupPC-Link takes ages for every backupped
host on the new server, while it's incredibly fast on the other BackupPC
server.

Even for hosts with several 100GB of data, BackupPC_Link takes only few
minutes on the old BackupPC server. On the new (re-setup) one, it takes
several hours up to days for backupped hosts.

Do you have any idea or suggestions why BackupPC_Link takes that long
and how we can change it?

Best regards,
jonas



------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Initial backups on the new server would take longer than what you're seeing with the established server which only needs to do changes after having done the initial full.  Your new server has already done the initial backups of your host and it's the subsequent backups that are taking so long?

But on this topic, my new server seems slow, with only a 3-4 MB/s speed rating on a GB network (Windows backup over SMB).  Is this normal?  What kind of speeds should I be expecting?


On Thu, Jul 5, 2012 at 7:12 AM, jonas <jonas < at > freesources.org ([email]jonas < at > freesources.org[/email])> wrote:
Hello,

recently we were forced to re-setup a large BackupPC-server with up to
150 clients and 5.0TB (12 disks in RAID10). This was several weeks ago.

Unfortunately the server has problems keeping up with the backups since
then. (see
http://sourceforge.net/mailarchive/message.php?msg_id=29433950) Even
after enabling rsync checksum caching and playing around with raising or
lowering MaxBackups, MaxPendingCmds, the server doesn't scale as
expected. And just for the record: before the re-setup, it managed the
same amount of backups and the same hosts.

Now while comparing BackupPC logs to another big BackupPC-server on our
side, I recognized that BackupPC-Link takes ages for every backupped
host on the new server, while it's incredibly fast on the other BackupPC
server.

Even for hosts with several 100GB of data, BackupPC_Link takes only few
minutes on the old BackupPC server. On the new (re-setup) one, it takes
several hours up to days for backupped hosts.

Do you have any idea or suggestions why BackupPC_Link takes that long
and how we can change it?

Best regards,
  jonas



------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net ([email]BackupPC-users < at > lists.sourceforge.net[/email])
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On 05/07/12 22:12, jonas wrote:
Hello,

recently we were forced to re-setup a large BackupPC-server with up to
150 clients and 5.0TB (12 disks in RAID10). This was several weeks ago.

Unfortunately the server has problems keeping up with the backups since
then. (see
http://sourceforge.net/mailarchive/message.php?msg_id=29433950) Even
after enabling rsync checksum caching and playing around with raising or
lowering MaxBackups, MaxPendingCmds, the server doesn't scale as
expected. And just for the record: before the re-setup, it managed the
same amount of backups and the same hosts.

Now while comparing BackupPC logs to another big BackupPC-server on our
side, I recognized that BackupPC-Link takes ages for every backupped
host on the new server, while it's incredibly fast on the other BackupPC
server.

Even for hosts with several 100GB of data, BackupPC_Link takes only few
minutes on the old BackupPC server. On the new (re-setup) one, it takes
several hours up to days for backupped hosts.

Do you have any idea or suggestions why BackupPC_Link takes that long
and how we can change it?
One thing this sounds like is that you moved the location for the backups, therefore instead of linking the files, every backup is actually a full copy (because the links fail to the pool/cpool). Check the backuppc log files and see if anything unusual is being logged.

Failing that, consider the things that have changed from the old server to the "re-setup server":
1) Hardware - hard drives, controller, amount of RAM, etc....
2) Filesystem or hard drive layout/configuration (ie, RAID level, layout, chunk size, ext3 compared to jfs, etc)
3) Double check (while the system is idle if possible) the performance of the server, disk speed, especially random read/write performance).
4) Consider that if all clients are configured at the same time, then all clients will start doing full backups on the same days, which will screw with performance. Try to manually run full backups on some clients to skew the timelines, so that full backups are spread across different days.

Hope something above helps or triggers some idea of the issue.

Regards,
Adam

--
Adam Goryachev
Website Managers
www.websitemanagers.com.au

Hello again,

first, I forgot to mention an important detail: we changed from ext3 to
xfs on the 5TB RAID10 array which holds all BackupPC data during the
re-setup.

Am 05.07.2012 15:04, schrieb Adam Goryachev:
One thing this sounds like is that you moved the location for the
backups, therefore instead of linking the files, every backup is
actually a full copy (because the links fail to the pool/cpool).
Check
the backuppc log files and see if anything unusual is being logged.

no, this doesn't seem to be the problem. I didn't find any unexpected
logs. The pc/$HOST directory shrinks a lot during the link-process, so I
consider this as evidence that pooling actually works as expected. I did
recognize though, that the pool directory is empty on all BackupPC
servers. All pooled files seem to be stored in cpool instead. Is this
expected?

Failing that, consider the things that have changed from the old
server to the "re-setup server":
1) Hardware - hard drives, controller, amount of RAM, etc....

Kept exactly the same. We actually shredded the filesystem and thus had
to re-setup. Hardware didn't change.

2) Filesystem or hard drive layout/configuration (ie, RAID level,
layout, chunk size, ext3 compared to jfs, etc)

As written above: moved from ext3 to xfs as we thought that this might
increase performance.

3) Double check (while the system is idle if possible) the
performance of the server, disk speed, especially random read/write
performance).

Seems normal to me. BackupPC produces heavy load and IO-wait, but
that's expected for large backup-setups, isn't it?

Here's some value from 'iostat -kx' (/dev/sdb contains all the BackupPC
data):

Linux 2.6.32-5-amd64 (backuphost) 07/05/12 _x86_64_ (8 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
13.42 0.00 1.73 25.96 0.00 58.89

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz
await svctm %util
sdb 42.65 1141.40 336.04 168.85 7006.47 5613.14 49.99 0.27
0.52 0.33 16.42


4) Consider that if all clients are configured at the same time,
then all clients will start doing full backups on the same days,
which
will screw with performance. Try to manually run full backups on some
clients to skew the timelines, so that full backups are spread across
different days.

Yes, already did this, I even deactivated most of the hosts for several
days to give BackupPC some time to keep up with backlog. Didn't help
either.

Am 05.07.2012 14:22, schrieb Bryan Keadle (.net):
Initial backups on the new server would take longer than what you're
seeing with the established server which only needs to do changes
after having done the initial full. Your new server has already done
the initial backups of your host and it's the subsequent backups that
are taking so long?

Yes, all initial full backups are done since more than a week ago.
Still the described issues.

Best Regards,
jonas



------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On 05/07/12 23:53, jonas wrote:
Hello again,

first, I forgot to mention an important detail: we changed from
ext3 to xfs on the 5TB RAID10 array which holds all BackupPC data
during the re-setup.

Am 05.07.2012 15:04, schrieb Adam Goryachev: no, this doesn't seem
to be the problem. I didn't find any unexpected logs. The pc/$HOST
directory shrinks a lot during the link-process, so I consider this
as evidence that pooling actually works as expected. I did
recognize though, that the pool directory is empty on all BackupPC
servers. All pooled files seem to be stored in cpool instead. Is
this expected?
compressed files are in cpool, so this suggests you have compression
enabled (for file storage, not in transit). This is 'normal'.
2) Filesystem or hard drive layout/configuration (ie, RAID level,
layout, chunk size, ext3 compared to jfs, etc) As written above:
moved from ext3 to xfs as we thought that this might increase
performance.
Obviously this is the big change, so you should probably start here. I
would suggest reading up on how to optimize xfs for backuppc, or
generically, a large number of small read/writes.

I recently subscribed to the linux-raid mailing list for a backuppc
related issue in fact, and saw a few un-related (to me) posts regarding
default raid stripe size, and (I think it was) xfs file system being
problematic. I think there is some relationship with the FS storing some
type of data every x bytes per disk, and if this data all ends up on the
same physical disk (or in your case pair of disks) then you end up with
a dis-proportionate amount of load on a small subset of your disk array.
It seems you are using hardware raid which presents the disk as "sdk",
however, the basic idea would still apply (stripe size of the raid and
block size of the FS).

I'd suggest to subscribe to the xfs mailing list, and/or look into how
you should optimise your filesystem to improve performance (defaults are
not always ideal).

Regards,
Adam

--
Adam Goryachev
Website Managers
Ph: +61 2 8304 0000 adam < at > websitemanagers.com.au
Fax: +61 2 8304 0001 www.websitemanagers.com.au

--
Adam Goryachev
Website Managers
www.websitemanagers.com.au


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Good thread here.

Jonas - sorry for my reply about the initial backups - you clearly know what you're doing, and thus you're issue is after your initial backups.  That XFS might be the problem sounds plausible.  Please report back what you find for those of us following this thread.


Finally, what are "typical" SMB backup speeds I should be seeing?



On Thu, Jul 5, 2012 at 9:10 AM, Adam Goryachev <mailinglists < at > websitemanagers.com.au ([email]mailinglists < at > websitemanagers.com.au[/email])> wrote:
On 05/07/12 23:53, jonas wrote:
Hello again,

first, I forgot to mention an important detail: we changed from
ext3 to xfs on the 5TB RAID10 array which holds all BackupPC data
during the re-setup.


Am 05.07.2012 15:04, schrieb Adam Goryachev: no, this doesn't seem
to be the problem. I didn't find any unexpected logs. The pc/$HOST
directory shrinks a lot during the link-process, so I consider this
as evidence that pooling actually works as expected. I did
recognize though, that the pool directory is empty on all BackupPC
servers. All pooled files seem to be stored in cpool instead. Is
this expected?

compressed files are in cpool, so this suggests you have compression
enabled (for file storage, not in transit). This is 'normal'.
2) Filesystem or hard drive layout/configuration (ie, RAID level,
layout, chunk size, ext3 compared to jfs, etc) As written above:
moved from ext3 to xfs as we thought that this might increase
performance.

Obviously this is the big change, so you should probably start here. I
would suggest reading up on how to optimize xfs for backuppc, or
generically, a large number of small read/writes.

I recently subscribed to the linux-raid mailing list for a backuppc
related issue in fact, and saw a few un-related (to me) posts regarding
default raid stripe size, and (I think it was) xfs file system being
problematic. I think there is some relationship with the FS storing some
type of data every x bytes per disk, and if this data all ends up on the
same physical disk (or in your case pair of disks) then you end up with
a dis-proportionate amount of load on a small subset of your disk array.
It seems you are using hardware raid which presents the disk as "sdk",
however, the basic idea would still apply (stripe size of the raid and
block size of the FS).

I'd suggest to subscribe to the xfs mailing list, and/or look into how
you should optimise your filesystem to improve performance (defaults are
not always ideal).

Regards,
Adam

--
Adam Goryachev
Website Managers

Ph: adam < at > websitemanagers.com.au ([email]adam < at > websitemanagers.com.au[/email])
Fax: [url=tel:%2B61%202%208304%200001]+61 2 8304 0001[/url]                            www.websitemanagers.com.au

--
Adam Goryachev
Website Managers
www.websitemanagers.com.au



------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net ([email]BackupPC-users < at > lists.sourceforge.net[/email])
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Hi

But on this topic, my new server seems slow, with only a 3-4 MB/s speed
rating on a GB network (Windows backup over SMB). Is this normal? What
kind of speeds should I be expecting?

smb can be VERY slow with many small files, totally killing
its average speed.

Try testing with a smb with just a few very big files to
check the speed.

You may also need some optimization, in a GB network is
recommended to enable jumbo frames... and samba might need the
some tuned up linux settings:

socket options = TCP_NODELAY SO_KEEPALIVE SO_RCVBUF=65536 SO_SNDBUF=65536
deadtime = 15

add the IPTOS_LOWDELAY if you have many small files, it decrease
delay (helps on the small files), but also decrease throughput, so (bad
for the big ones)

Good luck
higuta
--
Naturally the common people don't want war... but after all it is the
leaders of a country who determine the policy, and it is always a
simple matter to drag the people along, whether it is a democracy, or
a fascist dictatorship, or a parliament, or a communist dictatorship.
Voice or no voice, the people can always be brought to the bidding of
the leaders. That is easy. All you have to do is tell them they are
being attacked, and denounce the pacifists for lack of patriotism and
exposing the country to danger. It works the same in every country.
-- Hermann Goering, Nazi and war criminal, 1883-1946

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Hey again,

Am 05.07.2012 16:10, schrieb Adam Goryachev:
first, I forgot to mention an important detail: we changed from
ext3 to xfs on the 5TB RAID10 array which holds all BackupPC data
during the re-setup.

Am 05.07.2012 15:04, schrieb Adam Goryachev: no, this doesn't seem
to be the problem. I didn't find any unexpected logs. The pc/$HOST
directory shrinks a lot during the link-process, so I consider this
as evidence that pooling actually works as expected. I did
recognize though, that the pool directory is empty on all BackupPC
servers. All pooled files seem to be stored in cpool instead. Is
this expected?
compressed files are in cpool, so this suggests you have compression
enabled (for file storage, not in transit). This is 'normal'.
2) Filesystem or hard drive layout/configuration (ie, RAID level,
layout, chunk size, ext3 compared to jfs, etc) As written above:
moved from ext3 to xfs as we thought that this might increase
performance.

Obviously this is the big change, so you should probably start here. I
would suggest reading up on how to optimize xfs for backuppc, or
generically, a large number of small read/writes.

I now set up a new server with comparable hardware resources and
migrated all backup clients to this new host. I added the clients in
three big steps, and so far, the new server keeps up pretty well. The
new server uses ext4 as filesystem.

But I recently found another reason why BackupPC_Link processes took so
long on the old server. The option to fill up incremental backups was
enabled for some reason. I turned it off now. I also lowered the maximum
number of pending link processes. I will keep the old server running for
some more weeks, to see whether these changes finally make the difference.

I recently subscribed to the linux-raid mailing list for a backuppc
related issue in fact, and saw a few un-related (to me) posts regarding
default raid stripe size, and (I think it was) xfs file system being
problematic. I think there is some relationship with the FS storing some
type of data every x bytes per disk, and if this data all ends up on the
same physical disk (or in your case pair of disks) then you end up with
a dis-proportionate amount of load on a small subset of your disk array.
It seems you are using hardware raid which presents the disk as "sdk",
however, the basic idea would still apply (stripe size of the raid and
block size of the FS).

Unfortunately I didn't find the time to dig into xfs tuning yet, but
soon I'll know whether it makes a big difference, or whether the 'fill
incemental backups' option was the sole reason for my problems.

Regards,
jonas


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/