Backup Central

Le 18/12/2011 20:44, Pedro M. S. Oliveira a écrit :

you may try to use a rsyncd directly on the server. This may speed up
things.
another thing is to split the large backup into several smaller ones.
I've an email cluster with 8TB and millions of small files (I'm using
dovecot), theres also a san involved. in order to use all the
bandwidth available I configured backup to run from username starting
in a to e, f to j and so on, then they all run at the same time.
incremental take about 1 hour and full about 5.
cheers
pedro


I directly mount the nfs share on the backuppc server so no need for
rsyncd here this is like local backup with the NFS overhead of course.

Do you won a lot from splitting instead of doing just one big backup ?
At least you seems to have the same kind of file numbers i have.

regards,
Jean.

------------------------------------------------------------------------------
Learn Windows Azure Live! Tuesday, Dec 13, 2011
Microsoft is holding a special Learn Windows Azure training event for
developers. It will provide a great way to learn Windows Azure and what it
provides. You can attend the event by watching it streamed LIVE online.
Learn more at http://p.sf.net/sfu/ms-windowsazure
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On Mon, Dec 19, 2011 at 12:04 PM, Jean Spirat <jean.spirat < at > squirk.org> wrote:

I directly mount the nfs share on the backuppc server so no need for
rsyncd here this is like local backup with the NFS overhead of course.

The whole point of rsync is that it can read the files locally with
block checksums to decide what it really has to copy over the network.
Doing it over NFS, you've already had to copy if over the network so
rsync at the wrong end can read it (and decide that it didn't have
to...).

--
Les Mikesell
lesmikesell < at > gmail.com

------------------------------------------------------------------------------
Learn Windows Azure Live! Tuesday, Dec 13, 2011
Microsoft is holding a special Learn Windows Azure training event for
developers. It will provide a great way to learn Windows Azure and what it
provides. You can attend the event by watching it streamed LIVE online.
Learn more at http://p.sf.net/sfu/ms-windowsazure
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On Mon, 2011-12-19 at 12:32 -0600, Les Mikesell wrote:
On Mon, Dec 19, 2011 at 12:04 PM, Jean Spirat <jean.spirat < at > squirk.org> wrote:

I directly mount the nfs share on the backuppc server so no need for
rsyncd here this is like local backup with the NFS overhead of course.

The whole point of rsync is that it can read the files locally with
block checksums to decide what it really has to copy over the network.
Doing it over NFS, you've already had to copy if over the network so
rsync at the wrong end can read it (and decide that it didn't have
to...).

I think the real problem is the metadata access, and after a bit of
digging I've dug this up, it's comparing iSCSI with NFS.

But what might help is tweaking the NFS settings to improve the metadata
caching etc.

6.2 Meta-data intensive applications
NFS and iSCSI show their greatest differences in their
handling of meta-data intensive applications. Overall,
we find that iSCSI outperforms NFS for meta-data in-
tensive workloads—workloads where the network traffic
is dominated by meta-data accesses.
The better performance of iSCSI can be attributed to
two factors. First, NFS requires clients to update meta-
data synchronously to the server. In contrast, iSCSI,
when used in conjunction with modern file systems, up-
dates meta-data asynchronously. An additional bene-
fit of asynchronous meta-data updates is that it enables
update aggregation—multiple meta-data updates to the
same cached cached block are aggregated into a single
network write, yielding significant savings. Such opti-
mizations are not possible in NFS v2 or v3 due to their
synchronous meta-data update requirement.
Second, iSCSI also benefits from aggressive meta-
data caching by the file system. Since iSCSI reads are
in granularity of disk blocks, the file system reads and
caches entire blocks containing meta-data; applications
with meta-data locality benefit from such caching. Al-
though the NFS client can also cache meta-data, NFS
clients need to perform periodic consistency checks with
the server to provide weak consistency guarantees across
client machines that share the same NFS namespace.
Since the concept of sharing does not exist in the SCSI
architectural model, the iSCSI protocol also does not pay
the overhead of such a consistency protocol.

Full details are here: http://lass.cs.umass.edu/papers/pdf/FAST04.pdf

--
Tim Fletcher <tim < at > night-shade.org.uk>


------------------------------------------------------------------------------
Write once. Port to many.
Get the SDK and tools to simplify cross-platform app development. Create
new or port existing apps to sell to consumers worldwide. Explore the
Intel AppUpSM program developer opportunity. appdeveloper.intel.com/join
http://p.sf.net/sfu/intel-appdev
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

sorry to take long to reply.
yes it saves me a lot of time,  let me explain.
although I have a fast san and servers the time for fetching lots of small files is high,  the max bandwidth i could get was about 5MB/s, increasing concurrecy i can get about 20-40MB/s depending on what im backingup at the moment.  this way i can get more out of the san and backup server. if i increase concurrency even more i can reach higher performance but i don't want to steal all io available for backuppc, but to be sincere I don't need it anyway as i get really good performance this way.
This setup is running on a large finantional group and it outperforms very expensive (and complex) proprietary solutions.
the backuppc should have a fairly amount of ram,  cpu,  and isn't virtualized. in my case a 4 core server and 8GB ram (although it swaps a bit), i'm also using ssh+ rsync which add some overhead but not critical in any way.
cheers
pedro
Sent from my galaxy nexus.
www.linux-geex. com
On Dec 19, 2011 6:05 PM, "Jean Spirat" <jean.spirat < at > squirk.org ([email]jean.spirat < at > squirk.org[/email])> wrote: Le 18/12/2011 20:44, Pedro M. S. Oliveira a écrit :

you may try to use a rsyncd directly on the server. This may speed up things.
another thing is to split the large backup into several smaller ones.  I've an email cluster with 8TB and millions of small files (I'm using dovecot),  theres also a san involved. in order to use all the bandwidth available I configured backup to run from username starting in a to e,  f to j and so on,  then they all run at the same time. incremental take about 1 hour and full about 5.
cheers
pedro


I directly mount the nfs share on the backuppc server so no need for rsyncd here this is like local backup with the NFS overhead of course.

Do you won a lot from splitting instead of doing just one big backup ? At least you seems to have the same kind of file numbers i have.

regards,
Jean.