Backup Central

Hello List,

Im using rsnapshot on the backup machine
Fetching the snapshot using ssh (1gb connection)
SOURCE TOTAL FILES FILES TRANS TOTAL MB
MB TRANS LIST GEN TIME FILE XFER TIME
----------------------------------------------------------------------------
----------------------------------------
192.168.1.31:/etc/sysconfig/vz 25 0 0.04
0.00 0.035 seconds 0.000 seconds
192.168.1.31:/vz/private/ 3333414 16072 121230.98
22497.51 2447.730 seco 0.000 seconds

The backup takes 8 hours and 10 minutes
Is that normal ?
/vz/private contains server containers
Is it better for speeding up to backup every container separately
I hope not then I have to add every container menualy when creating a new
one

Thanxs Steffan



------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

Hallo, Steffan,

Du meintest am 28.11.11:

Im using rsnapshot on the backup machine
Fetching the snapshot using ssh (1gb connection)

[...]

The backup takes 8 hours and 10 minutes
Is that normal ?

May be.

On which machine runs "rsnapshot"?
What tells the "rsnapshot" log file?
How big is 1 backup?

Viele Gruesse!
Helmut

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

Hallo Steffan,

Am Montag, 28. November 2011 10:40 schrieb Steffan:
Hello List,

Im using rsnapshot on the backup machine
Fetching the snapshot using ssh (1gb connection)
SOURCE TOTAL FILES FILES TRANS
TOTAL MB MB TRANS LIST GEN TIME FILE XFER TIME
-----------------------------------------------------------------
----------- ----------------------------------------
192.168.1.31:/etc/sysconfig/vz 25 0
0.04 0.00 0.035 seconds 0.000 seconds
192.168.1.31:/vz/private/ 3333414 16072
121230.98 22497.51 2447.730 seco 0.000 seconds

The backup takes 8 hours and 10 minutes
Is that normal ?

It looks like you backed up from /vz/private/ 16072 files out of
3333414 files and transfered 22 GB out of 121 GB.
Most of the time 22497secs = 6:14 hours was taken by the list
generation, 2447secs = 40 min for transfer, if I understand your
data right.

I think, the list generation would be much faster, if rsync is
running on the other machine, so you spare the encryption.

--
Herzliche Grüße!
Rolf Muth
Meine Adressen duerfen nicht fuer Werbung verwendet werden!
PGP Public Key:
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xF8DC41935544C89A

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

Rolf Muth <rolf.muth < at > web.de> wrote:

Hallo Steffan,

Am Montag, 28. November 2011 10:40 schrieb Steffan:
Hello List,

Im using rsnapshot on the backup machine
Fetching the snapshot using ssh (1gb connection)
SOURCE TOTAL FILES FILES TRANS
TOTAL MB MB TRANS LIST GEN TIME FILE XFER TIME
-----------------------------------------------------------------
----------- ----------------------------------------
192.168.1.31:/etc/sysconfig/vz 25 0
0.04 0.00 0.035 seconds 0.000 seconds
192.168.1.31:/vz/private/ 3333414 16072
121230.98 22497.51 2447.730 seco 0.000 seconds

The backup takes 8 hours and 10 minutes
Is that normal ?

It looks like you backed up from /vz/private/ 16072 files out of
3333414 files and transfered 22 GB out of 121 GB.
Most of the time 22497secs = 6:14 hours was taken by the list
generation, 2447secs = 40 min for transfer, if I understand your
data right.

I think, the list generation would be much faster, if rsync is
running on the other machine, so you spare the encryption.

I am having rsnapshot take a very long time on a local backup via a usb
drive. I started at 12 noon and the rsync is still going, there is very
little data to transfer, but its an 800G file system. Also the cp -al
takes quite a while, last time it took 3 hours. When I first tried, it
didn't take so long. I do have -v added to the rsync short arguments,
if that makes any differences -- and lots of excludes as well.

--
Your life is like a penny. You're going to lose it. The question is:
How do
you spend it?

John Covici
covici < at > ccs.covici.com

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

Eeh looks like rsnapshot is not supporting pushing transfer the files to
another server
So how do you do that ?



-----Oorspronkelijk bericht-----
Van: Rolf Muth [mailto:rolf.muth < at > web.de]
Verzonden: maandag 28 november 2011 16:51
Aan: rsnapshot-discuss < at > lists.sourceforge.net
Onderwerp: Re: [rsnapshot-discuss] rsnapshot takes a long time

Hallo Steffan,

Am Montag, 28. November 2011 10:40 schrieb Steffan:
Hello List,

Im using rsnapshot on the backup machine
Fetching the snapshot using ssh (1gb connection)
SOURCE TOTAL FILES FILES TRANS
TOTAL MB MB TRANS LIST GEN TIME FILE XFER TIME
-----------------------------------------------------------------
----------- ----------------------------------------
192.168.1.31:/etc/sysconfig/vz 25 0
0.04 0.00 0.035 seconds 0.000 seconds
192.168.1.31:/vz/private/ 3333414 16072
121230.98 22497.51 2447.730 seco 0.000 seconds

The backup takes 8 hours and 10 minutes
Is that normal ?

It looks like you backed up from /vz/private/ 16072 files out of
3333414 files and transfered 22 GB out of 121 GB.
Most of the time 22497secs = 6:14 hours was taken by the list
generation, 2447secs = 40 min for transfer, if I understand your
data right.

I think, the list generation would be much faster, if rsync is
running on the other machine, so you spare the encryption.

--
Herzliche Grüße!
Rolf Muth
Meine Adressen duerfen nicht fuer Werbung verwendet werden!
PGP Public Key:
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xF8DC41935544C89A


------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

Hallo, Steffan,

Du meintest am 29.11.11:

Im using rsnapshot on the backup machine

[...]

The backup takes 8 hours and 10 minutes
Is that normal ?

It looks like you backed up from /vz/private/ 16072 files out of
3333414 files and transfered 22 GB out of 121 GB.
Most of the time 22497secs = 6:14 hours was taken by the list
generation, 2447secs = 40 min for transfer, if I understand your
data right.

I think, the list generation would be much faster, if rsync is
running on the other machine, so you spare the encryption.


Eeh looks like rsnapshot is not supporting pushing transfer the files
to another server
So how do you do that ?

[Fullquote corrected]

As you have told me per direct mail "rsnapshot" works on that machine
which also holds the disk/partition for the backups. In that
configuration "rsnapshot" can work really fast.

How have you connected the machine which has to be backed up? NFS or
smbfs/cifs or rsync or some other way?

Where is the other machine: in the same LAN or somewhere in the WAN? How
fast is the connection?

Perhaps you should show your "rsnapshot.conf".

What tells the log file (it may be "/var/log/rsnapshot.log")?

Viele Gruesse!
Helmut

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

Hallo, Steffan,

Du meintest am 28.11.11:

/vz/private contains server containers
Is it better for speeding up to backup every container separately

Just try it - may be there is the time eater.

What do you mean with "server container"?

I hope not then I have to add every container menualy when creating a
new one

Where is the big problem?

Viele Gruesse!
Helmut

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

Hallo, Steffan,

Du meintest am 29.11.11:

Eeh looks like rsnapshot is not supporting pushing transfer the
files to another server
So how do you do that ?

[Fullquote corrected]

Perhaps you should show your "rsnapshot.conf".

What tells the log file (it may be "/var/log/rsnapshot.log")?

[Fullquote corrected - please stop this nasty behaviour of "outlook"!
Thank you!]

Oke here is my setup

Server 1 (the serevr I want to backup)

Can you please show your "rnsnapshot.conf"? Can you please show the log file?

I've just looked onto/into 2 systems:

System A: backing up about 160 GByte (home directories) from one disk in
the machine to another needs about 9 minutes.

System B: backing up about 155 GByte (home directories) from one machine
over Gigabit net to another machine using rsync connection needs about 9
minutes.

---------------- part of the log file -------------

[29/Nov/2011:06:21:34] /usr/bin/rsync -ax --delete --numeric-ids --relative --delete-excluded --include=home/tmp/* --exclude-from=/etc/rsnapshot.d/exclude-file --rsh=/usr/bin/ssh --link-dest=/srv2/backup/hourly.1/./ root < at > 10.0.0.1:/home /srv2/backup/hourly.0/./

---------------- end log file ---------------------

Can you please run

rsnapshot diff

and perhaps

rsnapshot du

and show the results?

Viele Gruesse!
Helmut

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

Hallo Steffan,

Am Dienstag, 29. November 2011 11:30 schrieb Steffan:
Eeh looks like rsnapshot is not supporting pushing transfer the
files to another server
So how do you do that ?

I don't do that and do not know how to do it.
I have my backup on the lokal machine on a second harddisk and rsync
it to an extern one.
Perhaps you could use a similar way (rsync a backup over ssh), but
you have really a lot of data.

I think, the list generation would be much faster, if rsync is
running on the other machine, so you spare the encryption.

--
Herzliche Grüße!
Rolf Muth
Meine Adressen duerfen nicht fuer Werbung verwendet werden!
PGP Public Key:
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xF8DC41935544C89A

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

#######################
# CONFIG FILE VERSION #
#######################

config_version 1.2
snapshot_root /extern1/servername
cmd_cp /bin/cp
cmd_rm /bin/rm
cmd_rsync /usr/bin/rsync
cmd_ssh /usr/bin/ssh
cmd_logger /usr/bin/logger
interval hourly 3
interval daily 6
interval weekly 3
interval monthly 3
verbose 3
loglevel 3
lockfile /var/run/rsnapshot.servername.pid
rsync_long_args --stats --delete --numeric-ids --delete-excluded
backup root < at > 192.168.1.31:/etc/sysconfig/vz-scripts/ servername/
backup root < at > 192.168.1.31:/vz/private/ servername/
#############################


/usr/bin/rsync -a --stats --rsh=/usr/bin/ssh
root < at > 192.168.1.31:/etc/sysconfig/vz-scripts/
/extern1/servername/hourly.0/servername/

/usr/bin/rsync -a --stats --rsh=/usr/bin/ssh root < at > 192.168.1.31:/vz/private/
/extern1/servername/hourly.0/servername/


-----Oorspronkelijk bericht-----
Van: Helmut Hullen [mailto:Hullen < at > t-online.de]
Verzonden: dinsdag 29 november 2011 14:02
Aan: rsnapshot-discuss < at > lists.sourceforge.net
Onderwerp: Re: [rsnapshot-discuss] rsnapshot takes a long time

Hallo, Steffan,

Du meintest am 29.11.11:

Eeh looks like rsnapshot is not supporting pushing transfer the
files to another server
So how do you do that ?

[Fullquote corrected]

Perhaps you should show your "rsnapshot.conf".

What tells the log file (it may be "/var/log/rsnapshot.log")?

[Fullquote corrected - please stop this nasty behaviour of "outlook"!
Thank you!]

Oke here is my setup

Server 1 (the serevr I want to backup)

Can you please show your "rnsnapshot.conf"? Can you please show the log
file?

I've just looked onto/into 2 systems:

System A: backing up about 160 GByte (home directories) from one disk in
the machine to another needs about 9 minutes.

System B: backing up about 155 GByte (home directories) from one machine
over Gigabit net to another machine using rsync connection needs about 9
minutes.

---------------- part of the log file -------------

[29/Nov/2011:06:21:34] /usr/bin/rsync -ax --delete --numeric-ids --relative
--delete-excluded --include=home/tmp/*
--exclude-from=/etc/rsnapshot.d/exclude-file --rsh=/usr/bin/ssh
--link-dest=/srv2/backup/hourly.1/./ root < at > 10.0.0.1:/home
/srv2/backup/hourly.0/./

---------------- end log file ---------------------

Can you please run

rsnapshot diff

and perhaps

rsnapshot du

and show the results?

Viele Gruesse!
Helmut

----------------------------------------------------------------------------
--
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss


------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

/vz/private contains server containers
Is it better for speeding up to backup every container separately

Just try it - may be there is the time eater.
Will try it this week

What do you mean with "server container"?
The main server is a vps server so there are servercontainers on it

I hope not then I have to add every container menualy when creating a
new one

Where is the big problem?
Not a big problem just something to think abouth when I add a new virtual
server(container)



------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

Hallo, Steffan,

Du meintest am 29.11.11:

What do you mean with "server container"?

The main server is a vps server so there are servercontainers on it

Are these containers 24/7 online (from the "rsnapshot" view point)?

I hope not then I have to add every container menualy when creating
a new one

Where is the big problem?

Not a big problem just something to think about when I add a new
virtual server(container)

You can include config files into the rsnapshot configuration, and you
can work with the (rsync) options "include-file" and "exclude-file".

Viele Gruesse!
Helmut

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

Hallo, Steffan,

Du meintest am 30.11.11:

Well i think i realy have to start splitting up my backups here is a
log of one run

[29/Nov/2011:18:01:01] /usr/local/bin/rsnapshot -c
/etc/rsnapshot.config/server1.conf hourly: started
[29/Nov/2011:18:01:01] echo 28355 > /var/run/rsnapshot.server1.pid
[29/Nov/2011:18:01:01] mv /extern1/server1/hourly.1/
/extern1/server1/hourly.2/

Ok.

[29/Nov/2011:18:01:01] /bin/cp -al /extern1/server1/hourly.0
/extern1/server1/hourly.1
[29/Nov/2011:21:58:17] /usr/bin/rsync -av --stats --rsh=/usr/bin/ssh
root < at > 192.168.1.31:/etc/sysconfig/vz-scripts/
/extern1/server1/hourly.0/...

What tells

mount | grep extern1
df | grep extern1

Is "/extern/server1" a partition on a disk in that machine which runs
"rsnapshot"? "extern" sounds like a target outside of this machine.

Viele Gruesse!
Helmut

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

Hello Helmut

mount | grep extern1
/dev/sdc1 on /extern1 type ext3 (rw)

df -h | grep extern1
/dev/sdc1 2.0T 1.1T 795G 59% /extern1


No i called it extern1
This machine has 2 build in sata disks for the os (running rsnapshot)
And a hardware raid controller running the extern disks
With 2 partitions of 2 TB


# Ch# ModelName Capacity Usage
============================================================================
===
1 1 WDC WD5002ABYS-02B1B0 500.1GB Raid Set # 00
2 2 ST3500320AS 500.1GB Raid Set # 00
3 3 ST3500320AS 500.1GB Raid Set # 00
4 4 ST3500320AS 500.1GB Raid Set # 00
5 5 ST3500320AS 500.1GB Raid Set # 00
6 6 WDC WD5002ABYS-02B1B0 500.1GB Raid Set # 00
7 7 WDC WD5002ABYS-02B1B0 500.1GB Raid Set # 00
8 8 WDC WD5002ABYS-02B1B0 500.1GB Raid Set # 00
9 9 WDC WD5002ABYS-02B1B0 500.1GB Free
10 10 WDC WD5002ABYS-02B1B0 500.1GB Raid Set # 00

ARC-1261-VOL#00 Raid Set # 00 Raid6 4000.0GB

Hallo, Steffan,

Du meintest am 30.11.11:

Well i think i realy have to start splitting up my backups here is a
log of one run

[29/Nov/2011:18:01:01] /usr/local/bin/rsnapshot -c
/etc/rsnapshot.config/server1.conf hourly: started
[29/Nov/2011:18:01:01] echo 28355 > /var/run/rsnapshot.server1.pid
[29/Nov/2011:18:01:01] mv /extern1/server1/hourly.1/
/extern1/server1/hourly.2/

Ok.

[29/Nov/2011:18:01:01] /bin/cp -al /extern1/server1/hourly.0
/extern1/server1/hourly.1
[29/Nov/2011:21:58:17] /usr/bin/rsync -av --stats --rsh=/usr/bin/ssh
root < at > 192.168.1.31:/etc/sysconfig/vz-scripts/
/extern1/server1/hourly.0/...

What tells

mount | grep extern1
df | grep extern1

Is "/extern/server1" a partition on a disk in that machine which runs
"rsnapshot"? "extern" sounds like a target outside of this machine.
<<<<<<<<<<<<<<<<<<<<<<<<


------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

Hallo, Steffan,

Du meintest am 30.11.11:

What tells

mount | grep extern1
df | grep extern1

mount | grep extern1
/dev/sdc1 on /extern1 type ext3 (rw)

df -h | grep extern1
/dev/sdc1 2.0T 1.1T 795G 59% /extern1

Ok.

Next step (I had asked you for these informations yesterday):

When "rsnapshot" has finished its "hourly" job (you can see it in the
log file):

rsnapshot diff > /tmp/rsnap-diff.txt

rsnapshot du > /tmp/rsnap-du.txt

and then please show at least the file "/tmp/rsnap-du.txt"

Maybe "rsnap-diff.txt" has many lines: then you shouldn't put it into
this mailing list. but

wc -l /tmp/rsnap-diff.txt

shows the number of the files which couldn't be hard linked.

Both jobs need some time; perhaps you can stop "rsnapshot du" via

killall rsnapshot

when you can see the entries for the hourly jobs and for some daily
jobs.

Viele Gruesse!
Helmut

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss