Backup Central

Hallo, Helmut,

So the logrotating takes now 45 minutes
Rsync takes almost 2 hours
So the problem is still rsync
One small hope: maybe the next run is quicker - the very first run may
need more time than all the following runs.

No I didn't delete the backup so I was not the first run

Maybe your configuration with one *.conf for many servers and one backup
directory for all the backups makes the problems ...
I prefer one separate directory for every machine. p.e.
/srv/backup1/
/srv/backup2/

I have for every server a own .conf file
/extern1 and /extern2 are hardware raid
ARC-1261-VOL#00 Raid Set # 00 Raid6 4000.
Holding 10 * 500 Gb sata disks
Server1.conf /extern2/server1
Server2.conf /extern2/server2
server3.conf /extern1/server3
server4.conf /extern1/server4
server5.conf /extern1/server5

calling it with crontab
01 18,2,10 * * * root /usr/local/bin/rsnapshot -c
/etc/rsnapshot.config/server1.conf hourly 2>&1 |
/usr/local/bin/rsnapreport.pl | mail -s ...
01 20,4,12 * * * root /usr/local/bin/rsnapshot -c
/etc/rsnapshot.config/server2.conf hourly 2>&1 |
/usr/local/bin/rsnapreport.pl | mail -s ...
01 22,6,14 * * * root /usr/local/bin/rsnapshot -c
/etc/rsnapshot.config/server3.conf hourly 2>&1 |
/usr/local/bin/rsnapreport.pl | mail -s ...
01 0,8,16 * * * root /usr/local/bin/rsnapshot -c
/etc/rsnapshot.config/server4.conf hourly 2>&1 |
/usr/local/bin/rsnapreport.pl | mail -s ...
01 3,11,19 * * * root /usr/local/bin/rsnapshot -c
/etc/rsnapshot.config/server5.conf hourly 2>&1 |
/usr/local/bin/rsnapreport.pl | mail -s ...

But rsync takes so long that I think I have to go for speed and go back to 2
backups a day so there is more time between the crons
Mayby raid is to slow and I have to go for another setup
Can test it by changing one directory to the internal OS disk (raid 1) see
what happens after a couple of runs
Other option is to disable all raid, and manually move daily.X to another
disk for failsave

What is your setup ?


Steffan


------------------------------------------------------------------------------
Cloud Services Checklist: Pricing and Packaging Optimization
This white paper is intended to serve as a reference, checklist and point of
discussion for anyone considering optimizing the pricing and packaging model
of a cloud services business. Read Now!
http://www.accelacomm.com/jaw/sfnl/114/51491232/
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

Hallo, Steffan,

Du meintest am 07.12.11:

[...]

But rsync takes so long that I think I have to go for speed and go
back to 2 backups a day so there is more time between the crons

No - there must be another reason than "rsync" itself.

My system (backing up from a disk in the machine to another disk in the
machine):

[06/Dec/2011:20:40:02] /usr/bin/rsnapshot -c /etc/rsnapshot.conf hourly:
started
[06/Dec/2011:20:40:02] echo 26316 > /var/run/rsnap-Arktur.pid
[06/Dec/2011:20:40:02] mv /srv/backup/Arktur/hourly.1/ /srv/backup/Arktur/_delete.26316/
[06/Dec/2011:20:40:02] mv /srv/backup/Arktur/hourly.0/ /srv/backup/Arktur/hourly.1/
[06/Dec/2011:20:40:02] mkdir -m 0755 -p /srv/backup/Arktur/hourly.0/
[06/Dec/2011:20:40:02] /usr/bin/rsync -ax --delete --numeric-ids --relative --delete-excluded --include-from=/etc/rsnapshot.d/include-file --exclude-from=/etc/rsnapshot.d/exclude-file --exclude=srv/backup/Arktur/ --link-dest=/srv/backup/Arktur/hourly.1/Wurzel/ /. /srv/backup/Arktur/hourly.0/Wurzel/
[06/Dec/2011:20:42:49] /usr/bin/rsync -ax --delete --numeric-ids --relative --delete-excluded --include-from=/etc/rsnapshot.d/include-file --exclude-from=/etc/rsnapshot.d/exclude-file --link-dest=/srv/backup/Arktur/hourly.1/./ /var /srv/backup/Arktur/hourly.0/./
[06/Dec/2011:20:43:32] /usr/bin/rsync -ax --delete --numeric-ids --relative --delete-excluded --include-from=/etc/rsnapshot.d/include-file --exclude-from=/etc/rsnapshot.d/exclude-file --link-dest=/srv/backup/Arktur/hourly.1/./ /home /srv/backup/Arktur/hourly.0/./
[06/Dec/2011:20:53:31] touch /srv/backup/Arktur/hourly.0/
[06/Dec/2011:20:53:31] rm -f /var/run/rsnap-Arktur.pid
[06/Dec/2011:20:53:31] /bin/rm -rf /srv/backup/Arktur/_delete.26316
[06/Dec/2011:21:02:17] /usr/bin/rsnapshot -c /etc/rsnapshot.conf hourly: completed successfully


with backup sizes (rsnapshot du):

141G /srv/backup/Arktur/hourly.0/
345M /srv/backup/Arktur/hourly.1/
2,7G /srv/backup/Arktur/daily.0/
349M /srv/backup/Arktur/daily.1/
339M /srv/backup/Arktur/daily.2/
366M /srv/backup/Arktur/daily.3/
952M /srv/backup/Arktur/daily.4/
70G /srv/backup/Arktur/daily.5/
367M /srv/backup/Arktur/daily.6/
2,6G /srv/backup/Arktur/weekly.0/
1,6G /srv/backup/Arktur/weekly.1/

and (p.e.)

backup / Wurzel/
backup /var/ ./
backup /home/ ./

You can see here: backing up the "home" partition with about 150 GByte needs about 10 minutes, using "rsync".
And deleting the safety (renamed) backup _delete.* also needs about 10
minutes.

"/srv/backup" belongs to "/dev/sdb1" (Samsung SATA),

hdparm -t /dev/sdb1

tells a read speed of about 60 MByte/s

I prefer a configuration with

link_dest 1
one_fs 1
use_lazy_deletes 1


Mayby raid is to slow and I have to go for another setup
Can test it by changing one directory to the internal OS disk (raid
1) see what happens after a couple of runs

Other option is to disable all raid, and manually move daily.X to
another disk for failsave

No really good idea ...

Another chaotic idea: can you add an USB disk with at least about 250
GByte and copy and modify your "rsnapshot.conf" so that it uses not (or
not only) the RAID but (additionally) the USB disk?

It can't run slower ...

Viele Gruesse!
Helmut

------------------------------------------------------------------------------
Cloud Services Checklist: Pricing and Packaging Optimization
This white paper is intended to serve as a reference, checklist and point of
discussion for anyone considering optimizing the pricing and packaging model
of a cloud services business. Read Now!
http://www.accelacomm.com/jaw/sfnl/114/51491232/
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

link_dest 1
one_fs 1
use_lazy_deletes 1
I made these changes


Other option is to disable all raid, and manually move daily.X to
another disk for failsave

No really good idea ...
Do you meen not a good idéa ?

Another chaotic idea: can you add an USB disk with at least about 250
GByte and copy and modify your "rsnapshot.conf" so that it uses not (or
not only) the RAID but (additionally) the USB disk?

Im 2 hour drive from the server so no option on this time

I have now run a couple of backups to the internail raid1 disk with your
options

[08/Dec/2011:12:01:01] /usr/local/bin/rsnapshot -c
/etc/rsnapshot.config/server1.conf hourly: started
[08/Dec/2011:12:01:01] echo 11402 > /var/run/server1.com.pid
[08/Dec/2011:12:01:01] mv /home/hourly.2/ /home/_delete.11402/
[08/Dec/2011:12:01:01] mv /home/hourly.1/ /home/hourly.2/
[08/Dec/2011:12:01:01] mv /home/hourly.0/ /home/hourly.1/
[08/Dec/2011:12:01:01] mkdir -m 0755 -p
/home/hourly.0/server1/etc/sysconfig/
[08/Dec/2011:12:01:01] /usr/bin/rsync -ax --delete --rsh=/usr/bin/ssh
--link-dest=/home/hourly.1/server1/etc/sysconfig/vz-scripts/ root < at > 192.168...
[08/Dec/2011:12:01:02] mkdir -m 0755 -p /home/hourly.0/server1/vz/
[08/Dec/2011:12:01:02] /usr/bin/rsync -ax --delete --rsh=/usr/bin/ssh
--link-dest=/home/hourly.1/server1/vz/private/ root < at > 192.168.1.....
[08/Dec/2011:13:07:17] touch /home/hourly.0/
[08/Dec/2011:13:07:17] rm -f /var/run/rsnapshot.server1.pid
[08/Dec/2011:13:07:17] /bin/rm -rf /home/_delete.11402
[08/Dec/2011:13:31:11] /usr/local/bin/rsnapshot -c
/etc/rsnapshot.config/server1.conf hourly: completed

So the backup is done in one hour not so fast as your backup but faster then
that it was
Will change it back to the raid 6 disks and let it run for a couple of days
see what happends



------------------------------------------------------------------------------
Cloud Services Checklist: Pricing and Packaging Optimization
This white paper is intended to serve as a reference, checklist and point of
discussion for anyone considering optimizing the pricing and packaging model
of a cloud services business. Read Now!
http://www.accelacomm.com/jaw/sfnl/114/51491232/
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

Hallo, Steffan,

Du meintest am 08.12.11:

Other option is to disable all raid, and manually move daily.X to
another disk for failsave

No really good idea ...

Do you mean not a good idéa ?

That's what I wanted to say ...

I have now run a couple of backups to the internail raid1 disk with
your options

[08/Dec/2011:12:01:01] /usr/local/bin/rsnapshot -c
/etc/rsnapshot.config/server1.conf hourly: started
[08/Dec/2011:12:01:01] echo 11402 > /var/run/server1.com.pid
[08/Dec/2011:12:01:01] mv /home/hourly.2/ /home/_delete.11402/
[08/Dec/2011:12:01:01] mv /home/hourly.1/ /home/hourly.2/
[08/Dec/2011:12:01:01] mv /home/hourly.0/ /home/hourly.1/
[08/Dec/2011:12:01:01] mkdir -m 0755 -p /home/hourly.0/server1/etc/sysconfig/
[08/Dec/2011:12:01:01] /usr/bin/rsync -ax --delete --rsh=/usr/bin/ssh
--link-dest=/home/hourly.1/server1/etc/sysconfig/vz-scripts/
root < at > 192.168....
[08/Dec/2011:12:01:02] mkdir -m 0755 -p /home/hourly.0/server1/vz/
[08/Dec/2011:12:01:02] /usr/bin/rsync -ax --delete --rsh=/usr/bin/ssh
--link-dest=/home/hourly.1/server1/vz/private/ root < at > 192.168.1.....
[08/Dec/2011:13:07:17] touch /home/hourly.0/
[08/Dec/2011:13:07:17] rm -f /var/run/rsnapshot.server1.pid
[08/Dec/2011:13:07:17] /bin/rm -rf /home/_delete.11402
[08/Dec/2011:13:31:11] /usr/local/bin/rsnapshot
-c /etc/rsnapshot.config/server1.conf hourly: completed

So the backup is done in one hour not so fast as your backup but
faster then that it was

Looks much better (but still not really good ...)
Especially: you can start the next "rsnapshot" job after about 1 hour;
the "/bin/rm -rf /home/_delete..." job doesn't block such a next run.

Viele Gruesse!
Helmut

------------------------------------------------------------------------------
Cloud Services Checklist: Pricing and Packaging Optimization
This white paper is intended to serve as a reference, checklist and point of
discussion for anyone considering optimizing the pricing and packaging model
of a cloud services business. Read Now!
http://www.accelacomm.com/jaw/sfnl/114/51491232/
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss