Backup Central

I am attempting to move an rsnapshot tree to a larger drive. My rsnapshot tree is made up of 7 daily backups and 191 (so far) weekly backups. daily.0 is ~138GB. rsnapshot-copy output for daily.0 is:
Number of files: 1012292
Number of files transferred: 935061
Total file size: 146189280898 bytes
Total transferred file size: 146189280187 bytes
Literal data: 146189280187 bytes
Matched data: 0 bytes
File list size: 21856826
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 146268656671
Total bytes received: 18075021

From reading the mailing list, it seems that rsnapshot-copy is the preferred solution to move an rsnapshot tree from one drive to another. Unfortunately, it seems that it will take multiple days to complete making my copy out of date before rsnapshot-copy is done. Both drives are connected via SATA.

Any suggestions or should I just let the rsnapshot-copy complete and ignore the fact that the copy is slightly out-of-date?

- Charlie
--

Charlie Steaderman
charlies < at > poliac.com ([email]charlies < at > poliac.com[/email])
VP Engineering
Poliac Research Corporation
Phone: 952.707.6245
Cell: 612.242.6364

Hallo, Charles,

Du meintest am 15.08.11:

From reading the mailing list, it seems that rsnapshot-copy is
the preferred solution to move an rsnapshot tree from one drive
to another. Unfortunately, it seems that it will take multiple
days to complete making my copy out of date before rsnapshot-copy
is done. Both drives are connected via SATA.

(Please don't use HTML - thank you!)

I prefer copying with

Quelle=/srv/backup
Ziel=/Pfad/zu/neuem/Backup
mkdir -p $Ziel
rsync -axH ${Quelle}/. $Ziel

See

http://arktur.de/Wiki/Zusatzprogramme:rsnapshot#Sicherung_kopieren

Viele Gruesse!
Helmut

------------------------------------------------------------------------------
uberSVN's rich system and user administration capabilities and model
configuration take the hassle out of deploying and managing Subversion and
the tools developers use with it. Learn more about uberSVN and get a free
download at: http://p.sf.net/sfu/wandisco-dev2dev
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

On Mon, Aug 15, 2011 at 11:56:00AM -0500, Charles Steaderman wrote:
I am attempting to move an rsnapshot tree to a larger drive. My
rsnapshot tree is made up of 7 daily backups and 191 (so far) weekly
backups. daily.0 is ~138GB. rsnapshot-copy output for daily.0 is:
Number of files: 1012292
Number of files transferred: 935061
Total file size: 146189280898 bytes
Total transferred file size: 146189280187 bytes
Literal data: 146189280187 bytes
Matched data: 0 bytes
File list size: 21856826
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 146268656671
Total bytes received: 18075021
From reading the mailing list, it seems that rsnapshot-copy is the
preferred solution to move an rsnapshot tree from one drive to another.
Unfortunately, it seems that it will take multiple days to complete
making my copy out of date before rsnapshot-copy is done. Both drives
are connected via SATA.
Any suggestions or should I just let the rsnapshot-copy complete and
ignore the fact that the copy is slightly out-of-date?

You might also consider a sector-level copy, which will probably be
much faster but will require somewhat more work and be somewhat more
sensitive.

Assuming your current tree is under /dev/sdx1 mounted as /rsnapshot
and is ext3, and the new drive is empty and is /dev/sdy, you can create
a partition /dev/sdy1 as large as you want, then:
umount /dev/sdx1
dd if=/dev/sdx1 of=/dev/sdy1 bs=1024k
resize2fs /dev/sdy1
then mount it etc. Similar stuff are applicable if it's lvm and not
simple partitions.

Good luck,
--
Didi


------------------------------------------------------------------------------
uberSVN's rich system and user administration capabilities and model
configuration take the hassle out of deploying and managing Subversion and
the tools developers use with it. Learn more about uberSVN and get a free
download at: http://p.sf.net/sfu/wandisco-dev2dev
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

On Mon, Aug 15, 2011 at 10:42:41PM +0300, Yedidyah Bar-David (didi+rsnapshot < at > isoc.org.il) wrote:

On Mon, Aug 15, 2011 at 11:56:00AM -0500, Charles Steaderman wrote:
I am attempting to move an rsnapshot tree to a larger drive.

You might also [...]
Similar stuff are applicable if it's lvm and not simple partitions.

Of course, if it's LVM the migration can be done with LVM tools
even while backups are being done at the same time, however long
it may take:

install new disk, pvcreate, vgextend, pvmove, vgreduce, remove old disk,
lvextend, resize2fs (or mount -o remount,resize for jfs, &c).

The time-consuming part is pvmove, but it will happily go on
in the background while the system is being used
(even through reboots and crashes).

With hotswap disks that can be done without ever booting
the machine. (Indeed even the system disk can be replaced
without booting, although that takes a few extra steps.)
This convenience alone justifies using LVM in systems
likely to grow out of their disks, IMHO.
(If you aren't using LVM, now would be a good time to
start, when you are replacing the disk anyway.)

Similarly, with RAID arrays the disks can be replaced one by one,
letting the array synchronize after each disk swap, and finally
using the extra space with mdadm --grow --size=... and resize2fs.

With a single non-LVM disk there's no such well-supported way
to move the data while it's being used at the same time.

(For the adventurous, one way it can be done: turn the disk into
a degraded 1-disk RAID1 array with mdadm ... --force --assume-clean,
add the new disk, synchronize, remove old disk, ...
But I wouldn't recommend that to anyone without considerable
experience with mdadm.)

--
Tapani Tarvainen

------------------------------------------------------------------------------
uberSVN's rich system and user administration capabilities and model
configuration take the hassle out of deploying and managing Subversion and
the tools developers use with it. Learn more about uberSVN and get a free
download at: http://p.sf.net/sfu/wandisco-dev2dev
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

On Monday 15 August 2011, Helmut Hullen wrote:
I prefer copying with

Quelle=/srv/backup
Ziel=/Pfad/zu/neuem/Backup
mkdir -p $Ziel
rsync -axH ${Quelle}/. $Ziel

Beside -H|--hard-links I would also use rsync's options
--sparse --xattrs --acls
(also within rsnapshot.conf, don't remember if these ones aren't even
set by default).

cu,
Rudi

------------------------------------------------------------------------------
uberSVN's rich system and user administration capabilities and model
configuration take the hassle out of deploying and managing Subversion and
the tools developers use with it. Learn more about uberSVN and get a free
download at: http://p.sf.net/sfu/wandisco-dev2dev
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

On Mon, Aug 15, 2011 at 12:56 PM, Charles Steaderman
<charlies < at > poliac.com> wrote:
I am attempting to move an rsnapshot tree to a larger drive. My rsnapshot
tree is made up of 7 daily backups and 191 (so far) weekly backups. daily.0
is ~138GB. rsnapshot-copy output for daily.0 is:

OK, I already predict a few things to look for.

* The initial compy can be done with whatever transfer tool you like.
If you use SELinux, and want to preserve the relevent setings, use
"star".

* "cp -a" is also fabulous for the initial copy on the same machine,
not so good if the transfer is interrupted or needs to be verified. It
does *not* transfer the SELinux settings, so beware of this.

* rsync is fabulous for verifying the final copy, but for such a large
number of hard-linked targets, you really, really want to have a
recent version of rsync. Make sure it's up to date: the rsync-2.6 on
RHEL 5.0, for example, has its performance wildly improved by grabbing
the rsync-3.x one from RHEL 5.6.

* Seriously consider reducing the number of weekly backups, switching
your rotations to provide a "monthly" backup, if feasible. It will
make ordinary recover easier, and the additional inodes for the
directories and changed files of all the weekly snapshots are merly
cheap. They are not free, and I'm slightly concerned that you will run
out of inodes if you do not tune your filesystems appropriately.

Number of files: 1012292
Number of files transferred: 935061
Total file size: 146189280898 bytes
Total transferred file size: 146189280187 bytes
Literal data: 146189280187 bytes
Matched data: 0 bytes
File list size: 21856826
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 146268656671
Total bytes received: 18075021

From reading the mailing list, it seems that rsnapshot-copy is the preferred
solution to move an rsnapshot tree from one drive to another. Unfortunately,
it seems that it will take multiple days to complete making my copy out of
date before rsnapshot-copy is done. Both drives are connected via SATA.

Any suggestions or should I just let the rsnapshot-copy complete and ignore
the fact that the copy is slightly out-of-date?

- Charlie
--

Charlie Steaderman
charlies < at > poliac.com
VP Engineering
Poliac Research Corporation
Phone: 952.707.6245
Cell: 612.242.6364


------------------------------------------------------------------------------
uberSVN's rich system and user administration capabilities and model
configuration take the hassle out of deploying and managing Subversion and
the tools developers use with it. Learn more about uberSVN and get a free
download at:  http://p.sf.net/sfu/wandisco-dev2dev

_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss



------------------------------------------------------------------------------
uberSVN's rich system and user administration capabilities and model
configuration take the hassle out of deploying and managing Subversion and
the tools developers use with it. Learn more about uberSVN and get a free
download at: http://p.sf.net/sfu/wandisco-dev2dev
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

Thanks for the heads up. A few additional pieces of information.

1) Running on FreeBSD 6.4 (FreeNAS) with UFS filesystem
2) Using rsync 3.0.4
3) Your comment about inodes really got me a bit worried as I hadn't
considered it as a possible issue. Here is the information for the
nearly full drive:
Filesystem 1K-blocks Used Avail Capacity iused ifree
%iused Mounted on
/dev/ar0p1 605459514 503472972 53549782 90% 14822774 63440520
19% /mnt/snapshots
Seems I should be OK.

Presently using Helmut's suggested rsync solution. Initially ran into an
out of memory issue (1GB ram, no swap file), but I created a 4GB swap
file and I am trying again. I will post my results to the list.

- Charlie

On 8/16/2011 11:06 AM, Nico Kadel-Garcia wrote:
On Mon, Aug 15, 2011 at 12:56 PM, Charles Steaderman
<charlies < at > poliac.com> wrote:
I am attempting to move an rsnapshot tree to a larger drive. My rsnapshot
tree is made up of 7 daily backups and 191 (so far) weekly backups. daily.0
is ~138GB. rsnapshot-copy output for daily.0 is:
OK, I already predict a few things to look for.

* The initial compy can be done with whatever transfer tool you like.
If you use SELinux, and want to preserve the relevent setings, use
"star".

* "cp -a" is also fabulous for the initial copy on the same machine,
not so good if the transfer is interrupted or needs to be verified. It
does *not* transfer the SELinux settings, so beware of this.

* rsync is fabulous for verifying the final copy, but for such a large
number of hard-linked targets, you really, really want to have a
recent version of rsync. Make sure it's up to date: the rsync-2.6 on
RHEL 5.0, for example, has its performance wildly improved by grabbing
the rsync-3.x one from RHEL 5.6.

* Seriously consider reducing the number of weekly backups, switching
your rotations to provide a "monthly" backup, if feasible. It will
make ordinary recover easier, and the additional inodes for the
directories and changed files of all the weekly snapshots are merly
cheap. They are not free, and I'm slightly concerned that you will run
out of inodes if you do not tune your filesystems appropriately.

Number of files: 1012292
Number of files transferred: 935061
Total file size: 146189280898 bytes
Total transferred file size: 146189280187 bytes
Literal data: 146189280187 bytes
Matched data: 0 bytes
File list size: 21856826
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 146268656671
Total bytes received: 18075021

From reading the mailing list, it seems that rsnapshot-copy is the preferred
solution to move an rsnapshot tree from one drive to another. Unfortunately,
it seems that it will take multiple days to complete making my copy out of
date before rsnapshot-copy is done. Both drives are connected via SATA.

Any suggestions or should I just let the rsnapshot-copy complete and ignore
the fact that the copy is slightly out-of-date?

- Charlie
--

Charlie Steaderman
charlies < at > poliac.com
VP Engineering
Poliac Research Corporation
Phone: 952.707.6245
Cell: 612.242.6364


------------------------------------------------------------------------------
uberSVN's rich system and user administration capabilities and model
configuration take the hassle out of deploying and managing Subversion and
the tools developers use with it. Learn more about uberSVN and get a free
download at: http://p.sf.net/sfu/wandisco-dev2dev

_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss




------------------------------------------------------------------------------
uberSVN's rich system and user administration capabilities and model
configuration take the hassle out of deploying and managing Subversion and
the tools developers use with it. Learn more about uberSVN and get a free
download at: http://p.sf.net/sfu/wandisco-dev2dev
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

On Tue, Aug 16, 2011 at 12:28 PM, Charles Steaderman
<charlies < at > poliac.com> wrote:
Thanks for the heads up. A few additional pieces of information.

1) Running on FreeBSD 6.4 (FreeNAS) with UFS filesystem
2) Using rsync 3.0.4
3) Your comment about inodes really got me a bit worried as I hadn't
considered it as a possible issue. Here is the information for the
nearly full drive:
Filesystem  1K-blocks       Used      Avail Capacity  iused     ifree
%iused  Mounted on
/dev/ar0p1   605459514 503472972   53549782    90% 14822774  63440520
19%   /mnt/snapshots
Seems I should be OK.

Presently using Helmut's suggested rsync solution. Initially ran into an
out of memory issue (1GB ram, no swap file), but I created a 4GB swap
file and I am trying again. I will post my results to the list.

- Charlie

Yeah, the preservation of all those ! < at > #$ < at > !# hardlinks gets tricky and
can consume the heck out of your RAM and swap on a big transfer.
That's where the rsnapshot-copy tool comes in really, really handy.

------------------------------------------------------------------------------
uberSVN's rich system and user administration capabilities and model
configuration take the hassle out of deploying and managing Subversion and
the tools developers use with it. Learn more about uberSVN and get a free
download at: http://p.sf.net/sfu/wandisco-dev2dev
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss