Deprecated: __autoload() is deprecated, use spl_autoload_register() instead in /home/pbuc/public_html/forum/mods/ext_phorummail/ezc/Base/src/ezc_bootstrap.php on line 36

Deprecated: Methods with the same name as their class will not be constructors in a future version of PHP; KeyCAPTCHA_CLASS has a deprecated constructor in /home/pbuc/public_html/forum/mods/keycaptcha/keycaptcha.php on line 108
Moving an rsnapshot root to a new server
Welcome! » Log In » Create A New Profile

Moving an rsnapshot root to a new server

Posted by Anonymous 
Moving an rsnapshot root to a new server
December 10, 2014 10:25AM
Hi rsnapshot wizards!

I’m moving our backups to a new server with more space, I’ve got about 5TB of data to move so it’ll take some time to migrate though I’m hoping to do it within 24 hours so it will not be copying during a backup.

From what I have read, the two options are a simple:

rsync -az -H --delete --numeric-ids /snapshots/ new.ser.ver.ip:/snapshots/

So using compression, archive, —delete (in case it needs to be re-run after a backup, which I’m not planning) and critically -H for the hard links.

Or there is a utility called rsnapshot-copy which looks like it’s supposed to do the job I’m attempting but I can’d find a lot of documentation around it and even fewer reported uses of it.

Is snapshot-copy simply designed to continue a copy when it’s been interrupted by a backup taking place (therefore the cycling of the directories will cause a normal rsync to be confused?)

I’m thinking that the safest bet would be to simply disable cron on the old server as soon as it’s completed a backup and then start the rsync, enabling cron on the new server when the copy is complete.

So my questions are:

1. Provided I can get the transfer done in-between backups, it that simple rsync command the correct one to use?
2. Is there any other reason to use the snapshot-copy utility?

Many thanks!

Chris.

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
Moving an rsnapshot root to a new server
December 10, 2014 11:47AM
I don't know about rsnapshot-copy, but there are at least two potential pitfalls you should be aware of:
[list][*]Both ends will need to have a table of directory entries and corresponding inode numbers in memory. Otherwise the hardlinks cannot be copied correctly. Depending on how many directory entries and inodes you have, you may want to add more memory to the servers or configure more swap. Normally, a server doesn't need much memory for running rsnapshot, but after some time you can end up with millions of directory entries.[*]The receiving side needs to keep the file system consistent which will require many seeks and small writes, e.g. for adding yet another link to an inode. So unless you have a fancy system with solid state drives and/or lots of NVRAM you will not be writing with anywhere near the maximal burst speed of your drives.[/list]
However, it should be safe for you to copy weekly, monthly, and/or yearly backups while the old system is still running. This will copy most of your data and may take a while. You can then disable the old backup system and copy the remaining files in hopefully very little time. Check when your backups are rotated before starting.

You may want to read my recent post to the coreutils mailing list where I describe my experience in copying a big set of files created with rsnapshot using the cp command:

[url=https://lists.gnu.org/archive/html/coreutils/2014-08/msg00012.html]https://lists.gnu.org/archive/html/coreutils/2014-08/msg00012.html[/url]

if the servers are well connected I wouldn't use compression. Your milage will vary.

Best,

Rasmus

[i]Intomics is a contract research organization specialized in deriving core biological insight from large scale data. We help our clients in the pharmaceutical industry develop tomorrow's medicines better, faster, and cheaper through optimized use of biomedical data.[/i]-----------------------------------------------------------------
Hansen, Rasmus Borup Intomics - from data to biology
System Administrator Diplomvej 377
Scientific Programmer DK-2800 Kgs. Lyngby
Denmark
E: rbh < at > intomics.com ([email]rbh < at > intomics.com[/email]) W: [url=http://www.intomics.com/]http://www.intomics.com/[/url]
P: +45 5167 7972 P: +45 8880 7979

[quote]On 10 Dec 2014, at 11:23, Chris Tebb <chris < at > chrispyfur.net ([email]chris < at > chrispyfur.net[/email])> wrote:
Hi rsnapshot wizards!I’m moving our backups to a new server with more space, I’ve got about 5TB of data to move so it’ll take some time to migrate though I’m hoping to do it within 24 hours so it will not be copying during a backup.From what I have read, the two options are a simple:rsync -az -H --delete --numeric-ids /snapshots/ new.ser.ver.ip:/snapshots/So using compression, archive, —delete (in case it needs to be re-run after a backup, which I’m not planning) and critically -H for the hard links.Or there is a utility called rsnapshot-copy which looks like it’s supposed to do the job I’m attempting but I can’d find a lot of documentation around it and even fewer reported uses of it.Is snapshot-copy simply designed to continue a copy when it’s been interrupted by a backup taking place (therefore the cycling of the directories will cause a normal rsync to be confused?)I’m thinking that the safest bet would be to simply disable cron on the old server as soon as it’s completed a backup and then start the rsync, enabling cron on the new server when the copy is complete.So my questions are:1. Provided I can get the transfer done in-between backups, it that simple rsync command the correct one to use?2. Is there any other reason to use the snapshot-copy utility?Many thanks!Chris.------------------------------------------------------------------------------Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Serverfrom Actuate! Instantly Supercharge Your Business Reports and Dashboardswith Interactivity, Sharing, Native Excel Exports, App Integration & moreGet technology previously reserved for billion-dollar corporations, FREE[url=http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk]http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk[/url]_______________________________________________rsnapshot-discuss mailing listrsnapshot-discuss < at > lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
[/quote]
Moving an rsnapshot root to a new server
December 10, 2014 11:58AM
There are multiple ways that I would handle this problem.

Option 1

1. start a new backup set, keep the old backup set for a period of time,
perhaps archive it, the new set will expand out naturally in time.

Option 2
2a. take a snapshot of the backup set, create a dump of the snapshot
(taking a dump of a live file system is fraught with danger, a snapshot
shouldn't be actively changing).

2b. restore the dump to the new location.

NB: building the dump may take a long time if there are many intervals
to consider. This is why I prefer option 1 -- option 1 gives an
opportunity for a clean start too.

Cheers
A.

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
Moving an rsnapshot root to a new server
December 10, 2014 12:00PM
Thanks Rasmus, both servers have top of the line Dell raid cards with all the extra nvram we could order so that should help mitigate the write issues a little. The old server has 12GB and the new 4GB, will both need extra memory (should I even them out to 8gb each) or is it the receiving server that needs it?
I hadn’t bought of syncing the monthly backup points first, I”m guessing I could rsync all the monthlies across first then just do the dailies once I’ve stopped the backups on the source, good idea!

Chris

[quote]On 10 Dec 2014, at 11:46, Rasmus Borup Hansen <rbh < at > intomics.com ([email]rbh < at > intomics.com[/email])> wrote:
I don't know about rsnapshot-copy, but there are at least two potential pitfalls you should be aware of:
[list][*]Both ends will need to have a table of directory entries and corresponding inode numbers in memory. Otherwise the hardlinks cannot be copied correctly. Depending on how many directory entries and inodes you have, you may want to add more memory to the servers or configure more swap. Normally, a server doesn't need much memory for running rsnapshot, but after some time you can end up with millions of directory entries.[*]The receiving side needs to keep the file system consistent which will require many seeks and small writes, e.g. for adding yet another link to an inode. So unless you have a fancy system with solid state drives and/or lots of NVRAM you will not be writing with anywhere near the maximal burst speed of your drives.[/list]
However, it should be safe for you to copy weekly, monthly, and/or yearly backups while the old system is still running. This will copy most of your data and may take a while. You can then disable the old backup system and copy the remaining files in hopefully very little time. Check when your backups are rotated before starting.

You may want to read my recent post to the coreutils mailing list where I describe my experience in copying a big set of files created with rsnapshot using the cp command:

[url=https://lists.gnu.org/archive/html/coreutils/2014-08/msg00012.html]https://lists.gnu.org/archive/html/coreutils/2014-08/msg00012.html[/url]

if the servers are well connected I wouldn't use compression. Your milage will vary.

Best,

Rasmus

[i]Intomics is a contract research organization specialized in deriving core biological insight from large scale data. We help our clients in the pharmaceutical industry develop tomorrow's medicines better, faster, and cheaper through optimized use of biomedical data.[/i]-----------------------------------------------------------------
Hansen, Rasmus Borup Intomics - from data to biology
System Administrator Diplomvej 377
Scientific Programmer DK-2800 Kgs. Lyngby
Denmark
E: rbh < at > intomics.com ([email]rbh < at > intomics.com[/email]) W: [url=http://www.intomics.com/]http://www.intomics.com/[/url]
P: +45 5167 7972 P: +45 8880 7979

[quote]On 10 Dec 2014, at 11:23, Chris Tebb <chris < at > chrispyfur.net ([email]chris < at > chrispyfur.net[/email])> wrote:
Hi rsnapshot wizards!I’m moving our backups to a new server with more space, I’ve got about 5TB of data to move so it’ll take some time to migrate though I’m hoping to do it within 24 hours so it will not be copying during a backup.From what I have read, the two options are a simple:rsync -az -H --delete --numeric-ids /snapshots/ new.ser.ver.ip:/snapshots/So using compression, archive, —delete (in case it needs to be re-run after a backup, which I’m not planning) and critically -H for the hard links.Or there is a utility called rsnapshot-copy which looks like it’s supposed to do the job I’m attempting but I can’d find a lot of documentation around it and even fewer reported uses of it.Is snapshot-copy simply designed to continue a copy when it’s been interrupted by a backup taking place (therefore the cycling of the directories will cause a normal rsync to be confused?)I’m thinking that the safest bet would be to simply disable cron on the old server as soon as it’s completed a backup and then start the rsync, enabling cron on the new server when the copy is complete.So my questions are:1. Provided I can get the transfer done in-between backups, it that simple rsync command the correct one to use?2. Is there any other reason to use the snapshot-copy utility?Many thanks!Chris.------------------------------------------------------------------------------Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Serverfrom Actuate! Instantly Supercharge Your Business Reports and Dashboardswith Interactivity, Sharing, Native Excel Exports, App Integration & moreGet technology previously reserved for billion-dollar corporations, FREErsnapshot-discuss < at > lists.sourceforge.net ([email]rsnapshot-discuss < at > lists.sourceforge.net[/email])https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
[/quote]

[/quote]
--
[url=http://chrispyfur.net]chrispyfur.net[/url]
Moving an rsnapshot root to a new server
December 10, 2014 12:01PM
Hallo, Chris,

Du meintest am 10.12.14:

[quote]I?m moving our backups to a new server with more space, I?ve got
about 5TB of data to move so it?ll take some time to migrate though
I?m hoping to do it within 24 hours so it will not be copying during
a backup.
[/quote]
[quote]From what I have read, the two options are a simple:
[/quote]
[quote]rsync -az -H --delete --numeric-ids /snapshots/
new.ser.ver.ip:/snapshots/
[/quote]
[quote]So using compression, archive, ?delete (in case it needs to be re-run
after a backup, which I?m not planning) and critically -H for the
hard links.
[/quote]
[quote]Or there is a utility called rsnapshot-copy which looks like it?s
supposed to do the job
[/quote]
I have used it some years ago. It works, but the above "rsync" command
does the job without such a special program.

[...]

[quote]So my questions are:
[/quote]
[quote]1. Provided I can get the transfer done in-between backups, it that
simple rsync command the correct one to use?
[/quote]
Yes.

[quote]2. Is there any other
reason to use the snapshot-copy utility?
[/quote]
No.

Viele Gruesse!
Helmut

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
Moving an rsnapshot root to a new server
December 10, 2014 12:04PM
Yeah, I did wonder about us starting again with a new backup set and copying the old set to a NAS so we can repurpose the old server.
That’s a lot of data to pull down though! I might even leave both servers backing up whilst the new one takes its first copy (which I can do with rsnapshot sync can I?)

Chris.[quote]On 10 Dec 2014, at 11:57, Andrew McGlashan <andrew.mcglashan < at > affinityvision.com.au ([email]andrew.mcglashan < at > affinityvision.com.au[/email])> wrote:
There are multiple ways that I would handle this problem.Option 11. start a new backup set, keep the old backup set for a period of time,perhaps archive it, the new set will expand out naturally in time.Option 22a. take a snapshot of the backup set, create a dump of the snapshot(taking a dump of a live file system is fraught with danger, a snapshotshouldn't be actively changing).2b. restore the dump to the new location.NB: building the dump may take a long time if there are many intervalsto consider. This is why I prefer option 1 -- option 1 gives anopportunity for a clean start too.CheersA.------------------------------------------------------------------------------Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Serverfrom Actuate! Instantly Supercharge Your Business Reports and Dashboardswith Interactivity, Sharing, Native Excel Exports, App Integration & moreGet technology previously reserved for billion-dollar corporations, FREE[url=http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk]http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk[/url]_______________________________________________rsnapshot-discuss mailing listrsnapshot-discuss < at > lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
[/quote]
--
[url=http://chrispyfur.net]chrispyfur.net[/url]
Moving an rsnapshot root to a new server
December 10, 2014 12:23PM
On Wed, Dec 10, 2014 at 10:23:57AM +0000, Chris Tebb wrote:

[quote]I???m moving our backups to a new server with more space, I???ve got about 5TB of data to move so it???ll take some time to migrate though I???m hoping to do it within 24 hours so it will not be copying during a backup.

From what I have read, the two options are a simple:

rsync -az -H --delete --numeric-ids /snapshots/ new.ser.ver.ip:/snapshots/

So using compression, archive, ???delete (in case it needs to be re-run after a backup, which I???m not planning) and critically -H for the hard links.
[/quote]
I'd add -S so that it will create sparse files if appropriate, saving
some I/O.

Also, don't bank on scanning a 5TB filesystem and copying it across a
network in under 24 hours. Use rsync 3 at both ends if possible as I
believe that it will help with this by doing the scan and transfer in
parallel.

--
David Cantrell | Nth greatest programmer in the world

Today's previously unreported paraphilia is tomorrow's Internet sensation

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
Moving an rsnapshot root to a new server
December 10, 2014 01:29PM
Both sides need some kind of data structure to remember which filenames correspond to which inodes, so that the right links can be created. I don't know how rsync implements this, but you could just go ahead and try rsync'ing a single monthly directory, then two, and then three to see how the memory usage depends on the number of files. If you write

/usr/bin/time -v -o rsync.time rsync ...

where ... are the arguments you'll usually give to rsync, then the file rsync.time will have information on how many resource the command used.

Another way to get an estimate of the order of magnitude of the required memory would be to simply count the number of bytes in the list of all the filenames:

find /backup | wc -c

Replace /backup with the path to your rsnapshot files. If it's much less than the memory of your server, then don't worry about it.

Best,

Rasmus
[i]Intomics is a contract research organization specialized in deriving core biological insight from large scale data. We help our clients in the pharmaceutical industry develop tomorrow's medicines better, faster, and cheaper through optimized use of biomedical data.[/i]-----------------------------------------------------------------
Hansen, Rasmus Borup Intomics - from data to biology
System Administrator Diplomvej 377
Scientific Programmer DK-2800 Kgs. Lyngby
Denmark
E: rbh < at > intomics.com ([email]rbh < at > intomics.com[/email]) W: [url=http://www.intomics.com/]http://www.intomics.com/[/url]
P: +45 5167 7972 P: +45 8880 7979

[quote]On 10 Dec 2014, at 12:59, Chris Tebb <chris < at > chrispyfur.net ([email]chris < at > chrispyfur.net[/email])> wrote:
Thanks Rasmus, both servers have top of the line Dell raid cards with all the extra nvram we could order so that should help mitigate the write issues a little. The old server has 12GB and the new 4GB, will both need extra memory (should I even them out to 8gb each) or is it the receiving server that needs it?
I hadn’t bought of syncing the monthly backup points first, I”m guessing I could rsync all the monthlies across first then just do the dailies once I’ve stopped the backups on the source, good idea!

Chris

[quote]On 10 Dec 2014, at 11:46, Rasmus Borup Hansen <rbh < at > intomics.com ([email]rbh < at > intomics.com[/email])> wrote:
I don't know about rsnapshot-copy, but there are at least two potential pitfalls you should be aware of:
[list][*]Both ends will need to have a table of directory entries and corresponding inode numbers in memory. Otherwise the hardlinks cannot be copied correctly. Depending on how many directory entries and inodes you have, you may want to add more memory to the servers or configure more swap. Normally, a server doesn't need much memory for running rsnapshot, but after some time you can end up with millions of directory entries.[*]The receiving side needs to keep the file system consistent which will require many seeks and small writes, e.g. for adding yet another link to an inode. So unless you have a fancy system with solid state drives and/or lots of NVRAM you will not be writing with anywhere near the maximal burst speed of your drives.[/list]
However, it should be safe for you to copy weekly, monthly, and/or yearly backups while the old system is still running. This will copy most of your data and may take a while. You can then disable the old backup system and copy the remaining files in hopefully very little time. Check when your backups are rotated before starting.

You may want to read my recent post to the coreutils mailing list where I describe my experience in copying a big set of files created with rsnapshot using the cp command:

[url=https://lists.gnu.org/archive/html/coreutils/2014-08/msg00012.html]https://lists.gnu.org/archive/html/coreutils/2014-08/msg00012.html[/url]

if the servers are well connected I wouldn't use compression. Your milage will vary.

Best,

Rasmus

[i]Intomics is a contract research organization specialized in deriving core biological insight from large scale data. We help our clients in the pharmaceutical industry develop tomorrow's medicines better, faster, and cheaper through optimized use of biomedical data.[/i]-----------------------------------------------------------------
Hansen, Rasmus Borup Intomics - from data to biology
System Administrator Diplomvej 377
Scientific Programmer DK-2800 Kgs. Lyngby
Denmark
E: rbh < at > intomics.com ([email]rbh < at > intomics.com[/email]) W: [url=http://www.intomics.com/]http://www.intomics.com/[/url]
P: +45 5167 7972 P: +45 8880 7979

[quote]On 10 Dec 2014, at 11:23, Chris Tebb <chris < at > chrispyfur.net ([email]chris < at > chrispyfur.net[/email])> wrote:
Hi rsnapshot wizards!I’m moving our backups to a new server with more space, I’ve got about 5TB of data to move so it’ll take some time to migrate though I’m hoping to do it within 24 hours so it will not be copying during a backup.From what I have read, the two options are a simple:rsync -az -H --delete --numeric-ids /snapshots/ new.ser.ver.ip:/snapshots/So using compression, archive, —delete (in case it needs to be re-run after a backup, which I’m not planning) and critically -H for the hard links.Or there is a utility called rsnapshot-copy which looks like it’s supposed to do the job I’m attempting but I can’d find a lot of documentation around it and even fewer reported uses of it.Is snapshot-copy simply designed to continue a copy when it’s been interrupted by a backup taking place (therefore the cycling of the directories will cause a normal rsync to be confused?)I’m thinking that the safest bet would be to simply disable cron on the old server as soon as it’s completed a backup and then start the rsync, enabling cron on the new server when the copy is complete.So my questions are:1. Provided I can get the transfer done in-between backups, it that simple rsync command the correct one to use?2. Is there any other reason to use the snapshot-copy utility?Many thanks!Chris.------------------------------------------------------------------------------Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Serverfrom Actuate! Instantly Supercharge Your Business Reports and Dashboardswith Interactivity, Sharing, Native Excel Exports, App Integration & moreGet technology previously reserved for billion-dollar corporations, FREErsnapshot-discuss < at > lists.sourceforge.net ([email]rsnapshot-discuss < at > lists.sourceforge.net[/email])[url=https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss]https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss[/url]
[/quote]

[/quote]
--
[url=http://chrispyfur.net/]chrispyfur.net[/url]

[/quote]
Moving an rsnapshot root to a new server
December 10, 2014 01:55PM
On Dec 10 12:02, David Cantrell (david < at > cantrell.org.uk) wrote:

[quote]Also, don't bank on scanning a 5TB filesystem and copying it across a
network in under 24 hours.
[/quote]
Yeah. If at all possible I'd try to physically move the disks from the
old machine to the new one temporarily and do the move in there.

(If you're using LVM you could then do it with pvmove without having
to worry about links or even downtime, new rsnapshots could run while
data is being moved. Actually that could be done over the network
as well with iSCSI, but I won't recommend that if unless you're
well familiar with all tecniques involved.)

--
Tapani Tarvainen

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
Moving an rsnapshot root to a new server
December 10, 2014 03:24PM
That would be ideal but it’s 4 drives on a dell sas raid controller going to 6 drives on another controller so no dice!
Chris.[quote]On 10 Dec 2014, at 12:30, Tapani Tarvainen <rsnapshot < at > tapanitarvainen.fi ([email]rsnapshot < at > tapanitarvainen.fi[/email])> wrote:
On Dec 10 12:02, David Cantrell (david < at > cantrell.org.uk ([email]david < at > cantrell.org.uk[/email])) wrote:[quote]Also, don't bank on scanning a 5TB filesystem and copying it across anetwork in under 24 hours.[/quote]Yeah. If at all possible I'd try to physically move the disks from theold machine to the new one temporarily and do the move in there.(If you're using LVM you could then do it with pvmove without havingto worry about links or even downtime, new rsnapshots could run whiledata is being moved. Actually that could be done over the networkas well with iSCSI, but I won't recommend that if unless you'rewell familiar with all tecniques involved.)-- Tapani Tarvainen------------------------------------------------------------------------------Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Serverfrom Actuate! Instantly Supercharge Your Business Reports and Dashboardswith Interactivity, Sharing, Native Excel Exports, App Integration & moreGet technology previously reserved for billion-dollar corporations, FREE[url=http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk]http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk[/url]_______________________________________________rsnapshot-discuss mailing listrsnapshot-discuss < at > lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
[/quote]
--
[url=http://chrispyfur.net]chrispyfur.net[/url]
Moving an rsnapshot root to a new server
December 10, 2014 06:34PM
On 12/10/2014 03:59 AM, Chris Tebb wrote:
[quote]I hadn’t bought of syncing the monthly backup points first, I”m
guessing I could rsync all the monthlies across first then just do the
dailies once I’ve stopped the backups on the source, good idea!
[/quote]
If you do the monthly backups first, and then "just" the daily backups,
you won't get files correctly linked, and you'll use a lot more space
than you need to.

In most cases, you can rsync the entire rsnapshot root from one server
to the other to get the bulk of the files copied, and then rsync again
later to copy only the files that have changed.

However you decide to copy the filesystem, the last time you run rsync
should include all of the directories, and you must have enough memory
for its inode list. If memory constraints are an issue, you can copy
each host's snapshot root (assuming you have a different root for each
host) individually, rather than the entire filesystem. If memory is
very constrained you could copy the individual snapshots in sequential
pairs (i.e. daily.0 and daily.1, then daily.1 and daily.2, etc). You
can't, however, copy sets of individual snapshots that don't intersect.

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
Moving an rsnapshot root to a new server
December 10, 2014 07:46PM
On 10/12/2014 11:02 PM, Chris Tebb wrote:
[quote]Yeah, I did wonder about us starting again with a new backup set and
copying the old set to a NAS so we can repurpose the old server.

That’s a lot of data to pull down though! I might even leave both
servers backing up whilst the new one takes its first copy (which I can
do with rsnapshot sync can I?)
[/quote]
Currently I back up a bunch of servers to two N4800eco devices running
mirrored full disk encryption volumes on Debian. Every disk is mirrored
using mdadm (every server).

Each N4800eco server device backs up a number of local and remote
servers at hourly intervals during the day, but 30 minutes apart -- for
example, one rsnapshot job starts at "n" minute past the hour whilst the
other server starts the same job at "n+30" minutes past the hour. This
gives me 3 copies of each file (at least), one on the actual server
(source) and one on each of the N4800eco units. It also gives me more
frequent backups. This way I have redundancy [beyond mdadm mirroring]
and if any server goes belly up for whatever reason, well I have more
bases covered than if I only had one (rsnapshot) backup server.

I expect to expand on that by taking file system snapshots (lvm2
volumes) and backing them up to external devices for secure off-site
storage. Actually, one of the N4800eco units is going to be re-located
to another site soon as well.

The one thing to be careful of with multiple rsnapshot servers is that
jobs don't overlap and fight each other for resources.

Cheers
A.

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
Moving an rsnapshot root to a new server
December 10, 2014 07:56PM
On 12/10/2014 11:44 AM, Andrew McGlashan wrote:
[quote]I expect to expand on that by taking file system snapshots (lvm2
volumes) and backing them up to external devices for secure off-site
storage.
[/quote]
If you're interested in doing that, I'd really like to get more people
involved in this project. Please have a look and see if it can be of
assistance:
https://bitbucket.org/gordonmessmer/dragonsdawn-snapshot

I also submitted a pull request that integrates rsnapshot with that project:
https://github.com/gordonmessmer/rsnapshot/compare/DrHyde:master...master

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
Moving an rsnapshot root to a new server
December 11, 2014 08:53AM
On Wed, Dec 10, 2014 at 10:57:26PM +1100, Andrew McGlashan wrote:
[quote]Option 2
2a. take a snapshot of the backup set, create a dump of the snapshot
(taking a dump of a live file system is fraught with danger, a snapshot
shouldn't be actively changing).

2b. restore the dump to the new location.
[/quote]
Start out with an operating system that supports ZFS on the new server
and use it's send/receive mechanism for snapshots.

--
Oliver PETER oliver < at > gfuzz.de 0x456D688F

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
Moving an rsnapshot root to a new server
December 11, 2014 11:04AM
[quote]On Dec 10, 2014, at 23:32, Oliver Peter <lists < at > peter.de.com> wrote:

Start out with an operating system that supports ZFS on the new server
and use it's send/receive mechanism for snapshots.
[/quote]
While this is true for the OP, those of us with large volumes (as in > hundreds of TB) are having memory issues (lack of) when doing rsnapshots copy with ZFS. Fixed it by doing a higher ashift along with adding a shitton of ram.

Just an observation, and moreso related to ZFS than rsnapshot.

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
Sorry, only registered users may post in this forum.

Click here to login