Welcome! » Log In » Create A New Profile

LVM -and- ssh backups?

Posted by Anonymous 
LVM -and- ssh backups?
February 22, 2015 06:15PM
All,

Perhaps my $searchengine-fu is failing me, but I'm unable to see a way
to accomplish the following. I'd like to snapshot an LVM on system A
and then store the backup on system B via ssh.

Unfortunately, the config nomenclature rsnapshot seems to make this
impossible. eg: On my existing Xen box that has the backup drive:

backup lvm://data0/jason_home/ jason_home/

where data0 is a VG on system A backed by a large raid5. jason_home/ is
on an external drive also attached to system A.

So, now on system B, I have a VG, no Xen, and a need to backup one of
the LVs to the backup drive on system A. I'd like to do:

backup lvm://titan0/jason_home/ jason < at > systemA-rsnapshot:titan_home/

But configtest barfs on that...

Am I missing something obvious here? Why, when I want to use ssh, does
everything say I must configure the first argument to 'backup'? I'd
think I'd want that info in the destination...

Thanks for any insight,

Jason.

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=190641631&iu=/4140/ostg.clktrk
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
LVM -and- ssh backups?
February 22, 2015 07:08PM
On 02/22/2015 05:02 PM, Jason Cooper wrote:
[quote]So, now on system B, I have a VG, no Xen, and a need to backup one of
the LVs to the backup drive on system A. I'd like to do:

backup lvm://titan0/jason_home/ jason < at > systemA-rsnapshot:titan_home/
[/quote]
rsnapshot doesn't support remote backup destinations. It doesn't have
code to handle directory rotation or other operations on the snapshots
on a remote system.

Instead, you need to run rsnapshot on the host where the backups are
kept, and use a remote description for the backup source. The problem
you'll run into there is that rsnapshot doesn't support remote LVM
snapshots. In order to do that, you would need to apply this patch:
https://github.com/rsnapshot/rsnapshot/pull/44

...and use "snapshot" on the server where you want the snapshots made.
https://bitbucket.org/gordonmessmer/dragonsdawn-snapshot

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=190641631&iu=/4140/ostg.clktrk
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
LVM -and- ssh backups?
February 22, 2015 08:39PM
On Sun, Feb 22, 2015 at 10:06 PM, Gordon Messmer
<gordon.messmer < at > gmail.com> wrote:
[quote]On 02/22/2015 05:02 PM, Jason Cooper wrote:
[quote]So, now on system B, I have a VG, no Xen, and a need to backup one of
the LVs to the backup drive on system A. I'd like to do:

backup lvm://titan0/jason_home/ jason < at > systemA-rsnapshot:titan_home/
[/quote]
rsnapshot doesn't support remote backup destinations. It doesn't have
code to handle directory rotation or other operations on the snapshots
on a remote system.
[/quote]
Another possibility is to mount the backup directory as an NFS
writable volume, to only the client that needs to write from the LVM
snapshot. It can be noticeably slower than some local disk operations,
but I've found it an effective way to avoid having to manage complex
SSH tunnels.

[quote]Instead, you need to run rsnapshot on the host where the backups are
kept, and use a remote description for the backup source. The problem
you'll run into there is that rsnapshot doesn't support remote LVM
snapshots. In order to do that, you would need to apply this patch:
https://github.com/rsnapshot/rsnapshot/pull/44

...and use "snapshot" on the server where you want the snapshots made.
https://bitbucket.org/gordonmessmer/dragonsdawn-snapshot

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=190641631&iu=/4140/ostg.clktrk
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
[/quote]
------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=190641631&iu=/4140/ostg.clktrk
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
LVM -and- ssh backups?
February 23, 2015 06:05AM
On Sun, Feb 22, 2015 at 07:06:52PM -0800, Gordon Messmer wrote:
[quote]On 02/22/2015 05:02 PM, Jason Cooper wrote:
[quote]So, now on system B, I have a VG, no Xen, and a need to backup one of
the LVs to the backup drive on system A. I'd like to do:

backup lvm://titan0/jason_home/ jason < at > systemA-rsnapshot:titan_home/
[/quote]
rsnapshot doesn't support remote backup destinations. It doesn't have
code to handle directory rotation or other operations on the snapshots
on a remote system.
[/quote]
Ok, well at least I know I wasn't missing something obvious. ;-)

[quote]Instead, you need to run rsnapshot on the host where the backups are
kept, and use a remote description for the backup source. The problem
you'll run into there is that rsnapshot doesn't support remote LVM
snapshots. In order to do that, you would need to apply this patch:
https://github.com/rsnapshot/rsnapshot/pull/44

...and use "snapshot" on the server where you want the snapshots made.
https://bitbucket.org/gordonmessmer/dragonsdawn-snapshot
[/quote]
hmmm, this seems to solve your specific problem, but isn't very generic.
Perhaps if rsnapshot were installed on both systems, an 'rsnapshot
server' command option would allow the user to configure remote and
local as needed...

eg, on systemB:

snapshot_root jason < at > systemA-rsnapshot:/mnt/backup/.snapshot
...
backup lvm://titan0/home/ jason_home/

-or-

snapshot_root none
...
backup lvm://titan0/home/ jason < at > systemA-rsnapshot:/mnt/backup/.snapshot/jason_home/
backup lvm://titan0/work/ jason < at > systemC-rsnapshot:/mnt/backup/.snapshot/jason_work/

On systemB, 'rsnapshot hourly' would ssh to systemA, calling 'rsnapshot
server' and forward over all the commands to rotate directories and sync
files.

thx,

Jason.

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=190641631&iu=/4140/ostg.clktrk
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
LVM -and- ssh backups?
February 23, 2015 06:20AM
On Sun, Feb 22, 2015 at 11:37:22PM -0500, Nico Kadel-Garcia wrote:
[quote]On Sun, Feb 22, 2015 at 10:06 PM, Gordon Messmer
<gordon.messmer < at > gmail.com> wrote:
[quote]On 02/22/2015 05:02 PM, Jason Cooper wrote:
[quote]So, now on system B, I have a VG, no Xen, and a need to backup one of
the LVs to the backup drive on system A. I'd like to do:

backup lvm://titan0/jason_home/ jason < at > systemA-rsnapshot:titan_home/
[/quote]
rsnapshot doesn't support remote backup destinations. It doesn't have
code to handle directory rotation or other operations on the snapshots
on a remote system.
[/quote]
Another possibility is to mount the backup directory as an NFS
writable volume, to only the client that needs to write from the LVM
snapshot. It can be noticeably slower than some local disk operations,
but I've found it an effective way to avoid having to manage complex
SSH tunnels.
[/quote]
The VGs on both systems, and the backup drive, are encrypted. I'd
prefer to use an encrypted protocol when transferring the data.

thx,

Jason.

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=190641631&iu=/4140/ostg.clktrk
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
LVM -and- ssh backups?
February 23, 2015 09:39AM
On 02/23/2015 05:43 AM, Jason Cooper wrote:
[quote]
hmmm, this seems to solve your specific problem, but isn't very generic.
[/quote]
I'm open to suggestions on improvement.

[quote]Perhaps if rsnapshot were installed on both systems, an 'rsnapshot
server' command option would allow the user to configure remote and
local as needed...
[/quote]
They say that you shouldn't get too attached to your solution to a
problem, because you might later find that your solution is the next
problem that has to be solved.

I think you might want to take a step back and look at the complexity of
the system that you're proposing.

In your system, all hosts have their own installation of rsnapshot. The
source connects to an "rsnapshot server" to perform rotation, etc. That
requires that the two systems have a protocol for exchanging their
configuration, sending commands, and error handling. All of that would
be in addition to the existing code which validates configurations,
verifies that specified external commands are present, etc.

Compare that to the existing setup which requires only one installation
of rsnapshot, on the destination host. No server protocol is required.
Generally, the only thing that's required on the source host (the one
you're backing up) is either an rsync daemon or ssh and rsync.

The existing code is kind of complicated already, but it's vastly
simpler than what you've proposed. And try as I might, I can't think of
a single advantage of your proposed configuration. The destination
server would still have to have rsnapshot, and you'd still need ssh keys
to launch the different pieces. What problem does it solve?

One of the very few things that rsnapshot doesn't already do is volume
snapshots, which is what I implemented. It extends rsnapshot's LVM
support (for ext4 and XFS) to support additional filesystems, to support
snapshots for remote hosts, and to support instructing daemons to flush
data to disk in preparation for backups.

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=190641631&iu=/4140/ostg.clktrk
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
LVM -and- ssh backups?
February 23, 2015 01:59PM
Hey Gordon,

On Mon, Feb 23, 2015 at 09:36:53AM -0800, Gordon Messmer wrote:
[quote]On 02/23/2015 05:43 AM, Jason Cooper wrote:
[quote]
Perhaps if rsnapshot were installed on both systems, an 'rsnapshot
server' command option would allow the user to configure remote and
local as needed...
[/quote]
In your system, all hosts have their own installation of rsnapshot. The
source connects to an "rsnapshot server" to perform rotation, etc. That
requires that the two systems have a protocol for exchanging their
configuration, sending commands, and error handling. All of that would
be in addition to the existing code which validates configurations,
verifies that specified external commands are present, etc.
[/quote]
What I see is a tool that works extremely well for same-system backups.
What I need, and I hope I'm not the only one, is a tool that can
securely push backups of LVs to a remote disk.

I'm interested in extending rsnapshot to support that. Yes, I understand
the complexities involved.

[quote]Compare that to the existing setup which requires only one installation
of rsnapshot, on the destination host. No server protocol is required.
Generally, the only thing that's required on the source host (the one
you're backing up) is either an rsync daemon or ssh and rsync.
[/quote]
There is a majority of the 'protocol' right there. rsync. I'm simply
proposing to extend this to support the directory rotation. It may even
be two ssh connections in the first revision. The first for directory
rotation, the second for rsync.

[quote]The existing code is kind of complicated already, but it's vastly
simpler than what you've proposed.
[/quote]
Of course, I'm proposing extending the capabilities. It will require
careful thought and review. Are you suggesting I should include cleanup
patches in my series? ;-)

[quote]And try as I might, I can't think of
a single advantage of your proposed configuration. The destination
server would still have to have rsnapshot, and you'd still need ssh keys
to launch the different pieces. What problem does it solve?
[/quote]
I have a laptop or two that need to do backups to a server on the
network. The laptops come and go, they sleep. Only *they* know "It's
been greater than an hour, I just woke up, and the backup server is
visible. Do a backup."

I'm trying hard not to mention specific implementations, but just about
every backup solution on the market is "Device pushes to Server/Cloud".
So, I'm hopeful I'm not the only one who thinks this is a good idea. :-)

thx,

Jason.

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=190641631&iu=/4140/ostg.clktrk
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
LVM -and- ssh backups?
February 23, 2015 04:59PM
G.Messmer makes good points about simplicity, e.g., single rsnapshot server and not getting too attached to an incomplete solution. Very astute!

My suggestion:

I&#39;m running rsync in daemon mode on my key backup sources already, so copying from one to another via rsnapshot is fairly easy. I&#39;d be looking at a solution along the lines of:

- An ssh script utilizing the ssh forced-command feature is run from the rsnapshot server. It causes serverA to create the LVM snapshot and mount it.

- If the script succeeds, run the rsnapshot backup, something like

backup  lvm::/backup/data0/jason_home systemC::/backup/jason_home

- Finally, if the backup succeeds, unmount & remove the snapshot via a second ssh script.

It would be nice if rsnapshot let you do more conditional execution with the pre-exec script features, but I don&#39;t think it&#39;s there yet.

HTH, PM

On Mon, Feb 23, 2015 at 1:57 PM, Jason Cooper <rsnapshot < at > lakedaemon.net ([email]rsnapshot < at > lakedaemon.net[/email])> wrote:
[quote]Hey Gordon,

On Mon, Feb 23, 2015 at 09:36:53AM -0800, Gordon Messmer wrote:
[quote]On 02/23/2015 05:43 AM, Jason Cooper wrote:
[quote]
Perhaps if rsnapshot were installed on both systems, an &#39;rsnapshot
server&#39; command option would allow the user to configure remote and
local as needed...
[/quote]
In your system, all hosts have their own installation of rsnapshot.  The
source connects to an "rsnapshot server" to perform rotation, etc.  That
requires that the two systems have a protocol for exchanging their
configuration, sending commands, and error handling.  All of that would
be in addition to the existing code which validates configurations,
verifies that specified external commands are present, etc.
[/quote]
What I see is a tool that works extremely well for same-system backups.
What I need, and I hope I&#39;m not the only one, is a tool that can
securely push backups of LVs to a remote disk.

I&#39;m interested in extending rsnapshot to support that.  Yes, I understand
the complexities involved.

[quote]Compare that to the existing setup which requires only one installation
of rsnapshot, on the destination host.  No server protocol is required.
   Generally, the only thing that&#39;s required on the source host (the one
you&#39;re backing up) is either an rsync daemon or ssh and rsync.
[/quote]
There is a majority of the &#39;protocol&#39; right there.  rsync.  I&#39;m simply
proposing to extend this to support the directory rotation.  It may even
be two ssh connections in the first revision.  The first for directory
rotation, the second for rsync.

[quote]The existing code is kind of complicated already, but it&#39;s vastly
simpler than what you&#39;ve proposed.
[/quote]
Of course, I&#39;m proposing extending the capabilities.  It will require
careful thought and review.  Are you suggesting I should include cleanup
patches in my series? ;-)

[quote]And try as I might, I can&#39;t think of
a single advantage of your proposed configuration.  The destination
server would still have to have rsnapshot, and you&#39;d still need ssh keys
to launch the different pieces.  What problem does it solve?
[/quote]
I have a laptop or two that need to do backups to a server on the
network.  The laptops come and go, they sleep.  Only *they* know "It&#39;s
been greater than an hour, I just woke up, and the backup server is
visible.  Do a backup."

I&#39;m trying hard not to mention specific implementations, but just about
every backup solution on the market is "Device pushes to Server/Cloud".
So, I&#39;m hopeful I&#39;m not the only one who thinks this is a good idea. :-)

thx,

Jason.

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
[url=http://pubads.g.doubleclick.net/gampad/clk?id=190641631&iu=/4140/ostg.clktrk]http://pubads.g.doubleclick.net/gampad/clk?id=190641631&iu=/4140/ostg.clktrk[/url]
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net ([email]rsnapshot-discuss < at > lists.sourceforge.net[/email])
[url=https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss]https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss[/url]

[/quote]

--
Paul Mackinney
Systems & Quality Manager
O.N. Diagnostics, LLC
2150 Shattuck Ave. Suite 610, Berkeley, CA 94704
510-204-0688 (phone) | 510-356-4349 (fax)
_____________________________________________
If you receive this message in error, please delete it immediately. This message may contain information that is privileged, confidential and exempt from disclosure and dissemination under applicable law.
LVM -and- ssh backups?
February 25, 2015 11:59AM
On Mon, 23 Feb 2015 16:57:30 -0500
Jason Cooper <rsnapshot < at > lakedaemon.net> wrote:

[quote]Hey Gordon,

On Mon, Feb 23, 2015 at 09:36:53AM -0800, Gordon Messmer wrote:
[quote]On 02/23/2015 05:43 AM, Jason Cooper wrote:
[quote]
Perhaps if rsnapshot were installed on both systems, an 'rsnapshot
server' command option would allow the user to configure remote and
local as needed...
[/quote]
In your system, all hosts have their own installation of rsnapshot.
The source connects to an "rsnapshot server" to perform rotation,
etc. That requires that the two systems have a protocol for
exchanging their configuration, sending commands, and error
handling. All of that would be in addition to the existing code
which validates configurations, verifies that specified external
commands are present, etc.
[/quote]
What I see is a tool that works extremely well for same-system backups.
What I need, and I hope I'm not the only one, is a tool that can
securely push backups of LVs to a remote disk.

I'm interested in extending rsnapshot to support that. Yes, I
understand the complexities involved.

[quote]Compare that to the existing setup which requires only one
installation of rsnapshot, on the destination host. No server
protocol is required. Generally, the only thing that's required on
the source host (the one you're backing up) is either an rsync
daemon or ssh and rsync.
[/quote]
There is a majority of the 'protocol' right there. rsync. I'm simply
proposing to extend this to support the directory rotation. It may
even be two ssh connections in the first revision. The first for
directory rotation, the second for rsync.

[quote]The existing code is kind of complicated already, but it's vastly
simpler than what you've proposed.
[/quote]
Of course, I'm proposing extending the capabilities. It will require
careful thought and review. Are you suggesting I should include
cleanup patches in my series? ;-)

[quote]And try as I might, I can't think of
a single advantage of your proposed configuration. The destination
server would still have to have rsnapshot, and you'd still need ssh
keys to launch the different pieces. What problem does it solve?
[/quote]
I have a laptop or two that need to do backups to a server on the
network. The laptops come and go, they sleep. Only *they* know "It's
been greater than an hour, I just woke up, and the backup server is
visible. Do a backup."

I'm trying hard not to mention specific implementations, but just about
every backup solution on the market is "Device pushes to Server/Cloud".
So, I'm hopeful I'm not the only one who thinks this is a good
idea. :-)

thx,

Jason.

[/quote]
I've used rsnapshot on my home box for many years reliably, using it in
it's intended way, for it's intended purpose, and knock on wood, it's
never failed me yet.

Until this thread I really hadn't bothered looking at the code, as it
always 'just worked'.

Now that I have looked at the code however, I don't believe extending
this code to do what you want would be a very good idea. It's already
quite large for what it's actually doing. Not knocking the Authors, but
mission creep has a way of, well, creeping in. As a result I think
it's gotten a bit over-engineered over the years.

If you really want full local/remote functionality, rolling a new
script designed specifically to do that from the beginning would not be
that difficult, and IMHO would be the correct approach here.

I have a non-trivial amount of experience rolling custom backup
solutions, being a *NIX sysadmin for over 25 years, and I've given this
problem more than a bit of thought (and code), and if I were to have the
time to attack it, the actual script itself would be quite tiny.

The script (but not jobs) would be able to run concurrently, and jobs
would simply be defined in small job-specific config files in a .d-style
directory structure, which loaded after a common config, and could
override common defaults as needed. Everything about a specific job was
completely contained in it's own simple config, extending common only
where required.

The main thing is in intelligently wrapping the functionality in ssh
(or not) depending on the source/target combination. The script need
only reside on the box you from which it's launched, but it can
backup any host's data to any other host, as long as appropriate keys
are installed. The script can even 'tear off' from the launching host
and stay running on the other hosts, without maintaining the ssh
connection to the launcher if desired.

--
Regards,
Christopher Barry

Random geeky fortune:
Nothing makes a person more productive than the last minute.

------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
LVM -and- ssh backups?
February 25, 2015 12:13PM
On 02/23/2015 01:57 PM, Jason Cooper wrote:
[quote]
What I see is a tool that works extremely well for same-system backups.
What I need, and I hope I'm not the only one, is a tool that can
securely push backups of LVs to a remote disk.
[/quote]
I think that's a good assessment of rsnapshot's current release. It has
some, minimal support for making LVM snapshots on a local system and
backing them up.

However, if you apply the patch that I wrote, and use "snapshot" on the
backup clients, you have a system that can securely pull backups of LVs
and other snapshot types.

[quote]There is a majority of the 'protocol' right there. rsync. I'm simply
proposing to extend this to support the directory rotation. It may even
be two ssh connections in the first revision. The first for directory
rotation, the second for rsync.
[/quote]
If you compare the size of rsync and rsnapshot, I certainly wouldn't
argue that rsync's protocol wouldn't be far more complex than what
you've proposed for rsnapshot.

But have you ever looked at the code for rsync? It's QUITE complex.
IIRC, there are over 20 protocol versions therein. rsnapshot would
probably never get *that* complex, but network protocols are not
trivial. Especially if you want to be able to add features to your
software without breaking backward compatibility.

[quote]Of course, I'm proposing extending the capabilities. It will require
careful thought and review. Are you suggesting I should include cleanup
patches in my series? ;-)
[/quote]
I'm suggesting that you evaluate the code that's already written. It's
less than 200 lines added to rsnapshot. Re-writing rsnapshot to operate
in a client-server fashion is going to be a massively more complex
undertaking, and once you do so you still have support only for LVM
backups, with almost no error handling.

[quote]I have a laptop or two that need to do backups to a server on the
network. The laptops come and go, they sleep. Only *they* know "It's
been greater than an hour, I just woke up, and the backup server is
visible. Do a backup."

I'm trying hard not to mention specific implementations, but just about
every backup solution on the market is "Device pushes to Server/Cloud".
So, I'm hopeful I'm not the only one who thinks this is a good idea. :-)
[/quote]
That's a use case that rsnapshot really isn't very good for. I wrote
some scripts to trigger rsnapshot backups remotely (from the client), a
while back. If you're interested I can share those as well.

------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
LVM -and- ssh backups?
February 25, 2015 12:13PM
On 02/23/2015 04:30 PM, Paul Mackinney wrote:
[quote]I'd be looking
at a solution along the lines of:
[/quote]...
[quote]- Finally, if the backup succeeds, unmount & remove the snapshot via a
second ssh script.
[/quote]
And if something goes wrong with the backup? Do you just leave the
stale snapshot mounted? :)

[quote]It would be nice if rsnapshot let you do more conditional execution with
the pre-exec script features, but I don't think it's there yet.
[/quote]
pre-exec is executed before each "backup" target individually. Suppose
you have an application which keeps SQL data on one filesystem, but
refers to files on a second. It's important to snapshot the entire
system at once in some cases, and pre-exec is unsuitable for that.

Which brings me back to the code that's already written.

When you're using "snapshot," the flow of execution looks something like:
1: rsnapshot begins by invoking "snapshot" on all hosts with defined
backups, including the local host.
1.a: "snapshot" runs on each host
1.b: services are quiesced.
1.c: filesystems are snapshotted
1.d: services are resumed.
1.e: "snapshot" indicates completion to rsnapshot
2: rsnapshot continues execution as normal, and keeps an open file
handle connected to "snapshot"
3: rsync runs to back up data
4: rsnapshot closes the file handles, which tells "snapshot" to unmount
and remove the snapshots.

This implementation has several aspects that I think are important.
First, it is a minimal modification to rsnapshot. Code to handle
various filesystems is maintained externally, to reduce complexity in
rsnapshot. Second, there is a minimum of additional code on each of the
backup clients. rsnapshot isn't needed on each one individually, only
the local snapshot client. Third, snapshot cleanup is handled well in
the event of failure of a network link or elsewhere in rsnapshot.

This implementation is reasonably clean. It's generic (especially in
that it's usable by other backup systems). And it's done.

------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
LVM -and- ssh backups?
February 25, 2015 12:41PM
[quote][quote]- Finally, if the backup succeeds, unmount & remove the snapshot via a
second ssh script.
[/quote][/quote]
[quote]And if something goes wrong with the backup?  Do you just leave the
stale snapshot mounted?  :)
[/quote]

Did I forget the footnote? "Error handling has been left as an exercise for the reader."

Cheers, PM ;-)

On Tue, Feb 24, 2015 at 11:05 AM, Gordon Messmer <gordon.messmer < at > gmail.com ([email]gordon.messmer < at > gmail.com[/email])> wrote:
[quote]On 02/23/2015 04:30 PM, Paul Mackinney wrote:
[quote]I&#39;d be looking
at a solution along the lines of:
[/quote]...
[quote]- Finally, if the backup succeeds, unmount & remove the snapshot via a
second ssh script.
[/quote]
And if something goes wrong with the backup?  Do you just leave the
stale snapshot mounted?  :)

[quote]It would be nice if rsnapshot let you do more conditional execution with
the pre-exec script features, but I don&#39;t think it&#39;s there yet.
[/quote]
pre-exec is executed before each "backup" target individually.  Suppose
you have an application which keeps SQL data on one filesystem, but
refers to files on a second.  It&#39;s important to snapshot the entire
system at once in some cases, and pre-exec is unsuitable for that.

Which brings me back to the code that&#39;s already written.

When you&#39;re using "snapshot," the flow of execution looks something like:
1: rsnapshot begins by invoking "snapshot" on all hosts with defined
backups, including the local host.
   1.a: "snapshot" runs on each host
   1.b: services are quiesced.
   1.c: filesystems are snapshotted
   1.d: services are resumed.
   1.e: "snapshot" indicates completion to rsnapshot
2: rsnapshot continues execution as normal, and keeps an open file
handle connected to "snapshot"
3: rsync runs to back up data
4: rsnapshot closes the file handles, which tells "snapshot" to unmount
and remove the snapshots.

This implementation has several aspects that I think are important.
First, it is a minimal modification to rsnapshot.  Code to handle
various filesystems is maintained externally, to reduce complexity in
rsnapshot.  Second, there is a minimum of additional code on each of the
backup clients.  rsnapshot isn&#39;t needed on each one individually, only
the local snapshot client.  Third, snapshot cleanup is handled well in
the event of failure of a network link or elsewhere in rsnapshot.

This implementation is reasonably clean.  It&#39;s generic (especially in
that it&#39;s usable by other backup systems).  And it&#39;s done.

------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. [url=http://goparallel.sourceforge.net/]http://goparallel.sourceforge.net/[/url]
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net ([email]rsnapshot-discuss < at > lists.sourceforge.net[/email])
[url=https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss]https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss[/url]

[/quote]

--
Paul Mackinney
Systems & Quality Manager
O.N. Diagnostics, LLC
2150 Shattuck Ave. Suite 610, Berkeley, CA 94704
510-204-0688 (phone) | 510-356-4349 (fax)
_____________________________________________
If you receive this message in error, please delete it immediately. This message may contain information that is privileged, confidential and exempt from disclosure and dissemination under applicable law.
LVM -and- ssh backups?
February 28, 2015 07:31AM
Hey Christopher,

On 2/24/15 1:35 AM, Christopher Barry wrote:
...
[quote]If you really want full local/remote functionality, rolling a new
script designed specifically to do that from the beginning would not be
that difficult, and IMHO would be the correct approach here.
[/quote]
While I fear you may be correct, I'd like to avoid that if possible.
For two reasons: 1) I would be the only person using the script, and 2)
I would lose the advantage of everyone's experience with rsnapshot.

For me the primary comfort of opensource is that I'm using and
exercising the same code everyone else is, the same way. I only deviate
from that by conscious decision and necessity. This way, I only have to
worry about being the *first* to discover a bug.

Also, I'm new to the "formally backup everything" arena. There are a
lot of problems I haven't encountered yet. Doing my own script means
that the rsnapshot community couldn't help with any problems I
encounter.

[quote]I have a non-trivial amount of experience rolling custom backup
solutions, being a *NIX sysadmin for over 25 years, and I've given this
problem more than a bit of thought (and code), and if I were to have the
time to attack it, the actual script itself would be quite tiny.
[/quote]
Yes, here is where it gets interesting. I've done some digging into
rsync. In order to extend rsnapshot, we need to know what the ideal
solution for my usecase looks like. Then, we can determine if modifying
rsnapshot is feasible.

Basically, rsync has 95% of what I need (recall my lack of experience,
mentioned above, though ;-) ). Using --link-dest and ssh with
single-purpose keys gets most of the heavy lifting.

The client side script would need to take the snapshot of the LV, but
that's a solved problem.

The biggest hurdle is relaying state to the server side. The most
important thing we need to relay is whether or not a given backup
directory is a complete backup.

I've looked at several temp directory type options in rsync, and not one
of them does what I think I need. The cleanest solution would be if
rsync had a --temp-dest-dir option which would mangle the destination
directory.

'jason < at > serverB:/path/to/backup/serverA/YYYYMMDD-HHMMSS'

would change to

'jason < at > serverB:/path/to/backup/serverA/.YYYYMMDD-HHMMSS.tmp'.

Once rsync knows the backup is complete, it would then rename the
tempdir to the requested dirname.

This would allow any software running on serverB to know a) which
backups are complete, and b) which are failed/in-progress. The only
task left to do on serverB is the culling of the directories. For me,
that can be solved with an asynchronous cron job.

The client script would also need to know the dirname of the most recent
successful backup. To avoid another ssh command (I *really* lock down
single-purpose keys [0]), we can record that in a state file on the
client. eg when rsync returns successfully, we record the
'YYYYMMDD-HHMMSS' name we used. This would be used the next time as the
relative path argument to --link-dest.

The only conflict I see remaining on the server side is the cron job
executing concurrently with a backup in progress. I suspect the answer
is to cull all tempdirs older than 24 hours. Leaving younger ones
intact in case they are in-progress. That's not ideal, but it shouldn't
fail. The motivated script-writer could have the cronjob check lsof for
rsync in progress.

I'm still thinking about how to handle full backup disks. I wonder if
rsync's --dry-run will return failure if there isn't enough space at the
destination? The man page doesn't say.

thx,

Jason.

[0] http://git.infradead.org/users/jcooper/secsh.git/blob/HEAD:/README

------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
Sorry, only registered users may post in this forum.

Click here to login