| View previous topic :: View next topic |
| Author |
Message |
Sylvain Viart - Gmail
Guest
|
 Posted: Thu Mar 11, 2010 5:55 am Post subject: backup of backuppc and schedule, is it archive? |
 |
|
Hi,
On 10/03/2010 17:39, Gerald Brandt
| Quote: | > Why not using the Precommand to
| Quote: |
# stop the backup violently
/etc/init.d/backuppc stop
# extract every hosts
for host in /etc/backuppc/hosts
extract the last backup for $host | ssh to_secondary_backup
/local/image/$host
done
|
Isn't this essentially what archive does?
|
Hum, may be.
I need to read more about this topic.
I've written the script which do what I've described above.
This is somewhat non trivial and require ssh action on each server.
But I try not to stop the backup now.
Your suggestion about archive may be very attractive, if I can setup something equivalent. 
Form the doc :
| Quote: | Starting an Archive
[...] You will see a list of previous archives and a summary on each. By clicking the ``Start Archive'' button you are presented with the list of hosts and the approximate backup size. Select the hosts you wish to archive and press the ``Archive Selected Hosts'' button.
|
I just have to know 'where' the archive will be created.
For my purpose they need to be on the secondary backup server.
If someone want to see the script and the config trick, I can post it somewhere.
Regards,
Sylvain.
| Quote: | --
Sylvain Viart.
Gmail. |
|
|
| Back to top |
|
 |
Gerald Brandt
Guest
|
| Posted: Thu Mar 11, 2010 6:03 am Post subject: backup of backuppc and schedule, is it archive? |
|
|
----- "Sylvain Viart - Gmail" <sylvain.viart < at > gmail.com> wrote:
| Quote: | Hi,
On 10/03/2010 17:39, Gerald Brandt
| Quote: | > > Why not using the Precommand to
| Quote: |
# stop the backup violently
/etc/init.d/backuppc stop
# extract every hosts
for host in /etc/backuppc/hosts
extract the last backup for $host | ssh to_secondary_backup
/local/image/$host
done
|
|
Isn't this essentially what archive does?
|
| Quote: | Hum, may be.
I need to read more about this topic.
I've written the script which do what I've described above.
This is somewhat non trivial and require ssh action on each server.
But I try not to stop the backup now.
Your suggestion about archive may be very attractive, if I can setup something equivalent.
Form the doc :
| Quote: | Starting an Archive
[...] You will see a list of previous archives and a summary on each. By clicking the ``Start Archive'' button you are presented with the list of hosts and the approximate backup size. Select the hosts you wish to archive and press the ``Archive Selected Hosts'' button.
|
I just have to know 'where' the archive will be created.
For my purpose they need to be on the secondary backup server.
If someone want to see the script and the config trick, I can post it somewhere.
Regards,
Sylvain.
|
You can specify where the archive is saved when you create an archive host, and again when you start an archive process.
| Quote: | --
Sylvain Viart.
Gmail. |
|
|
| Back to top |
|
|
Sylvain Viart - Gmail
Guest
|
| Posted: Thu Mar 11, 2010 8:40 am Post subject: backup of backuppc and schedule, is it archive? |
|
|
Gerald,
[...]
Summary of this thread:
Backing up some primaries backuppc servers, on a secondary backuppc.
One of the primary hold itself 86 host.
Strategy:
Using a BackupPC_tarCreate loop on the primary, called from a
DumpPreUserCmd on the secondary backup server.
Then the secondary backup itself the copy of every extracted host.
Gerald, noticed that it looks like archive mode, here's some pointers,
and keywords:
http://backuppc.sourceforge.net/faq/BackupPC.html#archive_functions
$Conf{XferMethod} = 'archive';
$Conf{ArchiveDest} = '/tmp';
BackupPC_archiveHost
On 11/03/2010 15:03, Gerald Brandt wrote:
| Quote: | You can specify where the archive is saved when you create an archive
host, and again when you start an archive process
|
yep, the archive mode seems, not very well documented.
this gives me:
primary backup server:
cat /etc/backuppc/archive-for-secondary.pl
$Conf{XferMethod} = 'archive';
$Conf{ArchiveDest} = '/tmp';
# an archive requested from the web
/tmp/host-01.96.tar.gz
Can I mount a ssh folder between the secondary and the primary for that
purpose?..
So I could, change:
$Conf{ArchiveDest} = '/mnt/secondary_archive';
I still need to extract them on the secondary before the backup.
Here is some part of my loop, which doesn't use archive mode, run on the
primary:
function extract_server()
{
host=$1
fullhost=$(awk "/^$host/ {print \$1}" $conf)
if [ -z "$fullhost" ]
then
echo "fullhost not found for $host, skipped"
return
fi
echo "host=$host, fullhost=$fullhost"
# as we tar extract, the folder should be removed before overwriting
# done on the secondary_backup by remote_pre_cmd
destdir=$destmachine_dir/$host
remote_pre_cmd="/etc/backuppc/remote_backup_cleanup.sh $host"
su - backuppc -c \
"time /usr/share/backuppc/bin/BackupPC_tarCreate -h $fullhost -n -1
-s / / |
ssh -l root $secondary_backup '$remote_pre_cmd && mkdir -p $destdir
&& cd $destdir && tar -x'"
}
# extract every hosts
for host in $host_list
do
# if I could do diff rsync would be better
extract_server $host
done
Regards,
Sylvain.
--
Sylvain Viart.
Gmail.
------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/ |
|
| Back to top |
|
|
Les Mikesell
Guest
|
| Posted: Thu Mar 11, 2010 9:08 am Post subject: backup of backuppc and schedule, is it archive? |
|
|
On 3/11/2010 10:38 AM, Sylvain Viart - Gmail wrote:
| Quote: | Summary of this thread:
Backing up some primaries backuppc servers, on a secondary backuppc.
One of the primary hold itself 86 host.
|
Are you trying to back up the primary backuppc servers complete with
history, or do you just need the latest full from each target in this copy?
| Quote: | Strategy:
Using a BackupPC_tarCreate loop on the primary, called from a
DumpPreUserCmd on the secondary backup server.
Then the secondary backup itself the copy of every extracted host.
|
This seems like you are adding a bottleneck compared to just backing up
the targets directly from the primary and secondary backuppc servers.
| Quote: | Gerald, noticed that it looks like archive mode, here's some pointers,
and keywords:
http://backuppc.sourceforge.net/faq/BackupPC.html#archive_functions
$Conf{XferMethod} = 'archive';
$Conf{ArchiveDest} = '/tmp';
BackupPC_archiveHost
On 11/03/2010 15:03, Gerald Brandt wrote:
| Quote: | You can specify where the archive is saved when you create an archive
host, and again when you start an archive process
|
yep, the archive mode seems, not very well documented.
|
The archive mode is really just a wrapper around BackuPC_tarCreate to
give it a web interface. If you are using your own scripts you might as
well run the commands directly there so you have complete control.
| Quote: | this gives me:
primary backup server:
cat /etc/backuppc/archive-for-secondary.pl
$Conf{XferMethod} = 'archive';
$Conf{ArchiveDest} = '/tmp';
# an archive requested from the web
/tmp/host-01.96.tar.gz
Can I mount a ssh folder between the secondary and the primary for that
purpose?..
So I could, change:
$Conf{ArchiveDest} = '/mnt/secondary_archive';
I still need to extract them on the secondary before the backup.
|
Sure, but it is probably more efficient to pipe to a remote cat or dd to
write a stream to a remote file.
| Quote: | Here is some part of my loop, which doesn't use archive mode, run on the
primary:
function extract_server()
{
host=$1
fullhost=$(awk "/^$host/ {print \$1}" $conf)
if [ -z "$fullhost" ]
then
echo "fullhost not found for $host, skipped"
return
fi
echo "host=$host, fullhost=$fullhost"
# as we tar extract, the folder should be removed before overwriting
# done on the secondary_backup by remote_pre_cmd
destdir=$destmachine_dir/$host
remote_pre_cmd="/etc/backuppc/remote_backup_cleanup.sh $host"
su - backuppc -c \
"time /usr/share/backuppc/bin/BackupPC_tarCreate -h $fullhost -n -1
-s / / |
ssh -l root $secondary_backup '$remote_pre_cmd&& mkdir -p $destdir
&& cd $destdir&& tar -x'"
}
# extract every hosts
for host in $host_list
do
# if I could do diff rsync would be better
extract_server $host
done
|
If you hit the hosts directly you could use rsync. Maybe you could use
the same ssh identity key and just script the config updates to
propagate changes as you add hosts to the primaries.
--
Les Mikesell
lesmikesell < at > gmail.com
------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/ |
|
| Back to top |
|
|
Sylvain Viart - Gmail
Guest
|
| Posted: Fri Mar 12, 2010 1:29 am Post subject: backup of backuppc and schedule, is it archive? |
|
|
Hi Les,
On 11/03/2010 18:05, Les Mikesell wrote: | Quote: | | Quote: | | Quote: | Backing up some primaries backuppc servers, on a secondary backuppc.
One of the primary hold itself 86 host.
|
Are you trying to back up the primary backuppc servers complete with
history, or do you just need the latest full from each target in this copy?
|
|
Just the last full.
Following your advice, I've changed my strategy to provide a quicker way to restore any host data, directly from the secondary.
Backuping /var/local/machine, I've all final host data living there, on the secondary.
This folder in maintained by my BackupPC_tarCreate loop.
The secondary backuppc provides the following:
- external backup, they are off site on another datacenter
- some primary host are also giving service other than backuppc, and should be secured in case of data lost
- secondary backup should be never used, just in case of serious damage
- no config sync, nor schedule sync is needed between secondary and primary backuppc
- don't need the history of each target (final host) on the secondary, only an up to date last backup
Here's how it looks like on the secondary, is it not somewhat a similar tree that is living somewhere in backuppc?
machine/
|-- server-01
| |-- etc
| `-- root
|-- server-02
| |-- etc
| `-- root
|-- server-03
| |-- etc
| `-- root
|-- other-10
| |-- etc
| |-- root
| `-- var
|-- other-11
| |-- etc
| |-- root
| `-- var
`-- moreserver-01
|-- etc
`-- root
| Quote: | | Quote: | | Quote: | Strategy:
Using a BackupPC_tarCreate loop on the primary, called from a
DumpPreUserCmd on the secondary backup server.
Then the secondary backup itself the copy of every extracted host.
|
This seems like you are adding a bottleneck compared to just backing up
the targets directly from the primary and secondary backuppc servers.
|
|
Backup of final host twice should be avoided, because:
- need to keep the config in sync between, primary and secondary backup server.
- the backup job is too heavy for the final host. I prefer to use the copy which lives on the primary backup.
What is the bottleneck?
| Quote: | | Quote: | If you hit the hosts directly you could use rsync. Maybe you could use
the same ssh identity key and just script the config updates to
propagate changes as you add hosts to the primaries.
|
|
Hum, yes...
But what about my earlier problem about blackout period sync between primary and secondary?
I don't want a final host to be backuped by both backuppc at the the same time, IO would be really poor!
I'm still interesting in a scheduler simulator!
May be some developer could point me, in the backuppc code?
I can tolerate poor IO on the primary backuppc, which is dedicated to backup.
Can I use a DumpPreUserCmd , or some other check, to schedule a deferred backup on a given host?
For now my script should work, except a small permission problem during tar extract on the secondary.
I'm going to perform everything as root, should kept user's id and original permission, as backuped from the final host.
Regards,
Sylvain.
| Quote: | --
Sylvain Viart.
Gmail. |
|
|
| Back to top |
|
|
Les Mikesell
Guest
|
| Posted: Fri Mar 12, 2010 6:18 am Post subject: backup of backuppc and schedule, is it archive? |
|
|
Sylvain Viart - Gmail wrote:
| Quote: | | Quote: | | Quote: | Backing up some primaries backuppc servers, on a secondary backuppc.
One of the primary hold itself 86 host.
|
Are you trying to back up the primary backuppc servers complete with
history, or do you just need the latest full from each target in this copy?
|
Just the last full.
Following your advice, I've changed my strategy to provide a quicker way
to restore any host data, directly from the secondary.
Backuping /var/local/machine, I've all final host data living there, on
the secondary.
|
Maybe it would be easier to reverse the concept and do a straight rsync to an
intermediate disk location at the primary site, keeping only one copy there,
letting the remote backuppc copy that and keep all the history. That has the
downside of having to write your own rsync scripts for the local copy or finding
something else, but would be a more efficient approach since you don't have to
copy in and out of backuppc's storage format all the time.
| Quote: | | Quote: | | Quote: | Strategy:
Using a BackupPC_tarCreate loop on the primary, called from a
DumpPreUserCmd on the secondary backup server.
Then the secondary backup itself the copy of every extracted host.
|
This seems like you are adding a bottleneck compared to just backing up
the targets directly from the primary and secondary backuppc servers.
|
Backup of final host twice should be avoided, because:
* need to keep the config in sync between, primary and secondary
backup server.
* the backup job is too heavy for the final host. I prefer to use
the copy which lives on the primary backup.
What is the bottleneck?
|
You are making the primary backuppc server copy everything in and out of
backuppc format - and I was thinking you were feeding the tar extract to the
secondary needing bandwidth for the complete copy. If you extract on the
primary, at least you can use rsync from the secondary.
| Quote: | | Quote: | If you hit the hosts directly you could use rsync. Maybe you could use
the same ssh identity key and just script the config updates to
propagate changes as you add hosts to the primaries.
|
Hum, yes...
But what about my earlier problem about blackout period sync between
primary and secondary?
I don't want a final host to be backuped by both backuppc at the the
same time, IO would be really poor!
|
That could be as simple as having non-overlapping times that aren't in the
blackout. That's assuming that you have time away from the server's peak load to
complete 2 runs. I'm used to 'business' type use patterns where you have all
night. If your servers have heavy international use that might not be the case.
| Quote: | I'm still interesting in a scheduler simulator!
May be some developer could point me, in the backuppc code?
|
Typically the schedule is mostly driven by how long it took the previous hosts
to complete because of the concurrency limit.
| Quote: | I can tolerate poor IO on the primary backuppc, which is dedicated to
backup.
Can I use a DumpPreUserCmd , or some other check, to schedule a deferred
backup on a given host?
|
The ping command would be a better place to decide to defer a run.
| Quote: | For now my script should work, except a small permission problem during
tar extract on the secondary.
I'm going to perform everything as root, should kept user's id and
original permission, as backuped from the final host.
|
If the primary has time and disk space to complete the backups and extracts, it
looks like it might work.
--
Les Mikesell
lesmikesell < at > gmail.com
------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/ |
|
| Back to top |
|
|
Sylvain Viart - Gmail
Guest
|
| Posted: Fri Mar 12, 2010 8:55 am Post subject: backup of backuppc and schedule, is it archive? |
|
|
Hi,
| Quote: | Maybe it would be easier to reverse the concept and do a straight rsync to an
intermediate disk location at the primary site, keeping only one copy there,
letting the remote backuppc copy that and keep all the history. That has the
downside of having to write your own rsync scripts for the local copy or finding
something else, but would be a more efficient approach since you don't have to
copy in and out of backuppc's storage format all the time.
|
Yes !
But the effort of writing the new rsync 'in between' seems not so easy.
Because of each host config in backuppc, need to be respected.
May be a wrapper of the original rsync command used by backuppc.
But your suggestion seems good.
Because, I've the same sort of result on the secondary, a full tree with
every host.
I had to rethink about the whole picture.
The storage is best on my secondary, but it is off site for all final
host target.
what about the purge, on the middle storage, rsync --delete ?
| Quote: | | Quote: | What is the bottleneck?
|
You are making the primary backuppc server copy everything in and out of
backuppc format - and I was thinking you were feeding the tar extract to the
secondary needing bandwidth for the complete copy. If you extract on the
primary, at least you can use rsync from the secondary.
|
right.
I will extract a full 200Gb each time.
Which need to be optimized...
Study of your other suggestions, about how to approach the "scheduler
simulator" and the "blackout generator", next time.
Thanks for your time.
Regards,
Sylvain.
--
Sylvain Viart.
Gmail.
------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/ |
|
| Back to top |
|
|
Les Mikesell
Guest
|
| Posted: Fri Mar 12, 2010 9:19 am Post subject: backup of backuppc and schedule, is it archive? |
|
|
On 3/12/2010 10:52 AM, Sylvain Viart - Gmail wrote:
| Quote: | Hi,
| Quote: | Maybe it would be easier to reverse the concept and do a straight rsync to an
intermediate disk location at the primary site, keeping only one copy there,
letting the remote backuppc copy that and keep all the history. That has the
downside of having to write your own rsync scripts for the local copy or finding
something else, but would be a more efficient approach since you don't have to
copy in and out of backuppc's storage format all the time.
|
Yes !
But the effort of writing the new rsync 'in between' seems not so easy.
Because of each host config in backuppc, need to be respected.
May be a wrapper of the original rsync command used by backuppc.
|
That would depend on how complicated the configs are. It might be as
simple as a shell loop that reads a file with lines containing hostnames
and paths to back up, cd's to a directory named for each host and
running the appropriate rsync commends there. Good error handling could
be a problem, though.
| Quote: | But your suggestion seems good.
Because, I've the same sort of result on the secondary, a full tree with
every host.
I had to rethink about the whole picture.
The storage is best on my secondary, but it is off site for all final
host target.
what about the purge, on the middle storage, rsync --delete ?
|
Yes, that would work - although you do set up a scenario where if the
target dies in mid-backup you have an inconsistent snapshot since you
only have one local copy. Or if a mount point is unmounted, you'd wipe
the previous copy - but you could recover from those hopefully unlikely
events from the secondary backuppc copy which could hold some history.
--
Les Mikesell
lesmikesell < at > gmail.com
------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/ |
|
| Back to top |
|
|
|
FAQ
Search
Register
Log in to check your private messages
Log in
