Backup Central

Hi,

I've a backup server which goal is to backup other backuppc servers.

So I've few hosts to backup, but they are pretty heavy, but not that
much compared to other Thread.

As described many times, backup use a best effort policy, trying to
start backup when it is allowed to do so, and when it detects than a
host is due to be backuped.

So I've found:

Quote:

It's possible to schedule a backup. Just disable backups of that host, and
schedule the backup using cron.

http://backuppc.wiki.sourceforge.net/serverMesg_commands

sounds an alternative, but, could be nice to achieve it the backuppc's way.

My question in fact is about interpreting current config?
Is there tools which displays when a host will be backuped given the
current full configuration?

For my purpose, I would like to backup hosts one per day, and a
particular day for each host.

Regards
Sylvain.

--
Sylvain Viart.
Gmail.


------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Hi,

----- "Sylvain Viart - Gmail" <sylvain.viart < at > gmail.com> wrote:

Quote:

Hi,

I've a backup server which goal is to backup other backuppc servers.

So I've few hosts to backup, but they are pretty heavy, but not that
much compared to other Thread.

As described many times, backup use a best effort policy, trying to
start backup when it is allowed to do so, and when it detects than a
host is due to be backuped.

So I've found:

Quote:

It's possible to schedule a backup. Just disable backups of that host, and
schedule the backup using cron.

http://backuppc.wiki.sourceforge.net/serverMesg_commands

sounds an alternative, but, could be nice to achieve it the backuppc's way.

My question in fact is about interpreting current config?
Is there tools which displays when a host will be backuped given the
current full configuration?

For my purpose, I would like to backup hosts one per day, and a
particular day for each host.

Regards
Sylvain.

--
Sylvain Viart.
Gmail.

I have a script that runs every day at 6:30. The script checks to see if it's a Friday, or the last day of the month. If it is, it forces a full backup of every host. I then let BackupPC handle all the incrementals. I tell BackupPC to schedules it's full backups every 10 days apart. Since I run fulls every 7 days from my script, BackupPC never schedules fulls itself.

Gerald

Sylvain Viart - Gmail wrote:

Quote:

Hi,

I've a backup server which goal is to backup other backuppc servers.

So I've few hosts to backup, but they are pretty heavy, but not that
much compared to other Thread.

As described many times, backup use a best effort policy, trying to
start backup when it is allowed to do so, and when it detects than a
host is due to be backuped.

So I've found:

Quote:

It's possible to schedule a backup. Just disable backups of that host, and
schedule the backup using cron.

http://backuppc.wiki.sourceforge.net/serverMesg_commands

sounds an alternative, but, could be nice to achieve it the backuppc's way.

My question in fact is about interpreting current config?
Is there tools which displays when a host will be backuped given the
current full configuration?

Not exactly, but if you look at the host summary page you can figure it out. A
backup happens when the time since the last backup (which is shown) reaches the
time you've set for the backup interval, and it will start at the first wakeup
that is not in a blackout interval.

Quote:

For my purpose, I would like to backup hosts one per day, and a
particular day for each host.

If you run a backup manually, the schedule will stay about the same as long a
the machine stays up because the scheduling is based on elapsed time afterwards.
But, you can probably control things by setting the blackout periods for the
times you don't want backups to start for each host.

--
Les Mikesell
lesmikesell < at > gmail.com


------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Hi,

Quote:

Quote:

My question in fact is about interpreting current config?
Is there tools which displays when a host will be backuped given the
current full configuration?

Not exactly, but if you look at the host summary page you can figure it out. A
backup happens when the time since the last backup (which is shown) reaches the
time you've set for the backup interval, and it will start at the first wakeup
that is not in a blackout interval.

hum, not really.
You're right, but for example backuppc started a backup somewhat during
my editing of the config files.
And the summary looks like :

Quote:

Host backuppc-05 Backup Summary

This PC has never been backed up!!

* This PC is used by backuppc.
* Last status is state "idle" (backup canceled by user) as of
2010-03-09 16:51.
* Last error is "fileListReceive failed".
* Pings to backuppc-05 have succeeded 1 consecutive times.

Which seems totally useless.
may be a debug mode?

The problem is that for each host I want to backup, will take 14h hour
or more for the backup to complete.
So I have to wait many days to see the result, and if I continue as it,
will take month to setup the right timing.

worse, I probably need to stop the backup on the host, because it's a
backuppc server.
hum, another $Conf{BlackoutPeriods} , should probably do the job on the
remote host here.

Of course the blackout behavior is a good thing for backup purpose.
I mean, it's better to exclude dir, than to miss a new not included dir,
form a backup point of view.

It's the same for blackout, unfortunately, it's not very human friendly.


May be the backuppc schedule engine could be hacked in some way to
produce a simulation tool?

Quote:

Quote:

For my purpose, I would like to backup hosts one per day, and a
particular day for each host.

If you run a backup manually, the schedule will stay about the same as long a
the machine stays up because the scheduling is based on elapsed time afterwards.
But, you can probably control things by setting the blackout periods for the
times you don't want backups to start for each host.

It may ends with crontab scheduling.

I will post back other userfull parameter I found about that.
Yes, BlackoutPeriod to stop remote host a specific day + a cron on the
backup server, will be easyier to setup.
I gonna try that.

Instead of reversing this behavior in full BlackoutPeriod syntax.
KISS principle.

Thanks for suggestions.

Regards,
Sylvain.

--
Sylvain Viart.
Gmail.


------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On 3/9/2010 10:21 AM, Sylvain Viart - Gmail wrote:

Quote:

Hi,

Quote:

Quote:

My question in fact is about interpreting current config?
Is there tools which displays when a host will be backuped given the
current full configuration?

Not exactly, but if you look at the host summary page you can figure it out. A
backup happens when the time since the last backup (which is shown) reaches the
time you've set for the backup interval, and it will start at the first wakeup
that is not in a blackout interval.

hum, not really.
You're right, but for example backuppc started a backup somewhat during
my editing of the config files.

The first backup will start immediately when you add a host unless
something else blocks it, so that's expected. The blackout interval
isn't respected until the host has been seen to be available outside of
it for a configurable number of times.

Quote:

And the summary looks like :

Quote:

Host backuppc-05 Backup Summary

This PC has never been backed up!!

* This PC is used by backuppc.
* Last status is state "idle" (backup canceled by user) as of
2010-03-09 16:51.
* Last error is "fileListReceive failed".
* Pings to backuppc-05 have succeeded 1 consecutive times.

Which seems totally useless.

No, if it is idle and hasn't had a backup it is going to try again at
the next wakeup - unless that number of pings is enough to make it
observe the blackout.

Quote:

may be a debug mode?

When you cancelled it, you had the opportunity to set a time to wait
before trying again.

Quote:

The problem is that for each host I want to backup, will take 14h hour
or more for the backup to complete.
So I have to wait many days to see the result, and if I continue as it,
will take month to setup the right timing.

Just add the hosts at the time you want each to start. Or go to the
cancel backup screen and put in a time to wait before starting.

Quote:

worse, I probably need to stop the backup on the host, because it's a
backuppc server.
hum, another $Conf{BlackoutPeriods} , should probably do the job on the
remote host here.

Of course the blackout behavior is a good thing for backup purpose.
I mean, it's better to exclude dir, than to miss a new not included dir,
form a backup point of view.

More relevant, you want the server to manage concurrency and to use
whatever time window it can without breaking other things.

Quote:

It may ends with crontab scheduling.

Then it is left up to you to handle concurrency - and you probably can't
do that as well as the server since you don't know when any backup will
finish.

Quote:

I will post back other userfull parameter I found about that.
Yes, BlackoutPeriod to stop remote host a specific day + a cron on the
backup server, will be easyier to setup.
I gonna try that.

Instead of reversing this behavior in full BlackoutPeriod syntax.
KISS principle.

Actually, I think scheduling is going to be the least of your problems
in trying to make one backuppc instance back up others. I'd expect it
to take weeks to restore one to a point where it would be usable again.
I'd test that before going much farther.

--
Les Mikesell
lesmikesell < at > gmail.com

------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Sylvain Viart - Gmail wrote:

Quote:

It may ends with crontab scheduling.

I will post back other userfull parameter I found about that.
Yes, BlackoutPeriod to stop remote host a specific day + a cron on the
backup server, will be easyier to setup.
I gonna try that.

This is what I do in cron for the servers where timing is critical:

00 21 * * 0 /usr/local/BackupPC/bin/BackupPC_serverMesg backup hudson
hudson backuppc 1
00 21 * * 1-6 /usr/local/BackupPC/bin/BackupPC_serverMesg backup hudson
hudson backuppc 0

This runs a full backup every Sunday night and incrementals Monday -
Saturday nights.

--
Bowie

------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Hi,

[...]

Quote:

My question in fact is about interpreting current config?

or a debug mode which interprets all the schedule?

Quote:

Quote:

Quote:

The problem is that for each host I want to backup, will take 14h hour
or more for the backup to complete.
So I have to wait many days to see the result, and if I continue as it,
will take month to setup the right timing.

Just add the hosts at the time you want each to start. Or go to the
cancel backup screen and put in a time to wait before starting.

Sorry, that's not the point, I think, I didn't explained the good things.

I blackout the remote host to backup, so the remote backup, will be "frozen" buring backup.

Quote:

Quote:

Quote:

Of course the blackout behavior is a good thing for backup purpose.
I mean, it's better to exclude dir, than to miss a new not included dir,
form a backup point of view.

More relevant, you want the server to manage concurrency and to use
whatever time window it can without breaking other things.

You're right another point for backuppc!

My problem, is that I want to have a more grained management on the time window. But doing that in the reverse fashion of the BlackOutPeriod, is really a pain.

That's why I would like to have a somewhat debug out of what backuppc will "understand" of my config.
Because every thing is linked :

• Monday backuppc-01
• Wednesday backuppc-03
  
• Friday backuppc-02
• etc.
  

And on each host a 1 day blackout period, also setuped.

$Conf{BlackoutPeriods} = [
# #741
# pas de backup le jeudi, c'est lui qui se fait backupé !
{
hourBegin => 0,
hourEnd => 23,
weekDays => [4],
},
{
hourBegin => 10.0,
hourEnd => 22.0,
weekDays => [1, 2, 3, 5],
},
# le WE c'est la fête !
];

Quote:

Quote:

Then it is left up to you to handle concurrency - and you probably can't
do that as well as the server since you don't know when any backup will
finish.

Yes you're right.
But I prefer to handle the concurrency because IO and consistency will be a nightmare, if backup overlap.

I may be, I'm on the wrong way.

Quote:

Quote:

Actually, I think scheduling is going to be the least of your problems
in trying to make one backuppc instance back up others. I'd expect it
to take weeks to restore one to a point where it would be usable again.
I'd test that before going much farther.

Thanks, I gonna try that also.
I also think It will be long, and may be it's a bad idea to backup that way...
Time will show me.

Regards,
Sylvain.

Quote:

--
Sylvain Viart.
Gmail.

Sylvain Viart - Gmail wrote:

Quote:

Quote:

Actually, I think scheduling is going to be the least of your problems
in trying to make one backuppc instance back up others. I'd expect it
to take weeks to restore one to a point where it would be usable again.
I'd test that before going much farther.

Thanks, I gonna try that also.
I also think It will be long, and may be it's a bad idea to backup that
way...
Time will show me.

I'd guess it would be more practical to just have the 2nd box hit the same
targets as the other one directly instead of copying the backups, perhaps
skewing the blackouts so they don't both hit the same target at once. Or if
that won't work, do some sort of image copy of the archive disks.

--
Les Mikesell
lesmikesell < at > gmail.com


------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Hi,

Quote:

Quote:

I'd guess it would be more practical to just have the 2nd box hit the same
targets as the other one directly instead of copying the backups, perhaps
skewing the blackouts so they don't both hit the same target at once. Or if
that won't work, do some sort of image copy of the archive disks.

Yes, may be.

In fact, the basic idea was to not resetup the full config of every backuppc (primaries backup server) on a secondary backup server.
Our main backuppc has 86 hosts and 278.15 Gb of data. On others, far less, but many more files or more data.

And the second stage backup (done by the secondary) could also be heavy for every host, plus I should keep all the config synchronized between every backuppc server. => nightmare too

I'm not sure it will be simpler.

Here is the backup policy I'm trying to setup on the secondary backup

• 1 full every 2 month
  
• 1 incr / week ~ 8 between full
  
• 1 server max per day
  
• stop backups on the remote server, when they are backuped, even before if some backup are long...
  
• don't backup one specific server during work hour
  

And I'm fighting hard.

The only difficulty in fact is the stop.
If I don't have to stop the backup on the remote server, it seems pretty simple.

I may stop violently the remote server, even if it was in a middle of a long running backup.... (I'm thinking)

With a magic wand, I wish that the secondary backup could tell the remote backup that it would be backuped tomorrow...

Now, I gonna make a magic wand!

Regards,
Sylvain.

Quote:

--
Sylvain Viart.
Gmail.

On 3/10/2010 9:13 AM, Sylvain Viart - Gmail wrote:

Quote:

Hi,

Quote:

I'd guess it would be more practical to just have the 2nd box hit the same
targets as the other one directly instead of copying the backups, perhaps
skewing the blackouts so they don't both hit the same target at once. Or if
that won't work, do some sort of image copy of the archive disks.

Yes, may be.

In fact, the basic idea was to not resetup the full config of every
backuppc (primaries backup server) on a secondary backup server.
Our main backuppc has 86 hosts and 278.15 Gb of data. On others, far
less, but many more files or more data.

And the second stage backup (done by the secondary) could also be heavy
for every host, plus I should keep all the config synchronized between
every backuppc server. => nightmare too

I'm not sure it will be simpler.

But, your real problem is that even if you get the scheme mostly
working, you will not be able to restore a primary backuppc server to a
usable state in a reasonable amount of time. Most people aren't able to
do this even with more efficient schemes like native rsync. And you
won't be able to use the files directly like you might if you did
straight rsync or an image copy.

Quote:

Here is the backup policy I'm trying to setup on the secondary backup

* 1 full every 2 month
* 1 incr / week ~ 8 between full
* 1 server max per day
* stop backups on the remote server, when they are backuped, even
before if some backup are long...
* don't backup one specific server during work hour


And I'm fighting hard.

Try a test restore to see if it is worth the fight.

Quote:

The only difficulty in fact is the stop.
If I don't have to stop the backup on the remote server, it seems pretty
simple.

I may stop violently the remote server, even if it was in a middle of a
long running backup.... (I'm thinking)

With a magic wand, I wish that the secondary backup could tell the
remote backup that it would be backuped tomorrow...

Now, I gonna make a magic wand!

You have the magic wand in the form of the pre/post commands that you
can run on a target.

--
Les Mikesell
lesmikesell < at > gmail.com

------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Les,

Quote:

But, your real problem is that even if you get the scheme mostly
working, you will not be able to restore a primary backuppc server to a
usable state in a reasonable amount of time. Most people aren't able to
do this even with more efficient schemes like native rsync. And you
won't be able to use the files directly like you might if you did
straight rsync or an image copy.

Totally.

In fact, I think another idea.

Why not using the Precommand to

# stop the backup violently
/etc/init.d/backuppc stop

# extract every hosts
for host in /etc/backuppc/hosts
extract the last backup for $host | ssh to_secondary_backup
/local/image/$host
done

# if I could do diff rsync would be better

nothing to really backup on this host, would be localhost (secondary
backup) may be.
/etc/init.d/backuppc start

Quote:

Try a test restore to see if it is worth the fight.

in progress...

Quote:

You have the magic wand in the form of the pre/post commands that you
can run on a target.

yep, I've put some bash's runes over it!

Is there a good section on the wiki for that?
(not for magic wand making, but for the bash script!!)

Thanks for your suggestions.
Regards,
Sylvain.

--
Sylvain Viart.
Gmail.


------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

----- "Sylvain Viart - Gmail" <sylvain.viart < at > gmail.com> wrote:

Quote:

Les,

Quote:

But, your real problem is that even if you get the scheme mostly
working, you will not be able to restore a primary backuppc server to a
usable state in a reasonable amount of time. Most people aren't able to
do this even with more efficient schemes like native rsync. And you
won't be able to use the files directly like you might if you did
straight rsync or an image copy.

Totally.

In fact, I think another idea.

Why not using the Precommand to

# stop the backup violently
/etc/init.d/backuppc stop

# extract every hosts
for host in /etc/backuppc/hosts
extract the last backup for $host | ssh to_secondary_backup
/local/image/$host
done

Isn't this essentially what archive does?

Quote:

# if I could do diff rsync would be better

nothing to really backup on this host, would be localhost (secondary
backup) may be.
/etc/init.d/backuppc start

Quote:

Try a test restore to see if it is worth the fight.

in progress...

Quote:

You have the magic wand in the form of the pre/post commands that you
can run on a target.

yep, I've put some bash's runes over it!

Is there a good section on the wiki for that?
(not for magic wand making, but for the bash script!!)

Thanks for your suggestions.
Regards,
Sylvain.

--
Sylvain Viart.
Gmail.


------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/