Backup Central

Attachments: bacula-dir.conf bacula-sd.conf

G'day guys,

Just trying to design a backup solution using Bacula for a small
company I work for and would appreciate some help with a few issues.
This email may be rather long, so certainly appreciate anyone taking
the time to read it, let alone offer any insight they may have!

The main problem I'm having is that I want backups both on-site (for
restoring files users accidentally deleted and other relatively
trivial matters) and off-site (for when the site gets stepped on by
Godzilla). Methinks I'm after (upcoming?) "copy job" magic... :-)

The company currently has two 200-odd GB USB-accessible HDDs and five
110-odd GB USB accessible HDDs. I'd like to avoid having to acquire
any further hardware at this point and think that this should be
enough to hold the required data anyway, at least following the scheme
outlined below.

My current idea runs like this:
- A monthly full backup of each machine to one of the 200 GB drives
(each machine uses it's own full backup pool)
- This drive is then taken off-site and the other 200 GB drive put
in its place for the next monthly full backup
- Weekday night-run differential backups to one of the 110 GB
drives (each machine uses it's own differential backup pool)
- This drive is then taken off-site and the 110 GB drive for the
next differential backup is put in its place

This would mean that with the just the full backup and the previous
day's differential backup drives from off-site, the previous day's
state could be completely restored. Bet I've missed some really
obviously nicer way of achieving this or something similar though! I
don't believe that too much data will be changing on a daily basis,
so hopefully the increasingly large differential backups throughout
the month won't be a problem.

Now I also want to be able to access the backups on-site, without
having to drag in off-site backup drives. I'd prefer to do the
actual backup to the removable drives in the first instance as
these are the "critical" ones and I'd like the job(s) to fail in
the case of full removable drives. I've thought of:
- Copying the backup volumes from the removable drives to a local
location following a backup. Problems / potential problems:
- Have to know the names of the relevant volumes on the removable
media
- Would really like to be able to specify restoring from the
relocated volumes in a nice manner, rather than those on the
removable media
- Migrating the volumes from the removable media to a local
location following a backup. Problems / potential problems:
- Would want to be able to easily use the removable-drive volumes
if the local ones go AWOL (e.g., Godzilla...)
- Would want matching volume names on local and removable
locations so that volumes are easily identifiable
- Would want volume recycling to occur on both locations

I've attached (slightly sanitised) Director and Software Director
config files for the current setup (very much alpha), in case this
helps.

Anyone have any ideas? Should I just hang on until "copy job" saves
everything? Am I being profoundly stupid in one / many ways?

By the way, the system's all-Windows and screaming along very nicely
using 2.0.2 - huzzah!

Any and all thoughts appreciated!
--
Nick Withers
email: nick < at > ni...
Web: http://www.nickwithers.com
Mobile: +61 414 397 446

Nick,

For what it might be worth to you, here's how I do this with a couple
of my clients.

Part I

Full install on a Windows 2003 NAS/Server unit with full jobs on two
of the workstations and 'sniping' production only data sitting on the
NAS/Server unit.

Two external HDD that rotate M,W,F and T,Th - same name, same drive
lettering. These house all jobs for the site. The drives go off-site
daily (small form factor - easy to transport.)

These provide historical data for some 60 days before the volumes are
marked USED and then recycled.

Part II

Director installed in our facility with VPN access to the client
site. Director has specific (custom) Job templates for NAS production
data and for one key workstation that houses its own production data.
Those jobs run and pull that specific key data off-site separate of
their disk based storage - and not the whole enchilada.

This data pool is good for 30 calendar days - one Full and 29
incremental backups. The first days volume gets recycled and another
Full runs on the 31st day starting the whole process again.

This configuration is relatively simple (and why I can pull it
off ) compared to some of the more fancy things Bacula can do.

Erich

On Mar 7, 2007, at 5:40 AM, Nick Withers wrote:

G'day guys,

Just trying to design a backup solution using Bacula for a small
company I work for and would appreciate some help with a few issues.
This email may be rather long, so certainly appreciate anyone taking
the time to read it, let alone offer any insight they may have!

The main problem I'm having is that I want backups both on-site (for
restoring files users accidentally deleted and other relatively
trivial matters) and off-site (for when the site gets stepped on by
Godzilla). Methinks I'm after (upcoming?) "copy job" magic... :-)

The company currently has two 200-odd GB USB-accessible HDDs and five
110-odd GB USB accessible HDDs. I'd like to avoid having to acquire
any further hardware at this point and think that this should be
enough to hold the required data anyway, at least following the scheme
outlined below.

My current idea runs like this:
- A monthly full backup of each machine to one of the 200 GB drives
(each machine uses it's own full backup pool)
- This drive is then taken off-site and the other 200 GB drive put
in its place for the next monthly full backup
- Weekday night-run differential backups to one of the 110 GB
drives (each machine uses it's own differential backup pool)
- This drive is then taken off-site and the 110 GB drive for the
next differential backup is put in its place

This would mean that with the just the full backup and the previous
day's differential backup drives from off-site, the previous day's
state could be completely restored. Bet I've missed some really
obviously nicer way of achieving this or something similar though! I
don't believe that too much data will be changing on a daily basis,
so hopefully the increasingly large differential backups throughout
the month won't be a problem.

Now I also want to be able to access the backups on-site, without
having to drag in off-site backup drives. I'd prefer to do the
actual backup to the removable drives in the first instance as
these are the "critical" ones and I'd like the job(s) to fail in
the case of full removable drives. I've thought of:
- Copying the backup volumes from the removable drives to a local
location following a backup. Problems / potential problems:
- Have to know the names of the relevant volumes on the removable
media
- Would really like to be able to specify restoring from the
relocated volumes in a nice manner, rather than those on the
removable media
- Migrating the volumes from the removable media to a local
location following a backup. Problems / potential problems:
- Would want to be able to easily use the removable-drive volumes
if the local ones go AWOL (e.g., Godzilla...)
- Would want matching volume names on local and removable
locations so that volumes are easily identifiable
- Would want volume recycling to occur on both locations

I've attached (slightly sanitised) Director and Software Director
config files for the current setup (very much alpha), in case this
helps.

Anyone have any ideas? Should I just hang on until "copy job" saves
everything? Am I being profoundly stupid in one / many ways?

By the way, the system's all-Windows and screaming along very nicely
using 2.0.2 - huzzah!

Any and all thoughts appreciated!
--
Nick Withers
email: nick < at > ni...
Web: http://www.nickwithers.com
Mobile: +61 414 397 446
<bacula-dir.conf>
<bacula-sd.conf>
----------------------------------------------------------------------
---
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to
share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?
page=join.php&p=sourceforge&CID=DEVDEV________________________________
_______________
Bacula-users mailing list
Bacula-users < at > li...
https://lists.sourceforge.net/lists/listinfo/bacula-users

Nick,

I use a similar approach with one significant difference and a few small
ones.

I have have periodic (daily and weekly) pools and write (in parallel
jobs for all the clients) all the backups to one volume to save disk
space and reduce the management complexity. This media goes off site.

Then, I have a separate set of jobs at a lower priority, scheduled to
run later, that write to another media for on site storage. This can be
an SD resource on a local hard drive volume. I use a disk rack with a
few TB, but anything with enough space will work. I have my volumes
configured to limit the size of each (thanks to a post I saw from Kern)
as they are used, and they get reused when the jobs expire.

The priority thing keeps the second set of jobs from overlapping with
the first set. I found that I got radically different (differential)
results when both FD jobs were running on a client at the same time. I
don't know how/why, but this seems to get me the results I want - two
relatively similar copies of the same data.

All my on site jobs expire after 2 weeks. Off site is kept for 4 weeks
(daily) or 52 weeks (weekly). Of course, you'll adjust retention for
your needs.

FWIW.

On Wed, 2007-03-07 at 22:40 +1100, Nick Withers wrote:
G'day guys,

Just trying to design a backup solution using Bacula for a small
company I work for and would appreciate some help with a few issues.
This email may be rather long, so certainly appreciate anyone taking
the time to read it, let alone offer any insight they may have!

The main problem I'm having is that I want backups both on-site (for
restoring files users accidentally deleted and other relatively
trivial matters) and off-site (for when the site gets stepped on by
Godzilla). Methinks I'm after (upcoming?) "copy job" magic... :-)

The company currently has two 200-odd GB USB-accessible HDDs and five
110-odd GB USB accessible HDDs. I'd like to avoid having to acquire
any further hardware at this point and think that this should be
enough to hold the required data anyway, at least following the scheme
outlined below.

My current idea runs like this:
- A monthly full backup of each machine to one of the 200 GB drives
(each machine uses it's own full backup pool)
- This drive is then taken off-site and the other 200 GB drive put
in its place for the next monthly full backup
- Weekday night-run differential backups to one of the 110 GB
drives (each machine uses it's own differential backup pool)
- This drive is then taken off-site and the 110 GB drive for the
next differential backup is put in its place

This would mean that with the just the full backup and the previous
day's differential backup drives from off-site, the previous day's
state could be completely restored. Bet I've missed some really
obviously nicer way of achieving this or something similar though! I
don't believe that too much data will be changing on a daily basis,
so hopefully the increasingly large differential backups throughout
the month won't be a problem.

Now I also want to be able to access the backups on-site, without
having to drag in off-site backup drives. I'd prefer to do the
actual backup to the removable drives in the first instance as
these are the "critical" ones and I'd like the job(s) to fail in
the case of full removable drives. I've thought of:
- Copying the backup volumes from the removable drives to a local
location following a backup. Problems / potential problems:
- Have to know the names of the relevant volumes on the removable
media
- Would really like to be able to specify restoring from the
relocated volumes in a nice manner, rather than those on the
removable media
- Migrating the volumes from the removable media to a local
location following a backup. Problems / potential problems:
- Would want to be able to easily use the removable-drive volumes
if the local ones go AWOL (e.g., Godzilla...)
- Would want matching volume names on local and removable
locations so that volumes are easily identifiable
- Would want volume recycling to occur on both locations

I've attached (slightly sanitised) Director and Software Director
config files for the current setup (very much alpha), in case this
helps.

Anyone have any ideas? Should I just hang on until "copy job" saves
everything? Am I being profoundly stupid in one / many ways?

By the way, the system's all-Windows and screaming along very nicely
using 2.0.2 - huzzah!

Any and all thoughts appreciated!
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________ Bacula-users mailing list Bacula-users < at > li... https://lists.sourceforge.net/lists/listinfo/bacula-users

On 7 Mar 2007 at 8:36, Don MacArthur wrote:

Nick,

I use a similar approach with one significant difference and a few small
ones.

I have have periodic (daily and weekly) pools and write (in parallel
jobs for all the clients) all the backups to one volume to save disk
space and reduce the management complexity. This media goes off site.

You say "in parallel jobs for all the clients". Please elaborate.
What do you mean?

--
Dan Langille : Software Developer looking for work
my resume: http://www.freebsddiary.org/dan_langille.php
PGCon - The PostgreSQL Conference - http://www.pgcon.org/

I apologize if I was ambiguous, it was unintentional.

I use the schedule to run a single job for each client, all starting at
10 PM, with a limit of 10 concurrent jobs.

So, about 20 jobs are initiated, and 10 run while the rest wait. When
one of the running jobs terminates, a waiting job starts running.

A similar set of jobs run several hours later, but with a different
destination (storage device) and at a different priority. When the last
of the first set of 20 jobs completes, the next set begins to run as
described in the third paragraph above.

Thanks for the question.

On Wed, 2007-03-07 at 10:44 -0500, Dan Langille wrote:
On 7 Mar 2007 at 8:36, Don MacArthur wrote:

Nick,

I use a similar approach with one significant difference and a few small
ones.

I have have periodic (daily and weekly) pools and write (in parallel
jobs for all the clients) all the backups to one volume to save disk
space and reduce the management complexity. This media goes off site.

You say "in parallel jobs for all the clients". Please elaborate.
What do you mean?

I currently have my "on-site" bacula performing backups to a large LVM
partition (no tapes). Some extra stuff gets copied to that partion (ie.
bacula db dump,etc...) and then gets rsync'd through a ssh connection
over the internet to an "off-site" LVM. One server has a monthly "full"
backup that is too large (> 120G) to transfer that way so I manually
copy it to a usb-book type disk and manually transfer it to the off-site
location. The dailys, diffs and all other fulls transfer fine (about
two/four hours in the middle of the night) without any problems. Rsync
cleans up unused files and has run hands-free every night. I have a
complete mirror copy of my bacula storage well before work starts the
next day. The two LVMs are about 1.2T in each location although your
size may vary.

Nick Withers wrote:
G'day guys,

Just trying to design a backup solution using Bacula for a small
company I work for and would appreciate some help with a few issues.
This email may be rather long, so certainly appreciate anyone taking
the time to read it, let alone offer any insight they may have!

The main problem I'm having is that I want backups both on-site (for
restoring files users accidentally deleted and other relatively
trivial matters) and off-site (for when the site gets stepped on by
Godzilla). Methinks I'm after (upcoming?) "copy job" magic... :-)

The company currently has two 200-odd GB USB-accessible HDDs and five
110-odd GB USB accessible HDDs. I'd like to avoid having to acquire
any further hardware at this point and think that this should be
enough to hold the required data anyway, at least following the scheme
outlined below.

My current idea runs like this:
- A monthly full backup of each machine to one of the 200 GB drives
(each machine uses it's own full backup pool)
- This drive is then taken off-site and the other 200 GB drive put
in its place for the next monthly full backup
- Weekday night-run differential backups to one of the 110 GB
drives (each machine uses it's own differential backup pool)
- This drive is then taken off-site and the 110 GB drive for the
next differential backup is put in its place

This would mean that with the just the full backup and the previous
day's differential backup drives from off-site, the previous day's
state could be completely restored. Bet I've missed some really
obviously nicer way of achieving this or something similar though! I
don't believe that too much data will be changing on a daily basis,
so hopefully the increasingly large differential backups throughout
the month won't be a problem.

Now I also want to be able to access the backups on-site, without
having to drag in off-site backup drives. I'd prefer to do the
actual backup to the removable drives in the first instance as
these are the "critical" ones and I'd like the job(s) to fail in
the case of full removable drives. I've thought of:
- Copying the backup volumes from the removable drives to a local
location following a backup. Problems / potential problems:
- Have to know the names of the relevant volumes on the removable
media
- Would really like to be able to specify restoring from the
relocated volumes in a nice manner, rather than those on the
removable media
- Migrating the volumes from the removable media to a local
location following a backup. Problems / potential problems:
- Would want to be able to easily use the removable-drive volumes
if the local ones go AWOL (e.g., Godzilla...)
- Would want matching volume names on local and removable
locations so that volumes are easily identifiable
- Would want volume recycling to occur on both locations

I've attached (slightly sanitised) Director and Software Director
config files for the current setup (very much alpha), in case this
helps.

Anyone have any ideas? Should I just hang on until "copy job" saves
everything? Am I being profoundly stupid in one / many ways?

By the way, the system's all-Windows and screaming along very nicely
using 2.0.2 - huzzah!

Any and all thoughts appreciated!

------------------------------------------------------------------------

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
------------------------------------------------------------------------

_______________________________________________
Bacula-users mailing list
Bacula-users < at > li...
https://lists.sourceforge.net/lists/listinfo/bacula-users

A few weeks ago there was a short thread on a similar theme. See
http://article.gmane.org/gmane.comp.sysutils.backup.bacula.general/31927
and other posts in the thread.

In that thread, the on-site disk-volumes were written first and then
copied to tape for off-site protection.

However your requirement ...

I'd prefer to do the actual backup to the removable drives in the
first instance as these are the "critical" ones and I'd like the
job(s) to fail in the case of full removable-drives.

makes this significantly different.

You said that you are all windows based. Does that include the server
running the storage daemon? Using Linux, you might be able to achieve
what you want using "bind" mounts. Here is an outline of a basic scheme.

Create a directory for the on-site disk-volumes and bind it to the
location where the storage-daemon expects to find them.

mount --bind /on-site/bacula/volumes /var/lib/bacula/volumes

This should be in /etc/rc.d/rc.local or similar.

Your storage-daemon config would include:

Device {
...
Archive Device = /var/lib/bacula/volumes
Device Type = File;
LabelMedia = yes;
Random Access = Yes;
AutomaticMount = yes;
RemovableMedia = no;
AlwaysOpen = no;
...
}

Have a pre-flight (RunBefore) script in a job with a unique, lowest
"Priority" value, to ensure that it runs alone and first. It should
unbind the on-site storage and bind the removable-media.

Pre-flight script:

umount /var/lib/bacula/volumes
mount --bind /media/usbdisk/volumes /var/lib/bacula/volumes

Have a post-flight script in a job with a unique, highest "Priority"
value to ensure that it runs alone and last. It should unbind the
removable disk, copy the disk volumes to the on-site storage, and bind
that to the appropriate disk-volume directory.

Post-flight script:

umount /var/lib/bacula/volumes
rsync -a /media/usbdisk/volumes/ /on-site/bacula/volumes/
mount --bind /on-site/bacula/volumes /var/lib/bacula/volumes

Now the storage daemon will be able to restore from the on-site copy of
the volumes.

In the resource of the Pool to which the disk volumes are assigned, you
should have "Maximum Volume Jobs = 1" to ensure that the disk-volumes
are closed after use. It won't be necessary to know a-priori the names
of the volumes since rsync will only copy new volumes. You should also
be dumping your catalogue and keeping copies of your config files
somewhere.

I haven't thought about issues with recycling but it should be
scriptable.

Hope this helps.

Kel

On Wed, 07 Mar 2007 08:36:44 -0700
Don MacArthur <macarthurd < at > cm...> wrote:

Nick,

I use a similar approach with one significant difference and a few
small ones.

I have have periodic (daily and weekly) pools and write (in parallel
jobs for all the clients) all the backups to one volume to save disk
space and reduce the management complexity.

I thought about having one big (well, bigger) volume for each day's
work, but am currently going with individual volumes for each host so
that each backup can be performed simultaneously. Have I missed
something here or would your parallel jobs not actually be parallel
due to each having to wait for the volume to become available?

This media goes off site.

Then, I have a separate set of jobs at a lower priority, scheduled
to run later, that write to another media for on site storage.
This can be an SD resource on a local hard drive volume. I use a
disk rack with a few TB, but anything with enough space will work.
I have my volumes configured to limit the size of each (thanks to a
post I saw from Kern) as they are used, and they get reused when
the jobs expire.

The priority thing keeps the second set of jobs from overlapping
with the first set. I found that I got radically different
(differential) results when both FD jobs were running on a client
at the same time. I don't know how/why, but this seems to get me
the results I want - two relatively similar copies of the same
data.

Wouldn't this create potential dramas with restores (assuming you're
using the same catalog for both jobs)? I mean, couldn't a restore
potentially demand both on-site and off-site volumes?

All my on site jobs expire after 2 weeks. Off site is kept for 4
weeks (daily) or 52 weeks (weekly). Of course, you'll adjust
retention for your needs.

FWIW.

Thanks!

On Wed, 2007-03-07 at 22:40 +1100, Nick Withers wrote:
G'day guys,

Just trying to design a backup solution using Bacula for a small
company I work for and would appreciate some help with a few
issues. This email may be rather long, so certainly appreciate
anyone taking the time to read it, let alone offer any insight
they may have!

The main problem I'm having is that I want backups both on-site
(for restoring files users accidentally deleted and other
relatively trivial matters) and off-site (for when the site gets
stepped on by Godzilla). Methinks I'm after (upcoming?) "copy
job" magic... :-)

The company currently has two 200-odd GB USB-accessible HDDs and
five 110-odd GB USB accessible HDDs. I'd like to avoid having to
acquire any further hardware at this point and think that this
should be enough to hold the required data anyway, at least
following the scheme outlined below.

My current idea runs like this:
- A monthly full backup of each machine to one of the 200 GB
drives (each machine uses it's own full backup pool)
- This drive is then taken off-site and the other 200 GB drive
put in its place for the next monthly full backup
- Weekday night-run differential backups to one of the 110 GB
drives (each machine uses it's own differential backup pool)
- This drive is then taken off-site and the 110 GB drive for the
next differential backup is put in its place

This would mean that with the just the full backup and the
previous day's differential backup drives from off-site, the
previous day's state could be completely restored. Bet I've
missed some really obviously nicer way of achieving this or
something similar though! I don't believe that too much data will
be changing on a daily basis, so hopefully the increasingly large
differential backups throughout the month won't be a problem.

Now I also want to be able to access the backups on-site, without
having to drag in off-site backup drives. I'd prefer to do the
actual backup to the removable drives in the first instance as
these are the "critical" ones and I'd like the job(s) to fail in
the case of full removable drives. I've thought of:
- Copying the backup volumes from the removable drives to a
local location following a backup. Problems / potential problems:
- Have to know the names of the relevant volumes on the
removable media
- Would really like to be able to specify restoring from the
relocated volumes in a nice manner, rather than those on the
removable media
- Migrating the volumes from the removable media to a local
location following a backup. Problems / potential problems:
- Would want to be able to easily use the removable-drive
volumes if the local ones go AWOL (e.g., Godzilla...)
- Would want matching volume names on local and removable
locations so that volumes are easily identifiable
- Would want volume recycling to occur on both locations

I've attached (slightly sanitised) Director and Software Director
config files for the current setup (very much alpha), in case this
helps.

Anyone have any ideas? Should I just hang on until "copy job"
saves everything? Am I being profoundly stupid in one / many ways?

By the way, the system's all-Windows and screaming along very
nicely using 2.0.2 - huzzah!

Any and all thoughts appreciated!
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to
share your opinions on IT & business topics through brief
surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________ Bacula-users
mailing list Bacula-users < at > li...
https://lists.sourceforge.net/lists/listinfo/bacula-users

--
Nick Withers
email: nick < at > ni...
Web: http://www.nickwithers.com
Mobile: +61 414 397 446

On Wed, 07 Mar 2007 10:31:27 -0600
Brad Larson <blarson < at > cr...> wrote:

I currently have my "on-site" bacula performing backups to a large
LVM partition (no tapes). Some extra stuff gets copied to that
partion (ie. bacula db dump,etc...) and then gets rsync'd through a
ssh connection over the internet to an "off-site" LVM. One server
has a monthly "full" backup that is too large (> 120G) to transfer
that way so I manually copy it to a usb-book type disk and manually
transfer it to the off-site location. The dailys, diffs and all
other fulls transfer fine (about two/four hours in the middle of
the night) without any problems. Rsync cleans up unused files and
has run hands-free every night. I have a complete mirror copy of
my bacula storage well before work starts the next day.

That'd be pretty dandy - a relatively elegant solution, methinks.
Unfortunately I don't really think there's a data link available to
any other location with sufficient capacity to transfer even the
nightly backup data off-site...

The two LVMs are about 1.2T in each location although your size may
vary.

Nick Withers wrote:
G'day guys,

Just trying to design a backup solution using Bacula for a small
company I work for and would appreciate some help with a few
issues. This email may be rather long, so certainly appreciate
anyone taking the time to read it, let alone offer any insight
they may have!

The main problem I'm having is that I want backups both on-site
(for restoring files users accidentally deleted and other
relatively trivial matters) and off-site (for when the site gets
stepped on by Godzilla). Methinks I'm after (upcoming?) "copy
job" magic... :-)

The company currently has two 200-odd GB USB-accessible HDDs and
five 110-odd GB USB accessible HDDs. I'd like to avoid having to
acquire any further hardware at this point and think that this
should be enough to hold the required data anyway, at least
following the scheme outlined below.

My current idea runs like this:
- A monthly full backup of each machine to one of the 200 GB
drives (each machine uses it's own full backup pool)
- This drive is then taken off-site and the other 200 GB drive
put in its place for the next monthly full backup
- Weekday night-run differential backups to one of the 110 GB
drives (each machine uses it's own differential backup pool)
- This drive is then taken off-site and the 110 GB drive for the
next differential backup is put in its place

This would mean that with the just the full backup and the
previous day's differential backup drives from off-site, the
previous day's state could be completely restored. Bet I've
missed some really obviously nicer way of achieving this or
something similar though! I don't believe that too much data will
be changing on a daily basis, so hopefully the increasingly large
differential backups throughout the month won't be a problem.

Now I also want to be able to access the backups on-site, without
having to drag in off-site backup drives. I'd prefer to do the
actual backup to the removable drives in the first instance as
these are the "critical" ones and I'd like the job(s) to fail in
the case of full removable drives. I've thought of:
- Copying the backup volumes from the removable drives to a
local location following a backup. Problems / potential problems:
- Have to know the names of the relevant volumes on the
removable media
- Would really like to be able to specify restoring from the
relocated volumes in a nice manner, rather than those on the
removable media
- Migrating the volumes from the removable media to a local
location following a backup. Problems / potential problems:
- Would want to be able to easily use the removable-drive
volumes if the local ones go AWOL (e.g., Godzilla...)
- Would want matching volume names on local and removable
locations so that volumes are easily identifiable
- Would want volume recycling to occur on both locations

I've attached (slightly sanitised) Director and Software Director
config files for the current setup (very much alpha), in case this
helps.

Anyone have any ideas? Should I just hang on until "copy job"
saves everything? Am I being profoundly stupid in one / many ways?

By the way, the system's all-Windows and screaming along very
nicely using 2.0.2 - huzzah!

Any and all thoughts appreciated!

------------------------------------------------------------------------

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to
share your opinions on IT & business topics through brief
surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
------------------------------------------------------------------------

_______________________________________________
Bacula-users mailing list
Bacula-users < at > li...
https://lists.sourceforge.net/lists/listinfo/bacula-users


--
Nick Withers
email: nick < at > ni...
Web: http://www.nickwithers.com
Mobile: +61 414 397 446

On Wed, 7 Mar 2007 13:06:15 -0800 (PST)
Kel Raywood <kelvin < at > pe...> wrote:

A few weeks ago there was a short thread on a similar theme. See
http://article.gmane.org/gmane.comp.sysutils.backup.bacula.general/31927
and other posts in the thread.

Hmmm... I think I had seen this one, but am glad you pointed it out
again, as I think I'm now much more receptive to the first idea.

In that thread, the on-site disk-volumes were written first and
then copied to tape for off-site protection.

However your requirement ...

I'd prefer to do the actual backup to the removable drives in the
first instance as these are the "critical" ones and I'd like the
job(s) to fail in the case of full removable-drives.

makes this significantly different.

It's really more of a desirable - I could handle the idea of a
separate "archive" job failing, if needs be.

I hadn't really put too much thought into the idea of having multiple
catalogs, but this could be quite dandy - I could have completely
different backup regimes for on-site and off-site stores, for
instance. Copy jobs'd sure be tops though! :-)

I guess since the off-site backups would only be used in an emergency
situation, and since a restore (using the "main" catalog) would ask
for the right volumes, which could then (or beforehand) easily
be restored from the off-site media to the on-site location this
could work quite well. I'd rather not be making the assumption that a
huge amount of storage will be available to restore the on-site
volumes to, but really that's not an unreasonable one anyway.

You said that you are all windows based. Does that include the
server running the storage daemon?

It does.

Using Linux, you might be able to achieve what you want using
"bind" mounts. Here is an outline of a basic scheme.

Create a directory for the on-site disk-volumes and bind it to the
location where the storage-daemon expects to find them.

mount --bind /on-site/bacula/volumes /var/lib/bacula/volumes

This should be in /etc/rc.d/rc.local or similar.

Your storage-daemon config would include:

Device {
...
Archive Device = /var/lib/bacula/volumes
Device Type = File;
LabelMedia = yes;
Random Access = Yes;
AutomaticMount = yes;
RemovableMedia = no;
AlwaysOpen = no;
...
}

Have a pre-flight (RunBefore) script in a job with a unique, lowest
"Priority" value, to ensure that it runs alone and first. It should
unbind the on-site storage and bind the removable-media.

Pre-flight script:

umount /var/lib/bacula/volumes
mount --bind /media/usbdisk/volumes /var/lib/bacula/volumes

Have a post-flight script in a job with a unique, highest
"Priority" value to ensure that it runs alone and last. It should
unbind the removable disk, copy the disk volumes to the on-site
storage, and bind that to the appropriate disk-volume directory.

Post-flight script:

umount /var/lib/bacula/volumes
rsync -a /media/usbdisk/volumes/ /on-site/bacula/volumes/
mount --bind /on-site/bacula/volumes /var/lib/bacula/volumes

Now the storage daemon will be able to restore from the on-site
copy of the volumes.

In the resource of the Pool to which the disk volumes are assigned,
you should have "Maximum Volume Jobs = 1" to ensure that the
disk-volumes are closed after use. It won't be necessary to know
a-priori the names of the volumes since rsync will only copy new
volumes. You should also be dumping your catalogue and keeping
copies of your config files somewhere.

I haven't thought about issues with recycling but it should be
scriptable.

Windows does allow for drives to be mounted on directories, but they
have to be empty - which unfortunately somewhat destroys the
possibility of tricking Bacula in a similar way to this (I'd quite
happily have the removable drive(s) mounted on both "G:" (for
instance) and "D:\Backups", then remove the "D:\Backups" mount to
perform a copy (assuming there's a scriptable way of doing this in
Windows), but the empty directory thing kills it.

Still - a nifty idea!

Hope this helps.

It did indeed - thanks very much.

Kel

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to
share your opinions on IT & business topics through brief
surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________ Bacula-users
mailing list Bacula-users < at > li...
https://lists.sourceforge.net/lists/listinfo/bacula-users

--
Nick Withers
email: nick < at > ni...
Web: http://www.nickwithers.com
Mobile: +61 414 397 446

Good questions.

The backups run concurrently, so the only waiting is on the max jobs
parameter. But, yes, more than one job is written to the tape at the
same time. This also keeps the data rate up and the tape drive busy, so
even if one client slows down momentarily the drive won't have to stop,
rewind the tape, and then restart. In my case, I'm using HP 960 drives,
which slow down and speed up somewhat to meet the data rate. Plus,
keeping the drive busy reduces wear-and-tear on the media and tape
heads. I don't know if the other LTO drive models have this feature.

For restores, it *could* cause me to have to use more than one tape if
files were created and then other files deleted between backups. But,
for the most part this is a remote possibility. If I'm doing disaster
recovery I'll probably use the later of the two (on site in my case) to
start the restore, and then when the off site media arrives add what the
local backups missed. If the local backups are lost due to catastrophe
(fire, earthquake, whatever...) then they'll be glad to have anything,
but the worst case will be a couple of hours of changes lost. If the
loss happens near the end of the day they won't notice the difference
between the two backups because 8 or 10 hours is already missing
anyway.

I run the off site backups first so that I have time to restart them and
still have them finish in time to get off site. Then the on site
backups run when the time crunch is over.

I haven't had to recall an offsite tape for an "oops" restore, yet...

HTH.

On Thu, 2007-03-08 at 17:25 +1100, Nick Withers wrote:
On Wed, 07 Mar 2007 08:36:44 -0700
Don MacArthur <macarthurd < at > cm...> wrote:

Nick,

I use a similar approach with one significant difference and a few
small ones.

I have have periodic (daily and weekly) pools and write (in parallel
jobs for all the clients) all the backups to one volume to save disk
space and reduce the management complexity.

I thought about having one big (well, bigger) volume for each day's
work, but am currently going with individual volumes for each host so
that each backup can be performed simultaneously. Have I missed
something here or would your parallel jobs not actually be parallel
due to each having to wait for the volume to become available?

This media goes off site.

Then, I have a separate set of jobs at a lower priority, scheduled
to run later, that write to another media for on site storage.
This can be an SD resource on a local hard drive volume. I use a
disk rack with a few TB, but anything with enough space will work.
I have my volumes configured to limit the size of each (thanks to a
post I saw from Kern) as they are used, and they get reused when
the jobs expire.

The priority thing keeps the second set of jobs from overlapping
with the first set. I found that I got radically different
(differential) results when both FD jobs were running on a client
at the same time. I don't know how/why, but this seems to get me
the results I want - two relatively similar copies of the same
data.

Wouldn't this create potential dramas with restores (assuming you're
using the same catalog for both jobs)? I mean, couldn't a restore
potentially demand both on-site and off-site volumes?

All my on site jobs expire after 2 weeks. Off site is kept for 4
weeks (daily) or 52 weeks (weekly). Of course, you'll adjust
retention for your needs.

FWIW.

Thanks!

On Wed, 2007-03-07 at 22:40 +1100, Nick Withers wrote:
G'day guys,

Just trying to design a backup solution using Bacula for a small
company I work for and would appreciate some help with a few
issues. This email may be rather long, so certainly appreciate
anyone taking the time to read it, let alone offer any insight
they may have!

The main problem I'm having is that I want backups both on-site
(for restoring files users accidentally deleted and other
relatively trivial matters) and off-site (for when the site gets
stepped on by Godzilla). Methinks I'm after (upcoming?) "copy
job" magic... :-)

The company currently has two 200-odd GB USB-accessible HDDs and
five 110-odd GB USB accessible HDDs. I'd like to avoid having to
acquire any further hardware at this point and think that this
should be enough to hold the required data anyway, at least
following the scheme outlined below.

My current idea runs like this:
- A monthly full backup of each machine to one of the 200 GB
drives (each machine uses it's own full backup pool)
- This drive is then taken off-site and the other 200 GB drive
put in its place for the next monthly full backup
- Weekday night-run differential backups to one of the 110 GB
drives (each machine uses it's own differential backup pool)
- This drive is then taken off-site and the 110 GB drive for the
next differential backup is put in its place

This would mean that with the just the full backup and the
previous day's differential backup drives from off-site, the
previous day's state could be completely restored. Bet I've
missed some really obviously nicer way of achieving this or
something similar though! I don't believe that too much data will
be changing on a daily basis, so hopefully the increasingly large
differential backups throughout the month won't be a problem.

Now I also want to be able to access the backups on-site, without
having to drag in off-site backup drives. I'd prefer to do the
actual backup to the removable drives in the first instance as
these are the "critical" ones and I'd like the job(s) to fail in
the case of full removable drives. I've thought of:
- Copying the backup volumes from the removable drives to a
local location following a backup. Problems / potential problems:
- Have to know the names of the relevant volumes on the
removable media
- Would really like to be able to specify restoring from the
relocated volumes in a nice manner, rather than those on the
removable media
- Migrating the volumes from the removable media to a local
location following a backup. Problems / potential problems:
- Would want to be able to easily use the removable-drive
volumes if the local ones go AWOL (e.g., Godzilla...)
- Would want matching volume names on local and removable
locations so that volumes are easily identifiable
- Would want volume recycling to occur on both locations

I've attached (slightly sanitised) Director and Software Director
config files for the current setup (very much alpha), in case this
helps.

Anyone have any ideas? Should I just hang on until "copy job"
saves everything? Am I being profoundly stupid in one / many ways?

By the way, the system's all-Windows and screaming along very
nicely using 2.0.2 - huzzah!

Any and all thoughts appreciated!
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to
share your opinions on IT & business topics through brief
surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________ Bacula-users
mailing list Bacula-users < at > li...
https://lists.sourceforge.net/lists/listinfo/bacula-users

I wrote:
Pre-flight script:

umount /var/lib/bacula/volumes
mount --bind /media/usbdisk/volumes /var/lib/bacula/volumes

Post-flight script:

umount /var/lib/bacula/volumes
rsync -a /media/usbdisk/volumes/ /on-site/bacula/volumes/
mount --bind /on-site/bacula/volumes /var/lib/bacula/volumes

On Thu, 8 Mar 2007, Nick Withers wrote:
Windows does allow for drives to be mounted on directories,

I assume this means you can't mount some sub-directory of a drive, as I
suggested. Note that in the above scheme, /var/lib/bacula/volumes is
empty until either the on-site or off-site storage is "bind" mounted on
it.

but they have to be empty - which unfortunately somewhat destroys the
possibility of tricking Bacula in a similar way to this (I'd quite
happily have the removable drive(s) mounted on both "G:" (for
instance) and "D:\Backups", then remove the "D:\Backups" mount to
perform a copy (assuming there's a scriptable way of doing this in
Windows), but the empty directory thing kills it.

Perhaps you could do it with some simple renaming. I'm not familiar with
windows/dos so don't know the right commands but the scripts would be
like:

Pre-flight

Rename D:\Backups to D:\On_site_Backups
Create directory D:\Backups
Mount G: on D:\Backups

Post-flight

Dismount G: from D:\Backups
Remove directory D:\Backups
Rename D:\On_site_Backups to D:\Backups
One-way-Sync (or copy) G:\Volumes\* to D:\Backups\Volumes\

In the last step (sync or copy) I assumed the storage-daemon
config contained: Archive Device = D:\Backups\Volumes

Your post-flight script should also contain commands to dump your
catalog to G: .

Kel

How about using mirroring using raid1 ? (you'd probably have to buy a
thirs 200gb).
This way, you achieve data synchronisation easily, always have a local
copy from which to run restores and you cycle between 2 disks to keep
an offsite copy.

Regards,
--
Mikael Kermorgant

On Sat, 10 Mar 2007 12:46:04 +0100
"Mikael Kermorgant" <mikael.kermorgant < at > gm...> wrote:

How about using mirroring using raid1 ? (you'd probably have to buy
a thirs 200gb).
This way, you achieve data synchronisation easily, always have a
local copy from which to run restores and you cycle between 2 disks
to keep an offsite copy.

Certainly a possibility - thanks for the suggestion!

Regards,
--
Mikael Kermorgant
--
Nick Withers
email: nick < at > ni...
Web: http://www.nickwithers.com
Mobile: +61 414 397 446