SearchFAQMemberlist Log in
Reply to topic Page 1 of 1
Chained copy job
Author Message
Post Chained copy job 
Hello,

is it possible to use chained copy jobs? For example i would like to
copy my full backups from local disk to usb disk and after that to an
nas storage.

Job {
Name = "backup-all"
JobDefs = "DefaultBackup"
Client = backup-fd
FileSet = "backup-all"
Storage = backup
Full Backup Pool = backup-monthly
Incremental Backup Pool = backup-daily
Differential Backup Pool = backup-weekly
}

Job {
Name = "backup-copy-monthly-usb"
JobDefs = "DefaultCopy"
Client = backup-fd
FileSet = "backup-all"
Schedule = "MonthlyCopy"
Storage = backup
Pool = backup-monthly
Selection Pattern = "
SELECT max(jobid)
FROM job
WHERE name = 'backup-all'
AND type = 'B'
AND level = 'F'
AND jobstatus = 'T';"
}

Job {
Name = "backup-copy-monthly-nas"
JobDefs = "DefaultCopy"
Client = backup-fd
FileSet = "backup-all"
Schedule = "MonthlyCopy2"
Storage = backup
Pool = backup-monthly
Selection Pattern = "
SELECT max(jobid)
FROM job
WHERE name = 'backup-copy-monthly-usb'
AND type = 'c'
AND level = 'F'
AND jobstatus = 'T';"
}

Pool {
Name = backup-monthly
Pool Type = Backup
Recycle = yes
RecyclePool = scratch
AutoPrune = yes
ActionOnPurge = Truncate
Volume Retention = 2 months
Volume Use Duration = 23 hours
LabelFormat =
"backup-full_${Year}-${Month:p/2/0/r}-${Day:p/2/0/r}-${Hour:p/2/0/r}:${Minute:p/2/0/r}"
Next Pool = backup-monthly-usb
}

Pool {
Name = backup-monthly-usb
Storage = backup-usb
Pool Type = Backup
Recycle = yes
RecyclePool = scratch
AutoPrune = yes
ActionOnPurge = Truncate
Volume Retention = 2 months
Volume Use Duration = 23 hours
LabelFormat =
"backup-full_${Year}-${Month:p/2/0/r}-${Day:p/2/0/r}-${Hour:p/2/0/r}:${Minute:p/2/0/r}"
Next Pool = backup-monthly-nas
}

Pool {
Name = backup-daily-nas
Storage = backup-nas
Pool Type = Backup
Recycle = yes
RecyclePool = scratch
AutoPrune = yes
ActionOnPurge = Truncate
Volume Retention = 7 days
Volume Use Duration = 23 hours
LabelFormat =
"backup-incr_${Year}-${Month:p/2/0/r}-${Day:p/2/0/r}-${Hour:p/2/0/r}:${Minute:p/2/0/r}"
}

If i run the sql statement from "backup-copy-monthly-nas" by hand, the
correct jobid is selected which should get the "read storage", "write
storage" and "next pool" from the job "backup-copy-monthly-usb".
Unfortunatly, bacula ignores the sql statement and is getting the jobid
from "backup-all" which will end in an duplicate copy at the storage
"backup-usb". Sad

Did i something wrong or is not bacula able to use two different "next
pools" / "storages"?

Regards, Dennis


------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here
http://p.sf.net/sfu/sfd2d-msazure
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Post Chained copy job 
On Mon, Apr 02, 2012 at 08:20:17PM +0200, Dennis Hoppe wrote:
Hello,

is it possible to use chained copy jobs? For example i would like to
copy my full backups from local disk to usb disk and after that to an
nas storage.


Hi Dennis,

I see your definition below is lacking the "SQLQuery" for the
"Selection Type", might this be part of the problem?

All the best,

Uwe

Job {
Name = "backup-all"
JobDefs = "DefaultBackup"
Client = backup-fd
FileSet = "backup-all"
Storage = backup
Full Backup Pool = backup-monthly
Incremental Backup Pool = backup-daily
Differential Backup Pool = backup-weekly
}

Job {
Name = "backup-copy-monthly-usb"
JobDefs = "DefaultCopy"
Client = backup-fd
FileSet = "backup-all"
Schedule = "MonthlyCopy"
Storage = backup
Pool = backup-monthly
Selection Pattern = "
SELECT max(jobid)
FROM job
WHERE name = 'backup-all'
AND type = 'B'
AND level = 'F'
AND jobstatus = 'T';"
}

Job {
Name = "backup-copy-monthly-nas"
JobDefs = "DefaultCopy"
Client = backup-fd
FileSet = "backup-all"
Schedule = "MonthlyCopy2"
Storage = backup
Pool = backup-monthly
Selection Pattern = "
SELECT max(jobid)
FROM job
WHERE name = 'backup-copy-monthly-usb'
AND type = 'c'
AND level = 'F'
AND jobstatus = 'T';"
}

Pool {
Name = backup-monthly
Pool Type = Backup
Recycle = yes
RecyclePool = scratch
AutoPrune = yes
ActionOnPurge = Truncate
Volume Retention = 2 months
Volume Use Duration = 23 hours
LabelFormat =
"backup-full_${Year}-${Month:p/2/0/r}-${Day:p/2/0/r}-${Hour:p/2/0/r}:${Minute:p/2/0/r}"
Next Pool = backup-monthly-usb
}

Pool {
Name = backup-monthly-usb
Storage = backup-usb
Pool Type = Backup
Recycle = yes
RecyclePool = scratch
AutoPrune = yes
ActionOnPurge = Truncate
Volume Retention = 2 months
Volume Use Duration = 23 hours
LabelFormat =
"backup-full_${Year}-${Month:p/2/0/r}-${Day:p/2/0/r}-${Hour:p/2/0/r}:${Minute:p/2/0/r}"
Next Pool = backup-monthly-nas
}

Pool {
Name = backup-daily-nas
Storage = backup-nas
Pool Type = Backup
Recycle = yes
RecyclePool = scratch
AutoPrune = yes
ActionOnPurge = Truncate
Volume Retention = 7 days
Volume Use Duration = 23 hours
LabelFormat =
"backup-incr_${Year}-${Month:p/2/0/r}-${Day:p/2/0/r}-${Hour:p/2/0/r}:${Minute:p/2/0/r}"
}

If i run the sql statement from "backup-copy-monthly-nas" by hand, the
correct jobid is selected which should get the "read storage", "write
storage" and "next pool" from the job "backup-copy-monthly-usb".
Unfortunatly, bacula ignores the sql statement and is getting the jobid
from "backup-all" which will end in an duplicate copy at the storage
"backup-usb". Sad

Did i something wrong or is not bacula able to use two different "next
pools" / "storages"?

Regards, Dennis




------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here
http://p.sf.net/sfu/sfd2d-msazure
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users


--
uwe.schuerkamp < at > nionex.net fon: [+49] 5242.91 - 4740, fax:-69 72
Hauptsitz: Avenwedder Str. 55, D-33311 Gütersloh, Germany
Registergericht Gütersloh HRB 4196, Geschäftsführer: H. Gosewehr, D. Suda
NIONEX --- Ein Unternehmen der Bertelsmann AG



------------------------------------------------------------------------------
Better than sec? Nothing is better than sec when it comes to
monitoring Big Data applications. Try Boundary one-second
resolution app monitoring today. Free.
http://p.sf.net/sfu/Boundary-dev2dev
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Post Chained copy job 
Hello Uwe,

Am 05.04.2012 10:47, schrieb Uwe Schuerkamp:
I see your definition below is lacking the "SQLQuery" for the
"Selection Type", might this be part of the problem?

the "Selection Type" is defined at the following "JobDefs". I read
somwhere that i have to use a "Selection Type" instead of
"PoolUncopiedJobs", because it does not set a "priorjobid".

JobDefs {
Name = "DefaultCopy"
Type = Copy
Selection Type = SQL Query
Level = Incremental
Client = backup-fd
FileSet = "backup-all"
Schedule = "DailyCopy"
Storage = backup-usb
Messages = Standard
Pool = backup-daily
Allow Duplicate Jobs = yes
Allow Higher Duplicates = no
Cancel Queued Duplicates = no
Priority = 20
Write Bootstrap = "/var/lib/bacula/%c.bsr"
}

Maybe i should give you a more verbose example.

The last backup of my Job "backup-all" looks like:
jobid | name | type | level | priorjobid
-------+------------+------+-------+------------
1123 | backup-all | B | F | 0

The last copy of my Job "backup-copy-month-usb" looks like:
jobid | name | type | level | priorjobid
-------+-------------------------+------+-------+------------
1139 | backup-copy-monthly-usb | c | F | 0

The current values of my Job "backup-copy-monthly-nas" would look like:
JobName: backup-copy-monthly-nas
Bootstrap: *None*
Client: backup-fd
FileSet: backup-all
Pool: backup-monthly (From Job resource)
Read Storage: backup (From Job resource)
Write Storage: backup-usb (From Storage from Pool's NextPool resource)
JobId: *None*
When: 2012-04-05 11:54:16
Catalog: MyCatalog
Priority: 20

At this point the "Pool" and "Read Storage" should be used from the
"JobId" "1139" and point via "NextPool" to "backup-monthly-nas". But in
this case the correct "JobId" is selected by bacula, but the wrong Pools
/ Storages are used.

05-Apr 12:02 backup-dir JobId 1324: The following 1 JobId was chosen to
be copied: 1139
05-Apr 12:02 backup-dir JobId 1324: Copying using JobId=1139
Job=backup-copy-monthly-usb.2012-04-01_06.05.00_51
05-Apr 12:02 backup-dir JobId 1324: No files found to read. No bootstrap
file written.
05-Apr 12:02 backup-dir JobId 1324: Previous Job has no data to copy.
05-Apr 12:02 backup-dir JobId 1324: Bacula backup-dir 5.0.2 (28Apr10):
05-Apr-2012 12:02:44
Build OS: i486-pc-linux-gnu debian squeeze/sid
Prev Backup JobId: 1139
Prev Backup Job: backup-copy-monthly-usb.2012-04-01_06.05.00_51
New Backup JobId: 0
Current JobId: 1324
Current Job: backup-copy-monthly-nas.2012-04-05_12.02.42_18
Backup Level: Full
Client: backup-fd
FileSet: "backup-all" 2012-02-22 23:05:00
Read Pool: "backup-monthly" (From Job resource)
Read Storage: "backup" (From Job resource)
Write Pool: "backup-monthly-usb" (From Job Pool's NextPool
resource)
Write Storage: "backup-usb" (From Storage from Pool's
NextPool resource)
Catalog: "MyCatalog" (From Client resource)
Start time: 05-Apr-2012 12:02:44
End time: 05-Apr-2012 12:02:44
Elapsed time: 0 secs
Priority: 20
SD Files Written: 0
SD Bytes Written: 0 (0 B)
Rate: 0.0 KB/s
Volume name(s):
Volume Session Id: 0
Volume Session Time: 0
Last Volume Bytes: 0 (0 B)
SD Errors: 0
SD termination status:
Termination: Copying -- no files to copy

I am wondering how Bacula is getting the "NextPool", because the
Database shows nothing and why Bacula thinks there is nothing to copy.

poolid | name | nextpoolid
--------+--------------------+------------
4 | backup-monthly | 0
5 | backup-daily | 0
6 | backup-weekly | 0
50 | backup-monthly-usb | 0
51 | backup-daily-usb | 0
52 | backup-weekly-usb | 0
89 | backup-monthly-nas | 0
90 | backup-daily-nas | 0
91 | backup-weekly-nas | 0
(9 Zeilen)

Regards, Dennis


------------------------------------------------------------------------------
Better than sec? Nothing is better than sec when it comes to
monitoring Big Data applications. Try Boundary one-second
resolution app monitoring today. Free.
http://p.sf.net/sfu/Boundary-dev2dev
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Post Chained copy job 
On Thu, Apr 05, 2012 at 12:30:43PM +0200, Dennis Hoppe wrote:
Hello Uwe,

the "Selection Type" is defined at the following "JobDefs". I read
somwhere that i have to use a "Selection Type" instead of
"PoolUncopiedJobs", because it does not set a "priorjobid".


Hi Dennis,

sorry I must have overlooked that bit. Thanks to your previous example
I have now been able to define a fine-grained copy job for one of my
pools as I wasn't even aware of the SQLQuery Selection type Wink

I'm happy to report that in my case, everything worked as expected so
checked your config for any obvious boo-boos (I think you can tell by
now that my bacula expertise is still very superficial, even after
five years of admin'ing multiple deployments).


I am wondering how Bacula is getting the "NextPool", because the
Database shows nothing and why Bacula thinks there is nothing to copy.

poolid | name | nextpoolid
--------+--------------------+------------
4 | backup-monthly | 0
5 | backup-daily | 0
6 | backup-weekly | 0
50 | backup-monthly-usb | 0
51 | backup-daily-usb | 0
52 | backup-weekly-usb | 0
89 | backup-monthly-nas | 0
90 | backup-daily-nas | 0
91 | backup-weekly-nas | 0

Just out of curiousity, are all your online / disk based pools living
in the same directory?

All the best, Uwe




--
NIONEX --- Ein Unternehmen der Bertelsmann AG



------------------------------------------------------------------------------
Better than sec? Nothing is better than sec when it comes to
monitoring Big Data applications. Try Boundary one-second
resolution app monitoring today. Free.
http://p.sf.net/sfu/Boundary-dev2dev
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Post Chained copy job 
Hello Uwe,

Am 05.04.2012 12:38, schrieb Uwe Schuerkamp:
the "Selection Type" is defined at the following "JobDefs". I read
somwhere that i have to use a "Selection Type" instead of
"PoolUncopiedJobs", because it does not set a "priorjobid".

sorry I must have overlooked that bit. Thanks to your previous example
I have now been able to define a fine-grained copy job for one of my
pools as I wasn't even aware of the SQLQuery Selection type Wink

I'm happy to report that in my case, everything worked as expected so
checked your config for any obvious boo-boos (I think you can tell by
now that my bacula expertise is still very superficial, even after
five years of admin'ing multiple deployments).

maybe you could send me your config files? Which distribution and
version are you using?

I am wondering how Bacula is getting the "NextPool", because the
Database shows nothing and why Bacula thinks there is nothing to copy.

poolid | name | nextpoolid
--------+--------------------+------------
4 | backup-monthly | 0
5 | backup-daily | 0
6 | backup-weekly | 0
50 | backup-monthly-usb | 0
51 | backup-daily-usb | 0
52 | backup-weekly-usb | 0
89 | backup-monthly-nas | 0
90 | backup-daily-nas | 0
91 | backup-weekly-nas | 0

Just out of curiousity, are all your online / disk based pools living
in the same directory?

I have one directory for my storages (disk, usb disk, nas) and beneath
one directory for each host which will contain the volumes for my full,
differential and incremental volumes.

Regards, Dennis


------------------------------------------------------------------------------
Better than sec? Nothing is better than sec when it comes to
monitoring Big Data applications. Try Boundary one-second
resolution app monitoring today. Free.
http://p.sf.net/sfu/Boundary-dev2dev
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Post Chained copy job 
On Thu, Apr 05, 2012 at 12:47:02PM +0200, Dennis Hoppe wrote:

maybe you could send me your config files? Which distribution and
version are you using?


Hello Dennis,

I'm running Bacula 5.2.6 compiled from source on CentOS 6.2 (64bit)
with a MySQL backend.

Here's the relevant config for the pool I'm copying to tape:

Pool {
Name = Online_full
Pool Type = Backup
Storage = FileStorage_full
Recycle = yes
AutoPrune = yes
Volume Retention = 180 days
Purge Oldest Volume = yes
Recycle Oldest Volume = yes
Maximum Volumes = 28
Maximum Volume Bytes = 195G
Label Format ="full-"
Next Pool = "Offline"
}

Pool {
Storage = lto4
Name = Offline
Pool Type = Backup
Recycle = yes
AutoPrune = yes
Volume Retention = 365 days
Purge Oldest Volume = yes
Recycle Oldest Volume = yes
Maximum Volumes = 21
Label Format ="Offline-"
}

and here's my copy job definition for the full backups completed
within the last 7 days:

Job {
Name = "copy-full"
Type = Copy
Level = Full
Priority = 15
Pool = Online_full
Messages = Standard
Maximum Concurrent Jobs = 2
Schedule = Copy-full # runs once a week for now
Selection Type = SQLQuery
Selection Pattern = "
SELECT jobid
FROM Job
WHERE PoolId = 4
AND type = 'B'
AND level = 'F'
AND jobstatus = 'T'
AND EndTime >= SUBDATE(NOW(), INTERVAL 7 DAY)
GROUP BY Name;"

}

I'm restricting the copy jobs artificially for now because we have run
the full pool for quite a while and I've only recently started messing
around with copy jobs, so for this pool I only want jobs completed in
the last week to be copied over to tape. I bet there's a more clever
way to implement this, but your SQLQuery thingy turned out to be a
godsend for the task at hand, although it could probably done with
AllowDuplicateSomething or other in a more supported and elegant
fashion.

All the best, Uwe



--
NIONEX --- Ein Unternehmen der Bertelsmann AG



------------------------------------------------------------------------------
Better than sec? Nothing is better than sec when it comes to
monitoring Big Data applications. Try Boundary one-second
resolution app monitoring today. Free.
http://p.sf.net/sfu/Boundary-dev2dev
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Post Chained copy job 
Hello Uwe,

Am 05.04.2012 13:08, schrieb Uwe Schuerkamp:
On Thu, Apr 05, 2012 at 12:47:02PM +0200, Dennis Hoppe wrote:
maybe you could send me your config files? Which distribution and
version are you using?

I'm running Bacula 5.2.6 compiled from source on CentOS 6.2 (64bit)
with a MySQL backend.

Here's the relevant config for the pool I'm copying to tape:

Pool {
Name = Online_full
Pool Type = Backup
Storage = FileStorage_full
Recycle = yes
AutoPrune = yes
Volume Retention = 180 days
Purge Oldest Volume = yes
Recycle Oldest Volume = yes
Maximum Volumes = 28
Maximum Volume Bytes = 195G
Label Format ="full-"
Next Pool = "Offline"
}

Pool {
Storage = lto4
Name = Offline
Pool Type = Backup
Recycle = yes
AutoPrune = yes
Volume Retention = 365 days
Purge Oldest Volume = yes
Recycle Oldest Volume = yes
Maximum Volumes = 21
Label Format ="Offline-"
}

and here's my copy job definition for the full backups completed
within the last 7 days:

Job {
Name = "copy-full"
Type = Copy
Level = Full
Priority = 15
Pool = Online_full
Messages = Standard
Maximum Concurrent Jobs = 2
Schedule = Copy-full # runs once a week for now
Selection Type = SQLQuery
Selection Pattern = "
SELECT jobid
FROM Job
WHERE PoolId = 4
AND type = 'B'
AND level = 'F'
AND jobstatus = 'T'
AND EndTime >= SUBDATE(NOW(), INTERVAL 7 DAY)
GROUP BY Name;"

}

I'm restricting the copy jobs artificially for now because we have run
the full pool for quite a while and I've only recently started messing
around with copy jobs, so for this pool I only want jobs completed in
the last week to be copied over to tape. I bet there's a more clever
way to implement this, but your SQLQuery thingy turned out to be a
godsend for the task at hand, although it could probably done with
AllowDuplicateSomething or other in a more supported and elegant
fashion.

i am also able to copy the volume from "backup-monthly" to
"backup-monthly-usb", but i also want to copy the volume from
"backup-monthly-usb" to "backup-monthly-nas".

backup-monthly -> backup-monthly-usb -> backup-monthly-nas
(primary) (fallback) (different building)

I think Bacula is not able to recognize the second "NextPool" at
"backup-monthly-usb", so it says something like "Hey, i already have
copied your volume from 'backup-monthly' to 'backup-monthly-usb'. There
is nothing else to do".

Regards, Dennis


------------------------------------------------------------------------------
Better than sec? Nothing is better than sec when it comes to
monitoring Big Data applications. Try Boundary one-second
resolution app monitoring today. Free.
http://p.sf.net/sfu/Boundary-dev2dev
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Display posts from previous:
Reply to topic Page 1 of 1
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
  


Magic SEO URL for phpBB