SearchFAQMemberlist Log in
Reply to topic Page 1 of 1
Maximum Concurrent Jobs in job definition
Author Message
Post Maximum Concurrent Jobs in job definition 
Hi everyone,
I have a job that runs very frequently, and successive runs take different sets of data (as determined by a pre-job script). To avoid problems, my pre-job script sets a lock file and the post-clears it.

When I configure my director for concurrent jobs = 2, i sometimes get 2 of these jobs trying to run in parallel (which is understandable, many times a run has more data than the average and is still running while the next one is due). My pre-script will then cancel the execution of the 2nd script (because it finds the lock file). Great.

Of course, this is generating a bunch of (expected) errored jobs . I read in the documentaiton [1]  that I can use "Maximum Concurrent Jobs" in the Job def, to determine how many of this jobs can start in parallel. So I set this to 1 [2]

But it has no effect . I get 2 jobs running in parallel (for a short while, until the pre-script kills it). If I change the director's Max Concurrent Jobs to 1, then it's forced to 1 (but i need more than one, as I as have spooling enabled and other jobs writing to the tape too..)

What is going on ?

TIA!
B_________________________
{Beto|Norberto|Numard} Meijome

"The only people that never change are the stupid and the dead"
 Jorge Luis Borges.

I speak for myself, not my employer. Contents may be hot. Slippery when wet.
Reading disclaimers makes you go blind. Writing them is worse. You have been
Warned.

[1] http://bacula.org/5.0.x-manuals/en/main/main/Configuring_Director.html#SECTION001730000000000000000 , way down...

[2]
[...]
Job {
  Name = "assetBackups"
  JobDefs = "DefaultJob"
  Level = Full
  FileSet="publishedAssetsSet"
  Messages = Standard
  Pool = "productionPool"
  Storage = "berthaTape"
  Schedule = "PublishedAssetsSchedule"
  Write Bootstrap = "/var/lib/bacula/%n.bsr"
  Priority = 10                
  Accurate = yes
  Spool Data = yes
  # We only want one instance of this job running < at > the same time.
  # enforced (errors out) in the apb_run.sh
  Maximum Concurrent Jobs = 1
  # This the list of asset versions to be backed up. The parameter passed is the number of versions to include in this run
  RunBeforeJob = "/etc/bacula/backupFilelist/apb_run.sh 400"
  # This moves the current list out of the way and updates the offset
  # parameters passed are the jobId and the jobStatus
  RunAfterJob  = "/etc/bacula/backupFilelist/apb_post.sh %i %e"
}
[...]

Post Maximum Concurrent Jobs in job definition 
Anyone?

btw, it is 5.0 , on Centos 5.4 64bit_________________________
{Beto|Norberto|Numard} Meijome

"The only people that never change are the stupid and the dead"
 Jorge Luis Borges.

I speak for myself, not my employer. Contents may be hot. Slippery when wet.
Reading disclaimers makes you go blind. Writing them is worse. You have been
Warned.


On 11 March 2010 13:40, Norberto Meijome <numardbsd < at > gmail.com ([email]numardbsd < at > gmail.com[/email])> wrote:
Hi everyone,
I have a job that runs very frequently, and successive runs take different sets of data (as determined by a pre-job script). To avoid problems, my pre-job script sets a lock file and the post-clears it.

When I configure my director for concurrent jobs = 2, i sometimes get 2 of these jobs trying to run in parallel (which is understandable, many times a run has more data than the average and is still running while the next one is due). My pre-script will then cancel the execution of the 2nd script (because it finds the lock file). Great.

Of course, this is generating a bunch of (expected) errored jobs . I read in the documentaiton [1]  that I can use "Maximum Concurrent Jobs" in the Job def, to determine how many of this jobs can start in parallel. So I set this to 1 [2]

But it has no effect . I get 2 jobs running in parallel (for a short while, until the pre-script kills it). If I change the director's Max Concurrent Jobs to 1, then it's forced to 1 (but i need more than one, as I as have spooling enabled and other jobs writing to the tape too..)

What is going on ?

TIA!
B_________________________
{Beto|Norberto|Numard} Meijome

"The only people that never change are the stupid and the dead"
 Jorge Luis Borges.

I speak for myself, not my employer. Contents may be hot. Slippery when wet.
Reading disclaimers makes you go blind. Writing them is worse. You have been
Warned.

[1] http://bacula.org/5.0.x-manuals/en/main/main/Configuring_Director.html#SECTION001730000000000000000 , way down...

[2]
[...]
Job {
  Name = "assetBackups"
  JobDefs = "DefaultJob"
  Level = Full
  FileSet="publishedAssetsSet"
  Messages = Standard
  Pool = "productionPool"
  Storage = "berthaTape"
  Schedule = "PublishedAssetsSchedule"
  Write Bootstrap = "/var/lib/bacula/%n.bsr"
  Priority = 10                
  Accurate = yes
  Spool Data = yes
  # We only want one instance of this job running < at > the same time.
  # enforced (errors out) in the apb_run.sh
  Maximum Concurrent Jobs = 1
  # This the list of asset versions to be backed up. The parameter passed is the number of versions to include in this run
  RunBeforeJob = "/etc/bacula/backupFilelist/apb_run.sh 400"
  # This moves the current list out of the way and updates the offset
  # parameters passed are the jobId and the jobStatus
  RunAfterJob  = "/etc/bacula/backupFilelist/apb_post.sh %i %e"
}
[...]


Post Maximum Concurrent Jobs in job definition 
Hi.

Did it work like it should in Bacula 3.0?

--
Silver

On Friday 12 March 2010 02:21:13 Norberto Meijome wrote:
Anyone?

btw, it is 5.0 , on Centos 5.4 64bit
_________________________
{Beto|Norberto|Numard} Meijome

"The only people that never change are the stupid and the dead"
Jorge Luis Borges.

I speak for myself, not my employer. Contents may be hot. Slippery when wet.
Reading disclaimers makes you go blind. Writing them is worse. You have been
Warned.


On 11 March 2010 13:40, Norberto Meijome <numardbsd < at > gmail.com> wrote:

Hi everyone,
I have a job that runs very frequently, and successive runs take different
sets of data (as determined by a pre-job script). To avoid problems, my
pre-job script sets a lock file and the post-clears it.

When I configure my director for concurrent jobs = 2, i sometimes get 2 of
these jobs trying to run in parallel (which is understandable, many times
a
run has more data than the average and is still running while the next one
is due). My pre-script will then cancel the execution of the 2nd script
(because it finds the lock file). Great.

Of course, this is generating a bunch of (expected) errored jobs . I read
in the documentaiton [1] that I can use "Maximum Concurrent Jobs" in the
Job def, to determine how many of this jobs can start in parallel. So I
set
this to 1 [2]

But it has no effect . I get 2 jobs running in parallel (for a short
while,
until the pre-script kills it). If I change the director's Max Concurrent
Jobs to 1, then it's forced to 1 (but i need more than one, as I as have
spooling enabled and other jobs writing to the tape too..)

What is going on ?

TIA!
B
_________________________
{Beto|Norberto|Numard} Meijome

"The only people that never change are the stupid and the dead"
Jorge Luis Borges.

I speak for myself, not my employer. Contents may be hot. Slippery when
wet.
Reading disclaimers makes you go blind. Writing them is worse. You have
been
Warned.

[1]
http://bacula.org/5.0.x-
manuals/en/main/main/Configuring_Director.html#SECTION001730000000000000000,
way down...

[2]
[...]
Job {
Name = "assetBackups"
JobDefs = "DefaultJob"
Level = Full
FileSet="publishedAssetsSet"
Messages = Standard
Pool = "productionPool"
Storage = "berthaTape"
Schedule = "PublishedAssetsSchedule"
Write Bootstrap = "/var/lib/bacula/%n.bsr"
Priority = 10
Accurate = yes
Spool Data = yes
# We only want one instance of this job running < at > the same time.
# enforced (errors out) in the apb_run.sh
Maximum Concurrent Jobs = 1
# This the list of asset versions to be backed up. The parameter passed
is the number of versions to include in this run
RunBeforeJob = "/etc/bacula/backupFilelist/apb_run.sh 400"
# This moves the current list out of the way and updates the offset
# parameters passed are the jobId and the jobStatus
RunAfterJob = "/etc/bacula/backupFilelist/apb_post.sh %i %e"
}
[...]

------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Post Maximum Concurrent Jobs in job definition 
On Fri, Mar 12, 2010 at 11:21:13AM +1100, Norberto Meijome wrote:
On 11 March 2010 13:40, Norberto Meijome <numardbsd < at > gmail.com> wrote:

Hi everyone,
I have a job that runs very frequently, and successive runs take different
sets of data (as determined by a pre-job script). To avoid problems, my
pre-job script sets a lock file and the post-clears it.

When I configure my director for concurrent jobs = 2, i sometimes get 2 of
these jobs trying to run in parallel (which is understandable, many times a
run has more data than the average and is still running while the next one
is due). My pre-script will then cancel the execution of the 2nd script
(because it finds the lock file). Great.


Why the pre/post locking script ?
Would it not be easier to just use "Allow Duplicate Jobs = no" ?

See
http://www.bacula.org/en/dev-manual/New_Features.html#SECTION003101000000000000000



--
Matija Nalis
Odjel racunalno-informacijskih sustava i servisa
Hrvatska akademska i istrazivacka mreza - CARNet
Josipa Marohnica 5, 10000 Zagreb
tel. +385 1 6661 616, fax. +385 1 6661 766
www.CARNet.hr

------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Post Maximum Concurrent Jobs in job definition 
Hi !

On 3/11/10 3:40 AM, Norberto Meijome wrote:
But it has no effect . I get 2 jobs running in parallel (for a short
while, until the pre-script kills it). If I change the director's Max
Concurrent Jobs to 1, then it's forced to 1 (but i need more than one,
as I as have spooling enabled and other jobs writing to the tape too..)

What is going on ?

Besides the hint about duplicate job control that might or might not
work depending on your configuration i have another one.

I assume you those jobs all run on the same Client. Just specify maximum
concurrent jobs = 1 in the client-resource (bacula-dir.conf) for that
client and it will never run more than one job simultaneously.

Be sure that you don´t do this in the bacula-fd config because you can´t
check the FDs status anymore when a backup job is running (backup +
fd-status check = 2 jobs).

Best regards,

Daniel Holtkamp

--
.............................................................
Riege Software International GmbH Fon: +49 (2159) 9148 0
Mollsfeld 10 Fax: +49 (2159) 9148 11
40670 Meerbusch Web: www.riege.com
Germany E-Mail: holtkamp < at > riege.com
--- ---
Handelsregister: Managing Directors:
Amtsgericht Neuss HRB-NR 4207 Christian Riege
USt-ID-Nr.: DE120585842 Gabriele Riege
Johannes Riege
.............................................................
YOU CARE FOR FREIGHT, WE CARE FOR YOU




------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Post Maximum Concurrent Jobs in job definition 
On Fri, 12 Mar 2010 09:00:10 +0200
Silver Salonen <silver < at > ultrasoft.ee> wrote:

Hi.

Did it work like it should in Bacula 3.0?


Hi Silver, only started using Bacula 5.0.
B

_________________________
{Beto|Norberto|Numard} Meijome

"Unix gives you just enough rope to hang yourself --
and then a couple of more feet, just to be sure."
Eric Allman

I speak for myself, not my employer. Contents may be hot. Slippery when wet. Reading disclaimers makes you go blind. Writing them is worse. You have been Warned.

------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Post Maximum Concurrent Jobs in job definition 
On Fri, 12 Mar 2010 10:13:49 +0100
Matija Nalis <mnalis+bacula < at > CARNet.hr> wrote:

On Fri, Mar 12, 2010 at 11:21:13AM +1100, Norberto Meijome wrote:
On 11 March 2010 13:40, Norberto Meijome <numardbsd < at > gmail.com> wrote:

Hi everyone,
I have a job that runs very frequently, and successive runs take different
sets of data (as determined by a pre-job script). To avoid problems, my
pre-job script sets a lock file and the post-clears it.

When I configure my director for concurrent jobs = 2, i sometimes get 2 of
these jobs trying to run in parallel (which is understandable, many times
a run has more data than the average and is still running while the next
one is due). My pre-script will then cancel the execution of the 2nd
script (because it finds the lock file). Great.


Why the pre/post locking script ?

Because I prefer explicitly checking for the errors I know i need to handle Smile

Would it not be easier to just use "Allow Duplicate Jobs = no" ?

I reckon it would be. I didn't change this value, and http://www.bacula.org/5.0.x-manuals/en/main/main/New_Features_in_3_0_0.html#SECTION007101000000000000000 says it defaults to NO.

Which is why I make sure via my own code things work properly Wink

I will review the whole thing soon and see whether i've missed anything,
thanks!
B


See
http://www.bacula.org/en/dev-manual/New_Features.html#SECTION003101000000000000000



_________________________
{Beto|Norberto|Numard} Meijome

"Egotism is the anesthetic that dulls the pain of stupidity."
Frank Leahy

I speak for myself, not my employer. Contents may be hot. Slippery when wet. Reading disclaimers makes you go blind. Writing them is worse. You have been Warned.

------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Post Maximum Concurrent Jobs in job definition 
On Sunday 21 March 2010 07:49:27 Norberto Meijome wrote:
On Fri, 12 Mar 2010 10:13:49 +0100
Matija Nalis <mnalis+bacula < at > CARNet.hr> wrote:
Would it not be easier to just use "Allow Duplicate Jobs = no" ?

I reckon it would be. I didn't change this value, and
http://www.bacula.org/5.0.x-
manuals/en/main/main/New_Features_in_3_0_0.html#SECTION007101000000000000000
says it defaults to NO.

Which is why I make sure via my own code things work properly Wink

If you didn't change it, doesn't it mean the duplicate jobs are NOT allowed
and your script shouldn't be even ran when there's an instance of the same job
already running in bacula? Is it still ran in some cases?

--
Silver

------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Display posts from previous:
Reply to topic Page 1 of 1
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
  


Magic SEO URL for phpBB