Backup Central

Hi,

I'm considering the following ideas for the next couple of Bacula releases,
and would like your ideas.

1. Release version 2.0.4 in the next week or two with a couple more bug
fixes.

2. Release either version 2.0.5 or more likely version 2.2.0 within 4-8
weeks, targeting 6 weeks.

ok

Version 2.2.0, would be essentially the current Bacula trunk, which has the
following major changes from the 2.0.x stream:

1. The new SQL attribute insertion code, which runs significantly faster on
PostgreSQL, and for multiple simultaneous jobs runs with much less database
locking.

2. The red/black in memory restore code, which in my tests on restoring
800K+ files, ran over 500 times as fast as the current code.

3. Some important new communications protocols that support bat.

4. A first cut of bat that "roughly" implements the same features as the
gnome-console. This is not much, but it does give a base for further
incremental improvements, and will permit easier user participation in the
development.

5. The additional features that Eric has already added (Scratch pool, ...).

6. Possibly a few additional features depending on what the developers
respond to this email.

We have also
Item: 40 Include JobID in spool file name
Item: 25 Implement huge exclude list support using dlist

And i will have time to add File relocation feature (regex where) in a couple
of days.

I think it's a good road map, it will not be a too big step.

Bye

On Sunday 18 March 2007 17:28, Eric Bollengier wrote:
Hi,

I'm considering the following ideas for the next couple of Bacula
releases,
and would like your ideas.

1. Release version 2.0.4 in the next week or two with a couple more bug
fixes.

2. Release either version 2.0.5 or more likely version 2.2.0 within 4-8
weeks, targeting 6 weeks.

ok

Version 2.2.0, would be essentially the current Bacula trunk, which has
the
following major changes from the 2.0.x stream:

1. The new SQL attribute insertion code, which runs significantly faster
on
PostgreSQL, and for multiple simultaneous jobs runs with much less
database
locking.

2. The red/black in memory restore code, which in my tests on restoring
800K+ files, ran over 500 times as fast as the current code.

3. Some important new communications protocols that support bat.

4. A first cut of bat that "roughly" implements the same features as the
gnome-console. This is not much, but it does give a base for further
incremental improvements, and will permit easier user participation in the
development.

5. The additional features that Eric has already added (Scratch
pool, ...).

6. Possibly a few additional features depending on what the developers
respond to this email.

We have also
Item: 40 Include JobID in spool file name
Item: 25 Implement huge exclude list support using dlist

Yes, and there are a number of other changes that I have made as well that I
probably should have listed, but I was a bit lazy. :-)


And i will have time to add File relocation feature (regex where) in a
couple
of days.

I think it's a good road map, it will not be a too big step.

Bye

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Bacula-devel mailing list
Bacula-devel < at > li...
https://lists.sourceforge.net/lists/listinfo/bacula-devel

Hi Kern and lists,

I discover problem with RunScript directive in version 2.0.2 and now in
2.0.3 too.
I have next directive in Job resource:
RunScript {
RunsWhen = After
RunsOnSuccess = yes
Command = "/etc/bacula/scripts/disk_check.sh %j"
}

and the script disk_check.sh is following:
size=`/usr/bin/mysql -e "select JobBytes from Job where Job='$job'" -u
bacula -D bacula -pxxx | tail -n 1`
echo $size >>/etc/bacula/working/watchdog.log

After backup job, $size return "0".

In version: 1.38.11 this script return the right number, but after
upgrading to Bacula 2.0.2 or 2.0.3 I have this problem.
PLS, do you have some idea to improve it?

Thanx

Ondra

Kern Sibbald napsal(a):
Hello,

I'm considering the following ideas for the next couple of Bacula releases,
and would like your ideas.

1. Release version 2.0.4 in the next week or two with a couple more bug fixes.

2. Release either version 2.0.5 or more likely version 2.2.0 within 4-8 weeks,
targeting 6 weeks.

Version 2.2.0, would be essentially the current Bacula trunk, which has the
following major changes from the 2.0.x stream:

1. The new SQL attribute insertion code, which runs significantly faster on
PostgreSQL, and for multiple simultaneous jobs runs with much less database
locking.

2. The red/black in memory restore code, which in my tests on restoring 800K+
files, ran over 500 times as fast as the current code.

3. Some important new communications protocols that support bat.

4. A first cut of bat that "roughly" implements the same features as the
gnome-console. This is not much, but it does give a base for further
incremental improvements, and will permit easier user participation in the
development.

5. The additional features that Eric has already added (Scratch pool, ...).

6. Possibly a few additional features depending on what the developers respond
to this email.

On 18 Mar 2007 at 16:27, Kern Sibbald wrote:

Hello,

I'm considering the following ideas for the next couple of Bacula releases,
and would like your ideas.

1. Release version 2.0.4 in the next week or two with a couple more bug fixes.

2. Release either version 2.0.5 or more likely version 2.2.0 within 4-8 weeks,
targeting 6 weeks.

I'm a fan of frequent releases. Smaller changes each time is a good
thing. Both from a user and a sysadmin point of view.

From a packager, a new release takes me about 30 minutes, most of
that is testing that it still builds. Other packagers may have
different workloads.

Version 2.2.0, would be essentially the current Bacula trunk, which has the
following major changes from the 2.0.x stream:

1. The new SQL attribute insertion code, which runs significantly faster on
PostgreSQL, and for multiple simultaneous jobs runs with much less database
locking.

I will certainly be looking forward to that.

2. The red/black in memory restore code, which in my tests on restoring 800K+
files, ran over 500 times as fast as the current code.

WOO HOO!

3. Some important new communications protocols that support bat.

4. A first cut of bat that "roughly" implements the same features as the
gnome-console. This is not much, but it does give a base for further
incremental improvements, and will permit easier user participation in the
development.

\o/

--
Dan Langille : Software Developer looking for work
my resume: http://www.freebsddiary.org/dan_langille.php
PGCon - The PostgreSQL Conference - http://www.pgcon.org/

Hello,

Please don't hijack an email message. Your subject is titled "Bacula release
strategy", but the contents has nothing to do with that subject.

Concerning your question: I cannot answer your question because you have
supplied insufficient information, because the text you show for
disk_check.sh is incorrect.

I'm sorry, but I cannot respond any more to reqests to help debug scripts. If
you can tie the problem down to a bug or a change in 2.0.2, I will be very
interested.

Regards,

Kern

On Sunday 18 March 2007 18:19, Ondrej PLANKA wrote:
Hi Kern and lists,

I discover problem with RunScript directive in version 2.0.2 and now in
2.0.3 too.
I have next directive in Job resource:
RunScript {
RunsWhen = After
RunsOnSuccess = yes
Command = "/etc/bacula/scripts/disk_check.sh %j"
}

and the script disk_check.sh is following:
size=`/usr/bin/mysql -e "select JobBytes from Job where Job='$job'" -u
bacula -D bacula -pxxx | tail -n 1`
echo $size >>/etc/bacula/working/watchdog.log

After backup job, $size return "0".

In version: 1.38.11 this script return the right number, but after
upgrading to Bacula 2.0.2 or 2.0.3 I have this problem.
PLS, do you have some idea to improve it?

Thanx


Ondra


Kern Sibbald napsal(a):
Hello,

I'm considering the following ideas for the next couple of Bacula
releases,
and would like your ideas.

1. Release version 2.0.4 in the next week or two with a couple more bug
fixes.

2. Release either version 2.0.5 or more likely version 2.2.0 within 4-8
weeks,
targeting 6 weeks.

Version 2.2.0, would be essentially the current Bacula trunk, which has
the
following major changes from the 2.0.x stream:

1. The new SQL attribute insertion code, which runs significantly faster
on
PostgreSQL, and for multiple simultaneous jobs runs with much less
database
locking.

2. The red/black in memory restore code, which in my tests on restoring
800K+
files, ran over 500 times as fast as the current code.

3. Some important new communications protocols that support bat.

4. A first cut of bat that "roughly" implements the same features as the
gnome-console. This is not much, but it does give a base for further
incremental improvements, and will permit easier user participation in the
development.

5. The additional features that Eric has already added (Scratch
pool, ...).

6. Possibly a few additional features depending on what the developers
respond
to this email.



-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Bacula-devel mailing list
Bacula-devel < at > li...
https://lists.sourceforge.net/lists/listinfo/bacula-devel

On Sunday 18 March 2007 18:22, Dan Langille wrote:
On 18 Mar 2007 at 16:27, Kern Sibbald wrote:

Hello,

I'm considering the following ideas for the next couple of Bacula
releases,
and would like your ideas.

1. Release version 2.0.4 in the next week or two with a couple more bug
fixes.

2. Release either version 2.0.5 or more likely version 2.2.0 within 4-8
weeks,
targeting 6 weeks.

I'm a fan of frequent releases. Smaller changes each time is a good
thing. Both from a user and a sysadmin point of view.

From a packager, a new release takes me about 30 minutes, most of
From a packager, a new release takes me about 30 minutes, most of
that is testing that it still builds. Other packagers may have
different workloads.

Version 2.2.0, would be essentially the current Bacula trunk, which has
the
following major changes from the 2.0.x stream:

1. The new SQL attribute insertion code, which runs significantly faster
on
PostgreSQL, and for multiple simultaneous jobs runs with much less
database
locking.

I will certainly be looking forward to that.

Yes, me too. However, at the moment, it doesn't pass the regression scripts,
so there is a bunch more work for the responsible developers ...


2. The red/black in memory restore code, which in my tests on restoring
800K+
files, ran over 500 times as fast as the current code.

WOO HOO!

Yes, that one surprised me :-)


3. Some important new communications protocols that support bat.

4. A first cut of bat that "roughly" implements the same features as the
gnome-console. This is not much, but it does give a base for further
incremental improvements, and will permit easier user participation in the
development.

\o/


--
Dan Langille : Software Developer looking for work
my resume: http://www.freebsddiary.org/dan_langille.php
PGCon - The PostgreSQL Conference - http://www.pgcon.org/

I'm a fan of frequent releases. Smaller changes each time is a good
thing. Both from a user and a sysadmin point of view.

In principle this is a great idea. Can I suggest a minor variant?=20

Frequent releases benefit sites that are capable of experimentation.
Enterprise deployments don't move so quickly, so I'd suggest you
designate two tracks: a "development" track (release early and often)
and a "stable" track that's a little more conservative (release on a
regular, planned schedule, order of once or twice a year, unless there's
something major).=20
This would help the folks offering support a great deal.

Tens of thousands of client machines would thank you..

On Sunday 18 March 2007 20:27, David Boyes wrote:
I'm a fan of frequent releases. Smaller changes each time is a good
thing. Both from a user and a sysadmin point of view.

In principle this is a great idea. Can I suggest a minor variant?

Frequent releases benefit sites that are capable of experimentation.
Enterprise deployments don't move so quickly, so I'd suggest you
designate two tracks: a "development" track (release early and often)
and a "stable" track that's a little more conservative (release on a
regular, planned schedule, order of once or twice a year, unless there's
something major).
This would help the folks offering support a great deal.

Tens of thousands of client machines would thank you.. 8-)

To a certain extent this alway occurs during a development cycle. I'm just
saying that in this particular case, there are important things already in
the development cycle, so I am not proposing something permanent.

If I had 10 or 20 programmers working on Bacula, this *might* be possible, but
it is not.

After receiving somewhat conflicting input (some pro quick releases, some
against), I've decided roughly to follow the following strategy:

For the immediate future:
1. Observe the progress of our current development work over the next week or
two.
2. As far as I can tell, all the code is stable *except* the new batch insert
database code, which is not yet of production quality.
3. After a week, if it is production quality, then we will release an update
to 2.0.x that contains the whole of the current development code.
4. If after a week, the database code is not production quality, then I will
merge all the development code except the batch insert database code into
2.0.4 and release it in the near future (3 - 6 weeks).

Longer term:
While developing the 2.0.x code, we made some 12 releases to version 1.38.x.
The first few, as with 2.0.x, were primarily bug fix releases, but a good
part of the subsequent releases were new features, but those which I found to
be stable and compatible. I would like to continue this sort of release
procedure. It allows us to make a new incremental release every two to four
months each time adding new features. Please keep in mind that each of the
2.0.x releases as was the case with 1.38.x will remain totally compatible.
There will be no need as some feared to upgrade lots or clients unless you
want new features that have been added to the clients. Normally new client
features don't come along too often, but we will probably have one shortly
for 2.0.x for Win32, because from what I understand the current client does
not run on Vista (thanks Microsoft), so we will very likely release an update
(hopefully sooner rather than later).

Obviously the above means that the 2.0.x feature updates will be limited to
those that are stable and cause no compatibility problems. Major features
updates will continue to be spaced at approximately 12-18 month intervals.

Basically this means maintaining the status quo. If someone has some problems
with this, now is the time to speak up.

Best regards,

Kern

Mandag 19 marts 2007 19:34 skrev Kern Sibbald:
After receiving somewhat conflicting input (some pro quick releases, some
against), I've decided roughly to follow the following strategy:

For the immediate future:
1. Observe the progress of our current development work over the next week
or two.
2. As far as I can tell, all the code is stable *except* the new batch
insert database code, which is not yet of production quality.
3. After a week, if it is production quality, then we will release an
update to 2.0.x that contains the whole of the current development code.
4. If after a week, the database code is not production quality, then I
will merge all the development code except the batch insert database code
into 2.0.4 and release it in the near future (3 - 6 weeks).
It is not clear to me if this then includes the features that you mentioned
for version 2.2 in the original post. Seems to me that the marked performance
improvements are so significant that the 2.2 version numbering was reasonable



Longer term:
While developing the 2.0.x code, we made some 12 releases to version
1.38.x. The first few, as with 2.0.x, were primarily bug fix releases, but
a good part of the subsequent releases were new features, but those which I
found to be stable and compatible. I would like to continue this sort of
release procedure. It allows us to make a new incremental release every
two to four months each time adding new features. Please keep in mind that
each of the 2.0.x releases as was the case with 1.38.x will remain totally
compatible. There will be no need as some feared to upgrade lots or clients
unless you want new features that have been added to the clients. Normally
new client features don't come along too often, but we will probably have
one shortly for 2.0.x for Win32, because from what I understand the current
client does not run on Vista (thanks Microsoft), so we will very likely
release an update (hopefully sooner rather than later).

Obviously the above means that the 2.0.x feature updates will be limited to
those that are stable and cause no compatibility problems. Major features
updates will continue to be spaced at approximately 12-18 month intervals.

Basically this means maintaining the status quo. If someone has some
problems with this, now is the time to speak up.
Depends on the answer above, as those features are some that I personally just
can't wait to see

Otherwise it seems as a very good way to keep a stable release for a
reasonable length of time. I am still running 1.38 because it does what I
need it to do, but just those performance improvements give me a good reason
to upgrade, and it is nice if it will then last a while.

Best regards,

Kern

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share
your opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Bacula-users mailing list
Bacula-users < at > li...
https://lists.sourceforge.net/lists/listinfo/bacula-users

--
Regards

Steen

On Tuesday 20 March 2007 00:49, Steen wrote:
Mandag 19 marts 2007 19:34 skrev Kern Sibbald:
After receiving somewhat conflicting input (some pro quick releases, some
against), I've decided roughly to follow the following strategy:

For the immediate future:
1. Observe the progress of our current development work over the next week
or two.
2. As far as I can tell, all the code is stable *except* the new batch
insert database code, which is not yet of production quality.
3. After a week, if it is production quality, then we will release an
update to 2.0.x that contains the whole of the current development code.
4. If after a week, the database code is not production quality, then I
will merge all the development code except the batch insert database code
into 2.0.4 and release it in the near future (3 - 6 weeks).

It is not clear to me if this then includes the features that you mentioned
for version 2.2 in the original post. Seems to me that the marked
performance
improvements are so significant that the 2.2 version numbering was
reasonable

Sorry if that was not clear. I am talking about the same subject. This was
a "conclusion" of the prior email.




Longer term:
While developing the 2.0.x code, we made some 12 releases to version
1.38.x. The first few, as with 2.0.x, were primarily bug fix releases, but
a good part of the subsequent releases were new features, but those which
I
found to be stable and compatible. I would like to continue this sort of
release procedure. It allows us to make a new incremental release every
two to four months each time adding new features. Please keep in mind
that
each of the 2.0.x releases as was the case with 1.38.x will remain totally
compatible. There will be no need as some feared to upgrade lots or
clients
unless you want new features that have been added to the clients.
Normally
new client features don't come along too often, but we will probably have
one shortly for 2.0.x for Win32, because from what I understand the
current
client does not run on Vista (thanks Microsoft), so we will very likely
release an update (hopefully sooner rather than later).

Obviously the above means that the 2.0.x feature updates will be limited
to
those that are stable and cause no compatibility problems. Major features
updates will continue to be spaced at approximately 12-18 month intervals.

Basically this means maintaining the status quo. If someone has some
problems with this, now is the time to speak up.
Depends on the answer above, as those features are some that I personally
just
can't wait to see

Otherwise it seems as a very good way to keep a stable release for a
reasonable length of time. I am still running 1.38 because it does what I
need it to do, but just those performance improvements give me a good reason
to upgrade, and it is nice if it will then last a while.

Yes, I don't see any reason to wait for users to benefit from the speed up
offered by the red/black restore tree code. For me, this is the main
improvement aside from the database batch insert code.