SearchFAQMemberlist Log in
Reply to topic Page 1 of 1
Bacula memory usage
Author Message
Post Bacula memory usage 
Date: Thu, 07 Jun 2007 21:33:57 -0400
From: Ryan Novosielski <novosirj < at > um...>
Subject: Re: [Bacula-users] Hardware recommendations
To: bacula-users < at > li...
Message-ID: <4668B205.10704 < at > um...>
Content-Type: text/plain; charset="iso-8859-1"

Message was signed with unknown key 0xD11CB1BE.
The validity of the signature cannot be verified.
I'm using a Sun UE450 with 3x296MHz and 896MB of RAM. However, the
maximum simultaneous number of backups I'm doing is 8, and probably my
total aggregate amount of data is less than 100GB for all of the systems
together.

There was some information on the list relatively recently about
optimizing MySQL memory usage, or perhaps making it act somewhat
differently to running out of memory.

AFAIK, as well, newer versions of bacula are slated to handle this
problem better as well. I personally would be somewhat surprised if 2G
of RAM on a dedicated machine were too little. I've never tried watching
my machine while backups were running, but it seems to me Bacula is
fairly lightweight most of the time.

Yes, Bacula is rather light weight in memory usage, and in addition, does
pretty much minimal copying of data within a daemon. E.g. in the FD, then
data is read into a buffer then transmitted to the SD in the same buffer. In
the SD, the data comes into a buffer then is copied into a block which is
written directly to the tape (unless spooling is enabled).

Typical memory usage for the Director is about 100-120K per Job for a backup
regardless of the number of files or size of the data involved. However,
there was a bug in the MySQL driver that could cause Bacula to lose memory
after running a lot of jobs. Even with the memory leak, I had no problem
running 80 simultaneous jobs. The MySQL driver memory leak is fixed in the
current BETA version.

MySQL/PostgreSQL server memory usage is another story typically controlled by
their respective conf/ini files.

Joseph Wright wrote:
I'd like to find out what kind of hardware configurations people are
using for their director. I plan to be backing up 50 or more clients
with ~250G of data each on a dedicated director with 2G RAM and a
120G SATA drive. I had an issue one night where the director shut
down with an out of memory error after the backup jobs started, with
only 20 clients. I've since backed off the number of clients and am
adding them back in more slowly, as well as distributing the startup
times so they don't all start at once. But I'd like to find out if
people think my hardware is adequate for the task and it would be
interesting to know what others are using.

Post Bacula memory usage 
It seems that the resources used by the database should be taken into
account when looking at Bacula's memory usage, because even though
the two are separate, the director is the one putting the records
there. Maybe I should re-ask the question as: How heavily does
Bacula tax the database with 50 or more clients being backed up
concurrently, and what kind of hardware is recommended for such a
situation?

(\_/)
(o,o)
()_() Joseph Wright
" "

On Jun 8, 2007, at 12:35 AM, Kern Sibbald wrote:

Date: Thu, 07 Jun 2007 21:33:57 -0400
From: Ryan Novosielski <novosirj < at > um...>
Subject: Re: [Bacula-users] Hardware recommendations
To: bacula-users < at > li...
Message-ID: <4668B205.10704 < at > um...>
Content-Type: text/plain; charset="iso-8859-1"

Message was signed with unknown key 0xD11CB1BE.
The validity of the signature cannot be verified.
I'm using a Sun UE450 with 3x296MHz and 896MB of RAM. However, the
maximum simultaneous number of backups I'm doing is 8, and
probably my
total aggregate amount of data is less than 100GB for all of the
systems
together.

There was some information on the list relatively recently about
optimizing MySQL memory usage, or perhaps making it act somewhat
differently to running out of memory.

AFAIK, as well, newer versions of bacula are slated to handle this
problem better as well. I personally would be somewhat surprised
if 2G
of RAM on a dedicated machine were too little. I've never tried
watching
my machine while backups were running, but it seems to me Bacula is
fairly lightweight most of the time.

Yes, Bacula is rather light weight in memory usage, and in
addition, does
pretty much minimal copying of data within a daemon. E.g. in the
FD, then
data is read into a buffer then transmitted to the SD in the same
buffer. In
the SD, the data comes into a buffer then is copied into a block
which is
written directly to the tape (unless spooling is enabled).

Typical memory usage for the Director is about 100-120K per Job for
a backup
regardless of the number of files or size of the data involved.
However,
there was a bug in the MySQL driver that could cause Bacula to lose
memory
after running a lot of jobs. Even with the memory leak, I had no
problem
running 80 simultaneous jobs. The MySQL driver memory leak is fixed
in the
current BETA version.

MySQL/PostgreSQL server memory usage is another story typically
controlled by
their respective conf/ini files.


Joseph Wright wrote:
I'd like to find out what kind of hardware configurations people are
using for their director. I plan to be backing up 50 or more
clients
with ~250G of data each on a dedicated director with 2G RAM and a
120G SATA drive. I had an issue one night where the director shut
down with an out of memory error after the backup jobs started, with
only 20 clients. I've since backed off the number of clients and am
adding them back in more slowly, as well as distributing the startup
times so they don't all start at once. But I'd like to find out if
people think my hardware is adequate for the task and it would be
interesting to know what others are using.

----------------------------------------------------------------------
---
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
Bacula-users mailing list
Bacula-users < at > li...
https://lists.sourceforge.net/lists/listinfo/bacula-users

Post Bacula memory usage 
In response to Joseph Wright <joseph < at > ma...>:

It seems that the resources used by the database should be taken into
account when looking at Bacula's memory usage, because even though
the two are separate, the director is the one putting the records
there. Maybe I should re-ask the question as: How heavily does
Bacula tax the database with 50 or more clients being backed up
concurrently, and what kind of hardware is recommended for such a
situation?

If you've got 50 jobs running simultaneously, you're liable to need a
massive DB server. Unfortunately, however, that's not enough information
to be sure.

The real load on the DB server is (mostly) determined by two factors.
1) The number of files being backed up.
2) The amount of duplication in pathnames and filenames.

If each backup job is 1000s of tiny files, you're going to be working
the database pretty hard, as each of those files will require that a
record be written to the database.

If filenames and pathnames change often, that will require that additional
database records be written for each file saved. For a file that has a
name never used before in a directory never backed up before, you have
3 database records created.

On the flip side, if you're filesets consist largely of the same files that
are constantly being edited, and your average filesize is large, you'll find
that the speed of your storage media will slow you down and the database
will have no trouble keeping up.

I suspect that's part of the reason that there are no published hardware
requirements for Bacula. The hardware requirements aren't really for Bacula,
their for your specific workload.

Regardless of all that, if you really mean to run 50 jobs in parallel
(i.e. at the same time, not just 50 jobs in series) I would recommend that
you get a dedicated DB server with about 12 disks arranged in a RAID 10 and
a battery-backed cache. If you can run the jobs in series, then just about
any reasonably powered server hardware should suffice.

--
Bill Moran
http://www.potentialtech.com

Display posts from previous:
Reply to topic Page 1 of 1
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
  


Magic SEO URL for phpBB