Welcome! » Log In » Create A New Profile

Struggling with full disk

Posted by cdn 
cdn
Struggling with full disk
July 06, 2016 07:55AM
Hey,
I am pretty new in bacula, but started a solution to backup my server via internet through a vpn tunnel. Everything works well, but I am struggling with my disk space:

I am using a RAID 5 for saving my backups. In my bacula-dir.conf the following pool is definied:

Pool {
Name = File
Pool Type = Backup
Recycle = yes # Bacula can automatically recycle Volumes
AutoPrune = yes # Prune expired volumes
Volume Retention = 30 days # one year
Maximum Volume Bytes = 4000G # Limit Volume size to something reasonable
Maximum Volumes = 100 # Limit number of Volumes in Pool
}

But after a while the disk running out of space. Why bacula will not delete old files?

Client conf:

Client {
Name = srv-fd
Address =
FDPort =
Catalog = MyCatalog
Password = "" # password for FileDaemon 2
File Retention = 30 days # 30 days
Job Retention = 2 months # six months
AutoPrune = yes # Prune expired Jobs/Files
}

Thanks for your help!
Struggling with full disk
July 06, 2016 01:35PM
Hello

I don't know how big is your storage, but you have configured something like 100 disks of 4TB (= 4000gb)

You define capacity of your pool configuring (Maximum Volume Bytes x Maximum Volumes)

Is interesting to set smaller volumes because if you have some volume error you will loose less data.

If you consider one hard disk of 1TB with a Full Backup daily of 100GB and you want 30 days retention, you will not reach it, because 30 x 100GB = 3.000gb or 3TB, so you only will reach 10 days retention (10 * 100GB) = 1TB

Best Regards

[b]Wanderlei Hüttel[/b]

[url=http://www.huttel.com.br]http://www.huttel.com.br[/url]

2016-07-06 11:55 GMT-03:00 cdn <bacula-forum < at > backupcentral.com ([email]bacula-forum < at > backupcentral.com[/email])>:
[quote]Hey,
I am pretty new in bacula, but started a solution to backup my server via internet through a vpn tunnel. Everything works well, but I am struggling with my disk space:

I am using a RAID 5 for saving my backups. In my bacula-dir.conf the following pool is definied:

Pool {
  Name = File
  Pool Type = Backup
  Recycle = yes                       # Bacula can automatically recycle Volumes
  AutoPrune = yes                     # Prune expired volumes
  Volume Retention = 30 days         # one year
  Maximum Volume Bytes = 4000G          # Limit Volume size to something reasonable
  Maximum Volumes = 100               # Limit number of Volumes in Pool
}

But after a while the disk running out of space. Why bacula will not delete old files?

Client conf:

Client {
  Name = srv-fd
  Address =
  FDPort =
  Catalog = MyCatalog
  Password = ""         # password for FileDaemon 2
  File Retention = 30 days            # 30 days
  Job Retention = 2 months            # six months
  AutoPrune = yes                     # Prune expired Jobs/Files
}

Thanks for your help!

+----------------------------------------------------------------------
|This was sent by esser < at > s-3-it.com ([email]esser < at > s-3-it.com[/email]) via Backup Central.
|Forward SPAM to abuse < at > backupcentral.com ([email]abuse < at > backupcentral.com[/email]).
+----------------------------------------------------------------------

------------------------------------------------------------------------------
Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
[url=http://sdm.link/attshape]http://sdm.link/attshape[/url]
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net ([email]Bacula-users < at > lists.sourceforge.net[/email])
[url=https://lists.sourceforge.net/lists/listinfo/bacula-users]https://lists.sourceforge.net/lists/listinfo/bacula-users[/url]
[/quote]
Struggling with full disk
July 06, 2016 02:02PM
On 06/07/16 10:55 AM, cdn wrote:
[quote]Hey,
I am pretty new in bacula, but started a solution to backup my server via internet through a vpn tunnel. Everything works well, but I am struggling with my disk space:

I am using a RAID 5 for saving my backups. In my bacula-dir.conf the following pool is definied:

Pool {
Name = File
Pool Type = Backup
Recycle = yes # Bacula can automatically recycle Volumes
AutoPrune = yes # Prune expired volumes
Volume Retention = 30 days # one year
Maximum Volume Bytes = 4000G # Limit Volume size to something reasonable
Maximum Volumes = 100 # Limit number of Volumes in Pool
}

But after a while the disk running out of space. Why bacula will not delete old files?

Client conf:

Client {
Name = srv-fd
Address =
FDPort =
Catalog = MyCatalog
Password = "" # password for FileDaemon 2
File Retention = 30 days # 30 days
Job Retention = 2 months # six months
AutoPrune = yes # Prune expired Jobs/Files
}

Thanks for your help!

[/quote]
I note that you are allocating 4T as the maximum volume size and
allowing up to 100 volumes. Do you have 400T of space on your backup
device? If not, reduce the sizes to something reasonable.

------------------------------------------------------------------------------
Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Struggling with full disk
July 06, 2016 02:03PM
On 06/07/16 04:29 PM, Wanderlei Huttel wrote:
[quote] Hello

I don't know how big is your storage, but you have configured something like 100 disks of 4TB (= 4000gb)

You define capacity of your pool configuring (Maximum Volume Bytes x Maximum Volumes)

Is interesting to set smaller volumes because if you have some volume error you will loose less data.

If you consider one hard disk of 1TB with a Full Backup daily of 100GB and you want 30 days retention, you will not reach it, because 30 x 100GB = 3.000gb or 3TB, so you only will reach 10 days retention (10 * 100GB) = 1TB

[/quote] That's if each backup is a full backup. Generally you don't do one that often. A full weekly backup with daily incremental backups would require far less storage.

You're right about keeping volumes small to reduce the loss due to a corrupted volume. When you're dealing with terabytes of data, it's hard to guarantee perfect reliability.
cdn
Struggling with full disk
July 07, 2016 06:01AM
Wow thanks for fast reply.

Oh a mistake. So I have 6TB in total and would like to do the following backupplan:

Schedule {
Name = "WeeklyCycle"
Run = Full 1st sun at 23:05
Run = Differential 2nd-5th sun at 23:05
Run = Incremental mon-sat at 23:05
}
Schedule {
Name = "srv10Filebase"
Run = Full 1st sun at 23:05
Run = Incremental mon-sat at 23:05
}

srv10 has large disks (1,5T) with constantly changing data, so I have created a seperately Schedule. srv10 will do a full backup once a month and incremental during the month.
All other machines will do a full backup once a month, differential on each sunday and incremental every day.

I changed volumes now to 4 and Volume Bytes to 1000GB.
So should bacula delete old backups now?
Struggling with full disk
July 07, 2016 08:39AM
On 7/6/2016 10:55 AM, cdn wrote:
[quote]Hey,
I am pretty new in bacula, but started a solution to backup my server via internet through a vpn tunnel. Everything works well, but I am struggling with my disk space:

I am using a RAID 5 for saving my backups. In my bacula-dir.conf the following pool is definied:

Pool {
Name = File
Pool Type = Backup
Recycle = yes # Bacula can automatically recycle Volumes
AutoPrune = yes # Prune expired volumes
Volume Retention = 30 days # one year
Maximum Volume Bytes = 4000G # Limit Volume size to something reasonable
Maximum Volumes = 100 # Limit number of Volumes in Pool
}

But after a while the disk running out of space. Why bacula will not delete old files?
[/quote]
Bacula never automatically deletes a volume file. When a volume is
purged, or when the last job is pruned, its status changes to Purged,
but that only happens in the catalog database. The volume file itself
will still be the same size after it is purged. It will remain that size
until it is recycled and used by a job. When a job writes to a recycled
volume it will be erased and rewritten. This is on purpose, as Bacula
keeps the data intact as long as possible. To reclaim the space of
purged volumes BEFORE they are recycled and used by a job, use the
Truncate command from within bconsole. It is possible, for example, to
have a daily Admin job that issues the Truncate command to reclaim the
space being used by purged volumes.

[quote]
Client conf:

Client {
Name = srv-fd
Address =
FDPort =
Catalog = MyCatalog
Password = "" # password for FileDaemon 2
File Retention = 30 days # 30 days
Job Retention = 2 months # six months
AutoPrune = yes # Prune expired Jobs/Files
}

Thanks for your help!

+----------------------------------------------------------------------
|This was sent by esser < at > s-3-it.com via Backup Central.
|Forward SPAM to abuse < at > backupcentral.com.
+----------------------------------------------------------------------

------------------------------------------------------------------------------
Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
[/quote]

------------------------------------------------------------------------------
Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
cdn
Struggling with full disk
July 08, 2016 08:49AM
Alright thanks. I already set recycle to yes. So my config now should work, that the disk will not running out of space?
Struggling with full disk
July 08, 2016 09:10AM
Hello

If your pool already contain volumes is necessary to run the update command in bconsole:

*update
2: Pool from resource

*update
1: Volume parameters

14: All Volumes from all Pools

Best Regards

[b]Wanderlei Hüttel[/b]

[url=http://www.huttel.com.br]http://www.huttel.com.br[/url]

2016-07-08 12:49 GMT-03:00 cdn <bacula-forum < at > backupcentral.com ([email]bacula-forum < at > backupcentral.com[/email])>:
[quote]Alright thanks. I already set recycle to yes. So my config now should work, that the disk will not running out of space?

+----------------------------------------------------------------------
|This was sent by esser < at > s-3-it.com ([email]esser < at > s-3-it.com[/email]) via Backup Central.
|Forward SPAM to abuse < at > backupcentral.com ([email]abuse < at > backupcentral.com[/email]).
+----------------------------------------------------------------------

------------------------------------------------------------------------------
Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
[url=http://sdm.link/attshape]http://sdm.link/attshape[/url]
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net ([email]Bacula-users < at > lists.sourceforge.net[/email])
[url=https://lists.sourceforge.net/lists/listinfo/bacula-users]https://lists.sourceforge.net/lists/listinfo/bacula-users[/url]

[/quote]
cdn
Struggling with full disk
August 15, 2016 04:00AM
Hmm the volume is getting full again....

08-Aug 02:41 dom0-0-dir JobId 1189: Start Backup JobId 1189, Job=Backup-srv10.2016-08-07_23.05.00_09
08-Aug 02:41 dom0-0-dir JobId 1189: Using Device "BaculaFileStorage"
08-Aug 02:41 dom0-0-sd JobId 1189: Volume "160402_File" previously written, moving to end of data.
08-Aug 02:41 dom0-0-sd JobId 1189: Ready to append to end of Volume "160402_File" size=3270025779245 09-Aug 03:35 dom0-0-sd JobId 1189: User defined maximum volume capacity 3,758,096,384,000 exceeded on device "BaculaFileStorage" (/mnt/RAID5/bacula/).
09-Aug 03:35 dom0-0-sd JobId 1189: End of medium on Volume "160402_File" Bytes=3,758,096,338,199 Blocks=58,254,301 at 09-Aug-2016 03:35.
09-Aug 03:37 dom0-0-sd JobId 1189: Job Backup-srv10.2016-08-07_23.05.00_09 is waiting. Cannot find any appendable volumes.
Please use the "label" command to create a new Volume for:

09-Aug 18:37 dom0-0-sd JobId 1189: Job Backup-srv10.2016-08-07_23.05.00_09 is waiting. Cannot find any appendable volumes.
Please use the "label" command to create a new Volume for:
Storage: "BaculaFileStorage" (/mnt/RAID5/bacula/)
Pool: File
Media type: File

I don't really know whats wrong ...I've done the steps you told me.
Struggling with full disk
August 15, 2016 07:06AM
Bacula can create &#39;Maximum Volumes&#39; files, each will grow to &#39;Maximum Volume Size&#39; in size before it&#39;ll try to delete (or reuse) anything.

Ultimately, the formula you need is:

Maximum Volumes = total available disk space / Maximum Volume Size

By the sounds of it, you&#39;ve asked for more volumes than you can store on your disk. Either add some disk, or configure for less &#39;Maximum Volumes&#39;.

...Ralph

On Mon, Aug 15, 2016 at 12:00 PM, cdn <bacula-forum < at > backupcentral.com ([email]bacula-forum < at > backupcentral.com[/email])> wrote:
[quote]Hmm the volume is getting full again....

08-Aug 02:41 dom0-0-dir JobId 1189: Start Backup JobId 1189, Job=Backup-srv10.2016-08-07_23.05.00_09
08-Aug 02:41 dom0-0-dir JobId 1189: Using Device "BaculaFileStorage"
08-Aug 02:41 dom0-0-sd JobId 1189: Volume "160402_File" previously written, moving to end of data.
08-Aug 02:41 dom0-0-sd JobId 1189: Ready to append to end of Volume "160402_File" size=3270025779245 09-Aug 03:35 dom0-0-sd JobId 1189: User defined maximum volume capacity 3,758,096,384,000 exceeded on device "BaculaFileStorage" (/mnt/RAID5/bacula/).
09-Aug 03:35 dom0-0-sd JobId 1189: End of medium on Volume "160402_File" Bytes=3,758,096,338,199 Blocks=58,254,301 at 09-Aug-2016 03:35.
09-Aug 03:37 dom0-0-sd JobId 1189: Job Backup-srv10.2016-08-07_23.05.00_09 is waiting. Cannot find any appendable volumes.
Please use the "label" command to create a new Volume for:

09-Aug 18:37 dom0-0-sd JobId 1189: Job Backup-srv10.2016-08-07_23.05.00_09 is waiting. Cannot find any appendable volumes.
Please use the "label" command to create a new Volume for:
    Storage:      "BaculaFileStorage" (/mnt/RAID5/bacula/)
    Pool:         File
    Media type:   File

I don&#39;t really know whats wrong ...I&#39;ve done the steps you told me.

+----------------------------------------------------------------------
|This was sent by esser < at > s-3-it.com ([email]esser < at > s-3-it.com[/email]) via Backup Central.
|Forward SPAM to abuse < at > backupcentral.com ([email]abuse < at > backupcentral.com[/email]).
+----------------------------------------------------------------------

------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. [url=http://sdm.link/zohodev2dev]http://sdm.link/zohodev2dev[/url]
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net ([email]Bacula-users < at > lists.sourceforge.net[/email])
[url=https://lists.sourceforge.net/lists/listinfo/bacula-users]https://lists.sourceforge.net/lists/listinfo/bacula-users[/url]
[/quote]

--

[b]Ralph Bolton[/b]
Systems Administrator

[b]Calltracks Ltd[/b]

Email:   ralph.bolton < at > calltracks.com ([email]ralph.bolton < at > calltracks.com[/email])
Web:    [url=http://www.calltracks.com/]www.calltracks.com[/url]
Tel:      [url=tel]+44 20 3199 9000[/url]
Fax:     [url=tel]+44 20 3199 9009[/url] High availability call monitoring, tracking and NTS services. The opinions expressed are those of the individual and not the company. Internet communications are not secure and therefore Calltracks Ltd ("the company") does not accept liability for any claims arising as a result of the use of this medium for transmissions by or to the company. This email and any files transmitted with it are confidential. If you are not the intended recipient, you are hereby notified that any disclosure, distribution or copying of this communication is strictly prohibited. Whilst we take every reasonable precaution to screen out computer viruses from emails, attachments to the email may contain such viruses. We cannot accept liability for loss or damage resulting from such viruses. Calltracks Ltd is registered in England and Wales 6539973 at Unit 15, 3rd Floor, 23-28 Penn Street, London, N1 5DL
cdn
Struggling with full disk
August 15, 2016 07:12AM
The disk isn't full. It's just the colume. Usually it should create a new file?
cdn
Struggling with full disk
August 16, 2016 07:01AM
Sorry I mean the bacula volume is full...
Sorry, only registered users may post in this forum.

Click here to login