Backup Central

(sorry if this comes thru as a dupe; i first sent this mail from an
unsubscribed account)


hi all,

i have a problem with bacula and incremental/differential backups
from an XFS filesystem.
to put it simple: bacula always creates full backups.


the long story:
we deployed bacula to backup our users' directory a while ago (that was
bacula-2.2.4 or so) and had good success so far (thanks, btw)
back then, the homedirectory on the fileserver (running debian/i386 with
bacula-fd) was a 2TB partition using ext3.
the backupserver (running debian/i386 with bacula-dir (postgresql) and
bacula-sd) stores all data on an DLT-S4 autochanger with a capacity of 12TB.
incremental backups of the user' homedirectories typically took less
than 30GB each day.
unfortunately our users filled up the homedirectory, so we decided to
upgrade the fileserver, and it now has a capacity of 20TB. since there
are no ext4 userland tools to handle such a capacity, we are now using
xfs as the filesystem. the fileserver is now also running an x86_64
system (still debian).
the new fileserver runs bacula-5.0.2 (as packaged for debian), so we had
to upgrade the backup-server from the originally used bacula-2.4.4 to
bacula-5.0.2; the upgrade had some problems (regarding the postgresql db
update), but it seems to have succeeded.
at least all the other servers that are being backed up, are doing fine.
these servers are running bacula-2.4.4 and bacula-2.2.8

unfortunately, the backup of the new fileserver makes problems:
there are 2 jobs for this server, one doing a system backup, and the
other one doing the backup of the users' homes.
the system backup seems to work fine, but the homes fail to do
incremental backups.
after an initial full backup started last wednesday (which consumed
1.33TB), an incremental backup was started on friday (which also
consumed 1.33TB, and is virtually identical to the full backup just
made); another incremental backup just started and has currently
consumed 280GB, indicating that it is running yet _another_ backup of
the entire fileset.

i wonder what went wrong?
i suspect that the problem is related to the use of XFS and/or to the
fileserver now being 64bit.

below you can find some technical info.

any help is appreciated.
esp. since i simply cannot afford to spend a DLT tape each day....


mfgasdr
IOhannes


bacula specific information
===------------------------
according to [2] i collected the directoris configuration and the output
of "list jobs" and "llist" for the Full-job and the Incr-job (that
wrongly did a full backup again). find them in the attached tgz.


operating systems
=----------------
backupserver$ uname -a
Linux backup.server 2.6.32-5-686 #1 SMP Thu Nov 3 04:23:54 UTC 2011 i686
GNU/Linux
fileserver$ uname -a
Linux file.server 2.6.32-5-amd64 #1 SMP Thu Nov 3 03:41:26 UTC 2011
x86_64 GNU/Linux

inode size
=---------
i compiled the following little program as found on [1] on both my
backupserver and my fileserver:
<snip>
#include <sys/stat.h>
#include <stdio.h>
int main(void) {
struct stat s;
printf("Size of inode is %d bytes\n\n", sizeof(s.st_ino));
return 0;
}
</snip>

results:
fileserver : Size of inode is 8 bytes
backupserver: Size of inode is 4 bytes

XFS on the fileserver
=--------------------
root < at > fileserver# xfs_info /dev/vda
meta-data=/dev/vda isize=256 agcount=20, agsize=268435455 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=5119999991, imaxpct=5
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=521728, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0



links
=----


[1]
http://www.backupcentral.com/phpBB2/two-way-mirrors-of-external-mailing-lists-3/bacula-25/bacula-xfs-and-inode64-98549/

[2]
http://www.bacula.org/5.2.x-manuals/en/problems/problems/Bacula_Frequently_Asked_Que.html#SECTION002280000000000000000


------------------------------------------------------------------------------
Write once. Port to many.
Get the SDK and tools to simplify cross-platform app development. Create
new or port existing apps to sell to consumers worldwide. Explore the
Intel AppUpSM program developer opportunity. appdeveloper.intel.com/join
http://p.sf.net/sfu/intel-appdev
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

On 01/10/2012 10:44 AM, IEM - network operating center (IOhannes m zmoelnig) wrote:
(sorry if this comes thru as a dupe; i first sent this mail from an
unsubscribed account)


hi all,

i have a problem with bacula and incremental/differential backups
from an XFS filesystem.
to put it simple: bacula always creates full backups.


the long story:
we deployed bacula to backup our users' directory a while ago (that was
bacula-2.2.4 or so) and had good success so far (thanks, btw)
back then, the homedirectory on the fileserver (running debian/i386 with
bacula-fd) was a 2TB partition using ext3.
the backupserver (running debian/i386 with bacula-dir (postgresql) and
bacula-sd) stores all data on an DLT-S4 autochanger with a capacity of 12TB.
incremental backups of the user' homedirectories typically took less
than 30GB each day.
unfortunately our users filled up the homedirectory, so we decided to
upgrade the fileserver, and it now has a capacity of 20TB. since there
are no ext4 userland tools to handle such a capacity, we are now using
xfs as the filesystem. the fileserver is now also running an x86_64
system (still debian).
the new fileserver runs bacula-5.0.2 (as packaged for debian), so we had
to upgrade the backup-server from the originally used bacula-2.4.4 to
bacula-5.0.2; the upgrade had some problems (regarding the postgresql db
update), but it seems to have succeeded.
at least all the other servers that are being backed up, are doing fine.
these servers are running bacula-2.4.4 and bacula-2.2.8

unfortunately, the backup of the new fileserver makes problems:
there are 2 jobs for this server, one doing a system backup, and the
other one doing the backup of the users' homes.
the system backup seems to work fine, but the homes fail to do
incremental backups.
after an initial full backup started last wednesday (which consumed
1.33TB), an incremental backup was started on friday (which also
consumed 1.33TB, and is virtually identical to the full backup just
made); another incremental backup just started and has currently
consumed 280GB, indicating that it is running yet _another_ backup of
the entire fileset.

i wonder what went wrong?
i suspect that the problem is related to the use of XFS and/or to the
fileserver now being 64bit.

below you can find some technical info.

any help is appreciated.
esp. since i simply cannot afford to spend a DLT tape each day....


mfgasdr
IOhannes


bacula specific information
===------------------------
according to [2] i collected the directoris configuration and the output
of "list jobs" and "llist" for the Full-job and the Incr-job (that
wrongly did a full backup again). find them in the attached tgz.


operating systems
=----------------
backupserver$ uname -a
Linux backup.server 2.6.32-5-686 #1 SMP Thu Nov 3 04:23:54 UTC 2011 i686
GNU/Linux
fileserver$ uname -a
Linux file.server 2.6.32-5-amd64 #1 SMP Thu Nov 3 03:41:26 UTC 2011
x86_64 GNU/Linux

inode size
=---------
i compiled the following little program as found on [1] on both my
backupserver and my fileserver:
<snip>
#include <sys/stat.h>
#include <stdio.h>
int main(void) {
struct stat s;
printf("Size of inode is %d bytes\n\n", sizeof(s.st_ino));
return 0;
}
</snip>

results:
fileserver : Size of inode is 8 bytes
backupserver: Size of inode is 4 bytes

XFS on the fileserver
=--------------------
root < at > fileserver# xfs_info /dev/vda
meta-data=/dev/vda isize=256 agcount=20, agsize=268435455 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=5119999991, imaxpct=5
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=521728, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0



links
=----


[1]
http://www.backupcentral.com/phpBB2/two-way-mirrors-of-external-mailing-lists-3/bacula-25/bacula-xfs-and-inode64-98549/

[2]
http://www.bacula.org/5.2.x-manuals/en/problems/problems/Bacula_Frequently_Asked_Que.html#SECTION002280000000000000000



First of all, I'm using also xfs on data source and/or backup media.
There's no special trouble on them to get incremental/differential backup.

What I suspect could be : an antivirus or any other scripts running on the xfs partition that change attribute or acl on files
It will be better for you have 5.0.3 or wait until 5.2.4 release version.

Check also the parameter used for your incremental (did you use Accurate Backup?)
If the filesystem xfs is not mounted with the noatime option, check to have noatime option in the bacula job.
so it will not change each file during the full backup, then creating a new fool backup for the incremental.


--

Bruno Friedmann
Ioda-Net Sàrl www.ioda-net.ch

openSUSE Member & Ambassador
GPG KEY : D5C9B751C4653227
irc: tigerfoot

------------------------------------------------------------------------------
Write once. Port to many.
Get the SDK and tools to simplify cross-platform app development. Create
new or port existing apps to sell to consumers worldwide. Explore the
Intel AppUpSM program developer opportunity. appdeveloper.intel.com/join
http://p.sf.net/sfu/intel-appdev
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

On 2012-01-10 14:19, Bruno Friedmann wrote:

First of all, I'm using also xfs on data source and/or backup media.
There's no special trouble on them to get incremental/differential backup.

thanks for your detailed answer.
i'm glad to hear that no troubles are to be expected in general. so not
all is lost...

What I suspect could be : an antivirus or any other scripts running on the xfs partition that change attribute or acl on files

good guess but unfortunately unlikely...or rather: the only script that
_is_ run and might update attributes is "mlocate.update"

It will be better for you have 5.0.3 or wait until 5.2.4 release version.

if i knew this would help, it might be an option.
until i know the cause of the problem, i'd rather stick with stock
debian packages (even if it means slightly outdated packages). i'm aware
that this is usually not what people want to hear on application
specific mailinglists, where a lot of work is put into making the latest
release as stable and bug-free as possible


Check also the parameter used for your incremental (did you use Accurate Backup?)

i might have to re-read the manual to understand what you mean by
"Accurate Backup". obviously i have read the manual to configure bacula
correctly for my needs back-then (using an old version of bacula & OS
and a different filesystem). since truths keep shifting, i might have to
refresh my knowledge...

If the filesystem xfs is not mounted with the noatime option, check to have noatime option in the bacula job.
so it will not change each file during the full backup, then creating a new fool backup for the incremental.

that is a very good point.
# cat /proc/mounts
/dev/vda /Net/data xfs rw,relatime,attr2,nobarrier,noquota 0 0

so the xfs-attribute "noatime" is not set (thought "relatime" is).
and the the bacula fileset misses the "noatime=yes" as well.

i will add "noatime=yes" to the fileset and see whether this helps....


mfgasdr
IOhannes



--
IEM - network operation center
mailto:noc < at > iem.at

------------------------------------------------------------------------------
Ridiculously easy VDI. With Citrix VDI-in-a-Box, you don't need a complex
infrastructure or vast IT resources to deliver seamless, secure access to
virtual desktops. With this all-in-one solution, easily deploy virtual
desktops for less than the cost of PCs and save 60% on VDI infrastructure
costs. Try it free! http://p.sf.net/sfu/Citrix-VDIinabox
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

On Wed, 11 Jan 2012 10:02:00 +0100, "IEM said:

On 2012-01-10 14:19, Bruno Friedmann wrote:

If the filesystem xfs is not mounted with the noatime option, check to have noatime option in the bacula job.
so it will not change each file during the full backup, then creating a new fool backup for the incremental.

that is a very good point.
# cat /proc/mounts
/dev/vda /Net/data xfs rw,relatime,attr2,nobarrier,noquota 0 0

so the xfs-attribute "noatime" is not set (thought "relatime" is).
and the the bacula fileset misses the "noatime=yes" as well.

i will add "noatime=yes" to the fileset and see whether this helps....

This should make no difference -- incremental backup is based on ctime and
mtime, not atime.

Please post your fileset definition.

Also, what is the output of

stat /path/to/some/file

where /path/to/some/file is a file that you didn't expect to see in the
incremental backup?

__Martin

------------------------------------------------------------------------------
Ridiculously easy VDI. With Citrix VDI-in-a-Box, you don't need a complex
infrastructure or vast IT resources to deliver seamless, secure access to
virtual desktops. With this all-in-one solution, easily deploy virtual
desktops for less than the cost of PCs and save 60% on VDI infrastructure
costs. Try it free! http://p.sf.net/sfu/Citrix-VDIinabox
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

On 2012-01-11 12:10, Martin Simmons wrote:
On Wed, 11 Jan 2012 10:02:00 +0100, "IEM said:

On 2012-01-10 14:19, Bruno Friedmann wrote:

If the filesystem xfs is not mounted with the noatime option, check to have noatime option in the bacula job.
so it will not change each file during the full backup, then creating a new fool backup for the incremental.

that is a very good point.
# cat /proc/mounts
/dev/vda /Net/data xfs rw,relatime,attr2,nobarrier,noquota 0 0

so the xfs-attribute "noatime" is not set (thought "relatime" is).
and the the bacula fileset misses the "noatime=yes" as well.

i will add "noatime=yes" to the fileset and see whether this helps....

This should make no difference -- incremental backup is based on ctime and
mtime, not atime.

darn.


Please post your fileset definition.


i already included it in the director configuration.
anyhow, here it is again:
<snip>
FileSet {
Name = "_Net_data"
Include {
Options {
signature = MD5
wildfile="*.o"
wildfile="*.pd_linux_o"
wildfile="*~"
wilddir="*/tmp"
wilddir="*/temp"
wilddir="*/Temp"
wilddir="*/TEMP"
wilddir="*/.svn"
wilddir="*/.imap"
wilddir="*/Maildir/.SPAM*"
wilddir="*/Trash"
wilddir="*/.Trash"
wilddir="*/RECYCLER"
wilddir="*/Temporary Internet Files"
wilddir="*/.DS_Store"
wilddir="*/.AppleDouble"
exclude="yes"
}
File = /Net/data
}
Exclude {
File = /Net/data/Benutzer.rsync
}
}
</snip>


Also, what is the output of

stat /path/to/some/file

where /path/to/some/file is a file that you didn't expect to see in the
incremental backup?

# stat /Net/data/Sound/ablinger/Altar/05KStudie2.wav
File: `/Net/data/Sound/ablinger/Altar/05KStudie2.wav'
Size: 53315180 Blocks: 104136 IO Block: 4096 regular file
Device: fe00h/65024d Inode: 4133733 Links: 1
Access: (0744/-rwxr--r--) Uid: (10144/ablinger) Gid: ( 100/ users)
Access: 2012-01-11 00:35:17.963002882 +0100
Modify: 2003-12-09 14:12:50.000000000 +0100
Change: 2011-10-04 21:35:44.745746088 +0200

so the times seem to be fine (the ctime is when i copied the old files
to the new fileserver)


alas.
btw, is there a way to "estimate" the next backup?
"estimate" seems to only estimate the fileset (that is: the "full" set)


fmgasr
IOhannes

--
IEM - network operation center
mailto:noc < at > iem.at

------------------------------------------------------------------------------
Ridiculously easy VDI. With Citrix VDI-in-a-Box, you don't need a complex
infrastructure or vast IT resources to deliver seamless, secure access to
virtual desktops. With this all-in-one solution, easily deploy virtual
desktops for less than the cost of PCs and save 60% on VDI infrastructure
costs. Try it free! http://p.sf.net/sfu/Citrix-VDIinabox
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

2012/1/11 IEM - network operating center (IOhannes m zmoelnig) <noc < at > iem.at ([email]noc < at > iem.at[/email])>
alas.
btw, is there a way to "estimate" the next backup?
"estimate" seems to only estimate the fileset (that is: the "full" set)

Did you try a level option of estimate command?

* estimate job="MyJob" level=Incremental

It is working for me at Bacula 5.0.3.

best regards


--
Radosław Korzeniewski
radoslaw < at > korzeniewski.net ([email]radoslaw < at > korzeniewski.net[/email])

On 2012-01-11 19:45, Radosław Korzeniewski wrote:

Did you try a level option of estimate command?

* estimate job="MyJob" level=Incremental

doh!
that indeed works, thanks a lot.
i couldn't find the option with "help" in bconsole/bat

novertheless, it's in the online manual and i seem to have missed it...


fgmasdr
IOhannes

--
IEM - network operation center
mailto:noc < at > iem.at

------------------------------------------------------------------------------
RSA(R) Conference 2012
Mar 27 - Feb 2
Save $400 by Jan. 27
Register now!
http://p.sf.net/sfu/rsa-sfdev2dev2
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

On Wed, 11 Jan 2012 13:12:56 +0100, "IEM said:

i already included it in the director configuration.
anyhow, here it is again:

Sorry, I missed the attachment.


<snip>
FileSet {
Name = "_Net_data"
Include {
Options {
signature = MD5
wildfile="*.o"
wildfile="*.pd_linux_o"
wildfile="*~"
wilddir="*/tmp"
wilddir="*/temp"
wilddir="*/Temp"
wilddir="*/TEMP"
wilddir="*/.svn"
wilddir="*/.imap"
wilddir="*/Maildir/.SPAM*"
wilddir="*/Trash"
wilddir="*/.Trash"
wilddir="*/RECYCLER"
wilddir="*/Temporary Internet Files"
wilddir="*/.DS_Store"
wilddir="*/.AppleDouble"
exclude="yes"
}
File = /Net/data
}
Exclude {
File = /Net/data/Benutzer.rsync
}
}
</snip>

OK, no sign of accurate backup there.


# stat /Net/data/Sound/ablinger/Altar/05KStudie2.wav
File: `/Net/data/Sound/ablinger/Altar/05KStudie2.wav'
Size: 53315180 Blocks: 104136 IO Block: 4096 regular file
Device: fe00h/65024d Inode: 4133733 Links: 1
Access: (0744/-rwxr--r--) Uid: (10144/ablinger) Gid: ( 100/ users)
Access: 2012-01-11 00:35:17.963002882 +0100
Modify: 2003-12-09 14:12:50.000000000 +0100
Change: 2011-10-04 21:35:44.745746088 +0200

so the times seem to be fine (the ctime is when i copied the old files
to the new fileserver)

That is very strange, because the Modify and Change times are both old.
Without accurate backup, that is all that Bacula uses to detect changed files.

Can you repeat the problem with a fileset containing just that file?

__Martin

------------------------------------------------------------------------------
RSA(R) Conference 2012
Mar 27 - Feb 2
Save $400 by Jan. 27
Register now!
http://p.sf.net/sfu/rsa-sfdev2dev2
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

On 2012-01-12 17:32, Martin Simmons wrote:
On Wed, 11 Jan 2012 13:12:56 +0100, "IEM said:


# stat /Net/data/Sound/ablinger/Altar/05KStudie2.wav
File: `/Net/data/Sound/ablinger/Altar/05KStudie2.wav'
Size: 53315180 Blocks: 104136 IO Block: 4096 regular file
Device: fe00h/65024d Inode: 4133733 Links: 1
Access: (0744/-rwxr--r--) Uid: (10144/ablinger) Gid: ( 100/ users)
Access: 2012-01-11 00:35:17.963002882 +0100
Modify: 2003-12-09 14:12:50.000000000 +0100
Change: 2011-10-04 21:35:44.745746088 +0200

so the times seem to be fine (the ctime is when i copied the old files
to the new fileserver)

That is very strange, because the Modify and Change times are both old.
Without accurate backup, that is all that Bacula uses to detect changed files.

it seems that problem solved itself (well, i certainly didn't)...

after i found out how to do a job estimation for an incremental backup
and that estimation appeared to be sane, i re-run the job and since then
the backup volume is as expected (1.1GB rather than 1.3TB).

my fileset now includes the "noatime = yes" directive, since i don't see
how this could do any harm and mayne (just maybe) it did indeed fix the
problem.


thanks for your support.

fgmasdr
IOhannes

------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users