Backup Central

I ran into an issue with backuppc (Debian/sid, last upgraded 4/27/12),
and I'm trying to figure out what the problem is. Nagios alerted today
that the /var/lib/backuppc (500GB) filesystem was critical. I logged
in and found the following:

/dev/mapper/vg00-backuppc ext4 523733076 496248972 1269704 100%
/var/lib/backuppc

However, I can't reconcile those numbers with the numbers backuppc is
showing. My 4-week graph shows the following for the cpool:

Current 331.94GB Average 328.61GB Max 337.15GB

The stats also say

Nightly cleanup removed 0 files of size 0.00GB (around 5/29 01:00),

I have not added any machines or had any huge additions of files, but
also notice that backups haven't run in 2 days, presumably because of
the full filesystem.

I added 100GB to that filesystem, but would really like to reconcile
where the space went, and at some point, would like to stick that
spare 100GB back in reserve until I need it. (I know, some would say
I need it now, but I can think of no reason for the filesystem to
mushroom like that.)

So where might the space have gone (given that the backup landscape
hasn't changed)? Could something have kept the nightlies from running
and deleting old backups, which then accumulated?

Thanks,
--b

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

I know it is bad form to respond to one's own post, but I was digging
around in my Munin graphs, and noticed that the filesystem skyrocked
from about 70% to 100% late on the 26th or early on the 27th. I have
included both the 4-week pool graph from the backup machine and the
weekly graph from my munin server. Note that while the pool filesystem
is smooth and even at around 300GB, the munin graph climbs sharply...I
wasn't able to find anything that might have caused it, though doing
df on the pc directory, with all it's hard links, seems to be a Bad
Idea.

Thanks,
--b

On Tue, May 29, 2012 at 7:16 PM, Brad Alexander <storm16 < at > gmail.com> wrote:
I ran into an issue with backuppc (Debian/sid, last upgraded 4/27/12),
and I'm trying to figure out what the problem is. Nagios alerted today
that the /var/lib/backuppc (500GB) filesystem was critical. I logged
in and found the following:

/dev/mapper/vg00-backuppc ext4 523733076 496248972   1269704 100%
/var/lib/backuppc

However, I can't reconcile those numbers with the numbers backuppc is
showing. My 4-week graph shows the following for the cpool:

Current 331.94GB Average 328.61GB Max 337.15GB

The stats also say

Nightly cleanup removed 0 files of size 0.00GB (around 5/29 01:00),

I have not added any machines or had any huge additions of files, but
also notice that backups haven't run in 2 days, presumably because of
the full filesystem.

I added 100GB to that filesystem, but would really like to reconcile
where the space went, and at some point, would like to stick that
spare 100GB back in reserve until I need it. (I know, some would say
I need it now, but I can think of no reason for the filesystem to
mushroom like that.)

So where might the space have gone (given that the backup landscape
hasn't changed)? Could something have kept the nightlies from running
and deleting old backups, which then accumulated?

Thanks,
--b

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On Tue, 2012-05-29 at 19:27 -0400, Brad Alexander wrote:
I know it is bad form to respond to one's own post, but I was digging
around in my Munin graphs, and noticed that the filesystem skyrocked
from about 70% to 100% late on the 26th or early on the 27th. I have
included both the 4-week pool graph from the backup machine and the
weekly graph from my munin server. Note that while the pool filesystem
is smooth and even at around 300GB, the munin graph climbs sharply...I
wasn't able to find anything that might have caused it, though doing
df on the pc directory, with all it's hard links, seems to be a Bad
Idea.

I've been caught out by things like thunderbird indexing 10 gig of mail
suddenly and adding a massive chunk to my backups.

You can use du on parts of the pool file system as modern versions of du
will skip hardlinks after it's counted the first instance of them.

--
Tim Fletcher <tim < at > night-shade.org.uk>


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On 2012-05-30 00:27, Brad Alexander wrote:
I know it is bad form to respond to one's own post, but I was digging
around in my Munin graphs, and noticed that the filesystem skyrocked
from about 70% to 100% late on the 26th or early on the 27th. I have
included both the 4-week pool graph from the backup machine and the
weekly graph from my munin server. Note that while the pool filesystem
is smooth and even at around 300GB, the munin graph climbs sharply...

This indicates the disk usage is outside the pool.

Using du on these directories would be bad (take a long time, not be
helpful due to hard links): cpool pool pc

Everything else you can check. Please do so first.

1. Do you have anything else in this filesystem, or is it mounted on
/var/lib/backuppc?

2. Check a few backups made during the window that usage increased. Are
they linked into the pool? For instance:

root < at > backup:/var/lib/backuppc/pc/router07/547# stat f%2f/fvar/flog/fsyslog
File: `f%2f/fvar/flog/fsyslog'
Size: 20440 Blocks: 40 IO Block: 4096 regular file
Device: fb00h/64256d Inode: 105783860 Links: 2
Access: (0640/-rw-r-----) Uid: ( 106/backuppc) Gid: ( 116/backuppc)
Access: 2012-05-24 00:08:56.727735552 +0000
Modify: 2012-05-25 02:31:52.655574930 +0000
Change: 2012-05-25 02:31:52.655574930 +0000

This is a file that is unique on every host, but it has two links. Meaning
it exists here in the pc directory, and in the pool. If this is 1, this
backup is not linked into the pool and is contributing to this usage problem.

Regards,
Tyler

--
A bad analogy is like a leaky screwdriver.

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

I think I just found it. Ironically enough, it was my workstation. I
have an external drive that is normally plugged into my laptop for
files I need to transport I had plugged (and left plugged) this drive
into my desktop, which was being indexed and attempted to be backed
up. Apparently, this filled the filesystem, and the backup never
completed, but the files from the partial backup were apparently
stored somewhere. Once I umounted the portable hard drive, the backup
completed, and the drive space went back down to normal levels.

I have since commented out all of /media.

Thanks,
--b


On Wed, May 30, 2012 at 6:59 AM, Tyler J. Wagner <tyler < at > tolaris.com> wrote:
On 2012-05-30 00:27, Brad Alexander wrote:
I know it is bad form to respond to one's own post, but I was digging
around in my Munin graphs, and noticed that the filesystem skyrocked
from about 70% to 100% late on the 26th or early on the 27th. I have
included both the 4-week pool graph from the backup machine and the
weekly graph from my munin server. Note that while the pool filesystem
is smooth and even at around 300GB, the munin graph climbs sharply...

This indicates the disk usage is outside the pool.

Using du on these directories would be bad (take a long time, not be
helpful due to hard links): cpool pool pc

Everything else you can check. Please do so first.

1. Do you have anything else in this filesystem, or is it mounted on
/var/lib/backuppc?

2. Check a few backups made during the window that usage increased. Are
they linked into the pool? For instance:

root < at > backup:/var/lib/backuppc/pc/router07/547# stat f%2f/fvar/flog/fsyslog
 File: `f%2f/fvar/flog/fsyslog'
 Size: 20440           Blocks: 40         IO Block: 4096   regular file
Device: fb00h/64256d    Inode: 105783860   Links: 2
Access: (0640/-rw-r-----)  Uid: (  106/backuppc)   Gid: (  116/backuppc)
Access: 2012-05-24 00:08:56.727735552 +0000
Modify: 2012-05-25 02:31:52.655574930 +0000
Change: 2012-05-25 02:31:52.655574930 +0000

This is a file that is unique on every host, but it has two links. Meaning
it exists here in the pc directory, and in the pool. If this is 1, this
backup is not linked into the pool and is contributing to this usage problem.

Regards,
Tyler

--
A bad analogy is like a leaky screwdriver.

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On Wed, 2012-05-30 at 10:15 -0400, Brad Alexander wrote:
I think I just found it. Ironically enough, it was my workstation. I
have an external drive that is normally plugged into my laptop for
files I need to transport I had plugged (and left plugged) this drive
into my desktop, which was being indexed and attempted to be backed
up. Apparently, this filled the filesystem, and the backup never
completed, but the files from the partial backup were apparently
stored somewhere. Once I umounted the portable hard drive, the backup
completed, and the drive space went back down to normal levels.

I have since commented out all of /media.

Changing the rsync flags to include --one-file-system at the top level
will stop this catching you out again, I think it should be a
configuration default.

--
Tim Fletcher <tim < at > night-shade.org.uk>


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

I can't really do that, since I tend to segregate my filesystems
anyway. I used to have my non-standard stuff in /media/archive (which
was backed up), I just turned off /media/cdrom and an nfs mount there.
When I rebuilt, I decided /data was a better choice, but didn't think
to turn off /media.

--b

On Wed, May 30, 2012 at 12:12 PM, Tim Fletcher <tim < at > night-shade.org.uk> wrote:
On Wed, 2012-05-30 at 10:15 -0400, Brad Alexander wrote:
I think I just found it. Ironically enough, it was my workstation. I
have an external drive that is normally plugged into my laptop for
files I need to transport I had plugged (and left plugged) this drive
into my desktop, which was being indexed and attempted to be backed
up. Apparently, this filled the filesystem, and the backup never
completed, but the files from the partial backup were apparently
stored somewhere. Once I umounted the portable hard drive, the backup
completed, and the drive space went back down to normal levels.

I have since commented out all of /media.

Changing the rsync flags to include --one-file-system at the top level
will stop this catching you out again, I think it should be a
configuration default.

--
Tim Fletcher <tim < at > night-shade.org.uk>


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On Wed, May 30, 2012 at 8:00 PM, Brad Alexander <storm16 < at > gmail.com> wrote:
I can't really do that, since I tend to segregate my filesystems
anyway. I used to have my non-standard stuff in /media/archive (which
was backed up), I just turned off /media/cdrom and an nfs mount there.
When I rebuilt, I decided /data was a better choice, but didn't think
to turn off /media.

All you have to do is add explicit 'shares' for every mount point that
you do want to back up. As a bonus, --one-file-system will
automatically exclude the pseudo-mounts like /proc and /sys, etc.

--
Les Mikesell
lesmikesell < at > gmail.com

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/