Backup Central

I am running a server with Debian Squeeze installed. It runs basic
stuff like Samba, Transmission, Apache2, nfsv4.

I use the nfs server to mount /raid/home/user on the server to
/home/user on my desktop. I use Backuppc to make a backup every night.
This has run for ages successfully until I tried to restore a backup
of /raid/home/user with Backuppc. After this I got the weirdest errors
from services like Apache2, backuppc, Munin, etc. about permissions
denied, libraries missing etc.
After some hours of hunting what was wrong I assumed that maybe my
filesystem was corrupted and decided to re-install the whole thing.

So I did that, got my little usb stick, reformatted the whole OS
partition, followed the Debian installer, mounted my /raid and
configured all my services etc.
It was great, just like I wanted it to be.

Then, still having the warm fuzzy content feeling of a perfectly
running system, I decided to do the same restore of a backup to
/raid/home/user with Backuppc (you can see it coming huh?) and guess
what: CRASH!

This was very disappointing as you can imagine...

So this is a selection of what I am seeing in some logs, most apps
don't even run any more:
/var/log/apache2/error.log:
libgcc_s.so.1 must be installed for pthread_cancel to work
/var/log/squid/cache.log:
2011/12/08 20:28:36| storeDirClean: /var/spool/squid/08/8C: (13)
Permission denied
/var/log/messages:
Dec 8 20:15:43 shuttle kernel: [74184.367948] nfsd: last server has
exited, flushing export cache
Dec 8 20:15:45 shuttle kernel: [74185.544398] svc: failed to
register lockdv1 RPC service (errno 97).
Dec 8 20:15:45 shuttle kernel: [74185.547937] NFSD: Using
/var/lib/nfs/v4recovery as the NFSv4 state recovery directory
Dec 8 20:15:45 shuttle kernel: [74185.559908] NFSD: starting
90-second grace period
Dec 8 20:16:46 shuttle kernel: [74247.454903] nfsd: last server has
exited, flushing export cache
Dec 8 20:16:48 shuttle kernel: [74248.536972] svc: failed to
register lockdv1 RPC service (errno 97).
Dec 8 20:16:48 shuttle kernel: [74248.540423] NFSD: Using
/var/lib/nfs/v4recovery as the NFSv4 state recovery directory
Dec 8 20:16:48 shuttle kernel: [74248.540471] NFSD: starting
90-second grace period

service transmission-daemon restart:
Restarting bittorrent daemon:
transmission-daemonstart-stop-daemon: unable to start
/usr/bin/transmission-daemon (Permission denied)

etc....


service backuppc restart:
Restarting backuppc...No process in pidfile
'/var/run/backuppc/BackupPC.pid' found running; none killed.
start-stop-daemon: unable to start /usr/share/backuppc/bin/BackupPC
(Permission denied)

The Backuppc log says:

2011-12-08 20:17:36 User backuppc requested restore to localhost (localhost)
2011-12-08 20:17:37 Started restore on localhost (pid=11576)
2011-12-08 19:28:11 localhost: Can't locate Carp/Heavy.pm in < at > INC
( < at > INC contains: /usr/share/backuppc/lib /etc/perl
/usr/local/lib/perl/5.10.1 /usr/local/share/perl/5.10.1 /usr/lib/perl5
/usr/share/perl5 /usr/lib/perl/5.10 /usr/share/perl/5.10
/usr/local/lib/site_perl .) at /usr/share/perl/5.10/Carp.pm line 39.
Exec of /bin/df /backup failed
2011-12-08 19:38:18 Got signal TERM... cleaning up
Can't locate Carp/Heavy.pm in < at > INC ( < at > INC contains:
/usr/share/backuppc/lib /etc/perl /usr/local/lib/perl/5.10.1
/usr/local/share/perl/5.10.1 /usr/lib/perl5 /usr/share/perl5
/usr/lib/perl/5.10 /usr/share/perl/5.10 /usr/local/lib/site_perl .) at
/usr/share/perl/5.10/Carp.pm line 39.
root < at > shuttle:/backup/log# Can't locate Carp/Heavy.pm in < at > INC ( < at > INC
contains: /usr/share/backuppc/lib /etc/perl /usr/local/lib/perl/5.10.1
/usr/local/share/perl/5.10.1 /usr/lib/perl5 /usr/share/perl5
/usr/lib/perl/5.10 /usr/share/perl/5.10 /usr/local/lib/site_perl .) at
/usr/share/perl/5.10/Carp.pm line 39.


Does anyone have a clue what is going on here, and possibly a way to
fix my system?
Any help very much appreciated,

Michel

------------------------------------------------------------------------------
Cloud Services Checklist: Pricing and Packaging Optimization
This white paper is intended to serve as a reference, checklist and point of
discussion for anyone considering optimizing the pricing and packaging model
of a cloud services business. Read Now!
http://www.accelacomm.com/jaw/sfnl/114/51491232/
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On Fri, Dec 9, 2011 at 1:51 AM, Michel <forerunnert < at > gmail.com> wrote:
I am running a server with Debian Squeeze installed. It runs basic
stuff like Samba, Transmission, Apache2, nfsv4.

I use the nfs server to mount /raid/home/user on the server to
/home/user on my desktop. I use Backuppc to make a backup every night.
This has run for ages successfully until I tried to restore a backup
of /raid/home/user with Backuppc. After this I got the weirdest errors
from services like Apache2, backuppc, Munin, etc. about permissions
denied, libraries missing etc.

It is never a good idea to overwrite running programs or the shared
libraries they are using. Sometimes you can get away with it using
rsync because it will write a new file and rename it when complete and
the way unix/linux works, anything with the original file open will
still see it until the last close. But, it doesn't look like you
intended to do that.

So this is a selection of what I am seeing in some logs, most apps
don't even run any more:
/var/log/apache2/error.log:
   libgcc_s.so.1 must be installed for pthread_cancel to work
/var/log/squid/cache.log:
  2011/12/08 20:28:36| storeDirClean: /var/spool/squid/08/8C: (13)
Permission denied

Why would your restore to /raid/home/user be affecting things under
/var? In any case, I'd recommend using rsync over ssh to the server
for the backups instead of backing up the nfs-mounted view.


The Backuppc log says:

2011-12-08 20:17:36 User backuppc requested restore to localhost (localhost)
2011-12-08 20:17:37 Started restore on localhost (pid=11576)
2011-12-08 19:28:11 localhost: Can't locate Carp/Heavy.pm in < at > INC
( < at > INC contains: /usr/share/backuppc/lib /etc/perl
/usr/local/lib/perl/5.10.1 /usr/local/share/perl/5.10.1 /usr/lib/perl5
/usr/share/perl5 /usr/lib/perl/5.10 /usr/share/perl/5.10
/usr/local/lib/site_perl .) at /usr/share/perl/5.10/Carp.pm line 39.
Exec of /bin/df /backup failed
2011-12-08 19:38:18 Got signal TERM... cleaning up
Can't locate Carp/Heavy.pm in < at > INC ( < at > INC contains:
/usr/share/backuppc/lib /etc/perl /usr/local/lib/perl/5.10.1
/usr/local/share/perl/5.10.1 /usr/lib/perl5 /usr/share/perl5
/usr/lib/perl/5.10 /usr/share/perl/5.10 /usr/local/lib/site_perl .) at
/usr/share/perl/5.10/Carp.pm line 39.
root < at > shuttle:/backup/log# Can't locate Carp/Heavy.pm in < at > INC ( < at > INC
contains: /usr/share/backuppc/lib /etc/perl /usr/local/lib/perl/5.10.1
/usr/local/share/perl/5.10.1 /usr/lib/perl5 /usr/share/perl5
/usr/lib/perl/5.10 /usr/share/perl/5.10 /usr/local/lib/site_perl .) at
/usr/share/perl/5.10/Carp.pm line 39.


Does anyone have a clue what is going on here, and possibly a way to
fix my system?
Any help very much appreciated,

How have you configured it to back up and restore localhost? If it
isn't using the stock commands over ssh, you may have gotten something
wrong in the layers of shell quote escaping and the restore command is
being interpreted incorrectly.

--
Les Mikesell
lesmikesell < at > gmail.com

------------------------------------------------------------------------------
Cloud Services Checklist: Pricing and Packaging Optimization
This white paper is intended to serve as a reference, checklist and point of
discussion for anyone considering optimizing the pricing and packaging model
of a cloud services business. Read Now!
http://www.accelacomm.com/jaw/sfnl/114/51491232/
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

I use the nfs server to mount /raid/home/user on the server to
/home/user on my desktop. I use Backuppc to make a backup every night.
This has run for ages successfully until I tried to restore a backup
of /raid/home/user with Backuppc. After this I got the weirdest errors
from services like Apache2, backuppc, Munin, etc. about permissions
denied, libraries missing etc.

It is never a good idea to overwrite running programs or the shared
libraries they are using.   Sometimes you can get away with it using
rsync because it will write a new file and rename it when complete and
the way unix/linux works, anything with the original file open will
still see it until the last close.  But, it doesn't look like you
intended to do that.


I didn´t intend to overwrite any files: /raid certainly wasn´t in use
at the time.

Why would your restore to /raid/home/user be affecting things under
/var?   In any case, I'd recommend using rsync over ssh to the server
for the backups instead of backing up the nfs-mounted view.

That is a very good question indeed..

The folder I was trying to restore was a harmless one in /raid that
should have nothing to do with the rest of the system as far as I
know... It's just the place I store the /home of a completely
different system.

rsync is not really an alternative for me because at the time the
backup runs usually only the server is running, not the client (my
desktop).


Does anyone have a clue what is going on here, and possibly a way to
fix my system?
Any help very much appreciated,

How have you configured it to back up and restore localhost?   If it
isn't using the stock commands over ssh, you may have gotten something
wrong in the layers of shell quote escaping and the restore command is
being interpreted incorrectly.


Yes it should back up /etc and /var of localhost (the server). But
these weren´t selected to be restored.

I am a little bit further: I found that all applications depending on
Perl are broken, and I found this in Backuppc log:

2011-12-08 20:17:37 Started restore on localhost (pid=11576)
2011-12-08 19:28:11 localhost: Can't locate Carp/Heavy.pm in < at > INC
( < at > INC contains: /usr/share/backuppc/lib /etc/perl
/usr/local/lib/perl/5.10.1 /usr/local/share/perl/5.10.1
/usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.10
/usr/share/perl/5.10 /usr/local/lib/site_perl .) at
/usr/share/perl/5.10/Carp.pm line 39.
Exec of /bin/df /backup failed
2011-12-08 19:38:18 Got signal TERM... cleaning up
Can't locate Carp/Heavy.pm in < at > INC ( < at > INC contains:
/usr/share/backuppc/lib /etc/perl /usr/local/lib/perl/5.10.1
/usr/local/share/perl/5.10.1
/usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.10
/usr/share/perl/5.10 /usr/local/lib/site_perl .) at
/usr/share/perl/5.10/Carp.pm line 39.

But I think it exists (/usr/share/perl/5.10.1 is symlinked to
/usr/share/perl/5.10):

# locate Heavy.pm
/usr/share/perl/5.10.1/Carp/Heavy.pm
/usr/share/perl/5.10.1/Exporter/Heavy.pm

Backuppc is running as user backuppc which is in sudoers to run tar as root.

I am starting to suspect that my Perl is messed up, because everything
that uses it is broken. I've tried to re-install its packages but no
luck yet.

------------------------------------------------------------------------------
Cloud Services Checklist: Pricing and Packaging Optimization
This white paper is intended to serve as a reference, checklist and point of
discussion for anyone considering optimizing the pricing and packaging model
of a cloud services business. Read Now!
http://www.accelacomm.com/jaw/sfnl/114/51491232/
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On Fri, Dec 9, 2011 at 2:50 PM, Michel <forerunnert < at > gmail.com> wrote:

The folder I was trying to restore was a harmless one in /raid that
should have nothing to do with the rest of the system as far as I
know... It's just the place I store the /home of a completely
different system.

rsync is not really an alternative for me because at the time the
backup runs usually only the server is running, not the client (my
desktop).

Backuppc is perfectly capable of backing itself up with rsync over
ssh. It's not the most efficient thing to do but not horrible either.
But you mentioned NFS mounts - where do they originate, and why
can't you talk to that server via ssh/rsync?


How have you configured it to back up and restore localhost?   If it
isn't using the stock commands over ssh, you may have gotten something
wrong in the layers of shell quote escaping and the restore command is
being interpreted incorrectly.


Yes it should back up /etc and /var of localhost (the server). But
these weren´t selected to be restored.

You didn't answer the question about the commands you changed to back
up localhost. Or, since that part seems to be working, the more
critical one is the command to restore. If you are doing something
locally, the commands need to not use the same level of quoting as
they would if they were passed to another shell via ssh.

I am a little bit further: I found that all applications depending on
Perl are broken, and I found this in Backuppc log:

I thought this was an effect of a restore gone wrong, not a cause.
Was it broken before the restore?

--
Les Mikesell
lesmikesell < at > gmail.com

------------------------------------------------------------------------------
Cloud Services Checklist: Pricing and Packaging Optimization
This white paper is intended to serve as a reference, checklist and point of
discussion for anyone considering optimizing the pricing and packaging model
of a cloud services business. Read Now!
http://www.accelacomm.com/jaw/sfnl/114/51491232/
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On Fri, 2011-12-09 at 21:50 +0100, Michel wrote:

Why would your restore to /raid/home/user be affecting things under
/var? In any case, I'd recommend using rsync over ssh to the server
for the backups instead of backing up the nfs-mounted view.

That is a very good question indeed..

The folder I was trying to restore was a harmless one in /raid that
should have nothing to do with the rest of the system as far as I
know... It's just the place I store the /home of a completely
different system.

Could it be that the io of the backuppc process is triggering a hardware
or software failure in the OS?

Is there anything in dmesg or /var/log ?

--
Tim Fletcher <tim < at > night-shade.org.uk>


------------------------------------------------------------------------------
Cloud Services Checklist: Pricing and Packaging Optimization
This white paper is intended to serve as a reference, checklist and point of
discussion for anyone considering optimizing the pricing and packaging model
of a cloud services business. Read Now!
http://www.accelacomm.com/jaw/sfnl/114/51491232/
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Why would your restore to /raid/home/user be affecting things under
/var?   In any case, I'd recommend using rsync over ssh to the server
for the backups instead of backing up the nfs-mounted view.

That is a very good question indeed..

The folder I was trying to restore was a harmless one in /raid that
should have nothing to do with the rest of the system as far as I
know... It's just the place I store the /home of a completely
different system.

Could it be that the io of the backuppc process is triggering a hardware
or software failure in the OS?

Is there anything in dmesg or /var/log ?


Sorry I am not very clear, I was struggling to understand what is
going on myself..

So, this was my use case:
1. client and server running, client has /home mounted from server,
where it is located at /raid/home
2. client is turned off
3. server backs up /raid/home
4. many hours pass
5. I connect to server via web interface and initiate a restore on the
server of /raid/home
6. CRASH!!
7. web interface to backuppc lost, huh?
8. samba lost, huh?
9. try munin webinterface, transmission web interface, everything dead, huh?
10. try to recover from serious disappointment

I was running backuppc with defaults, just enter paths and backup
scheme. I didn´t fiddle with commands and was using it successfully
for a few years. This config was running ~8 months. The new
installation ran only a few hours and did exactly the same thing:
restore same folder from same backup, same result.

I included relevant sections of dmesg and syslog, the rest of syslog
is more of the same without serious OS errors.
Thanks for the help

------------------------------------------------------------------------------
Cloud Services Checklist: Pricing and Packaging Optimization
This white paper is intended to serve as a reference, checklist and point of
discussion for anyone considering optimizing the pricing and packaging model
of a cloud services business. Read Now!
http://www.accelacomm.com/jaw/sfnl/114/51491232/
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On Fri, Dec 9, 2011 at 4:47 PM, Michel <forerunnert < at > gmail.com> wrote:
Sorry I am not very clear, I was struggling to understand what is
going on myself..

So, this was my use case:
1. client and server running, client has /home mounted from server,
where it is located at /raid/home
2. client is turned off
3. server backs up /raid/home

So the backuppc server is also the nfs server?

4. many hours pass
5. I connect to server via web interface and initiate a restore on the
server of /raid/home
6. CRASH!!

Everything so far points to your restore affecting / or /var instead
of /raid/home.

I was running backuppc with defaults, just enter paths and backup
scheme.

Which backup scheme?

--
Les Mikesell
lesmikesell < at > gmail.com

------------------------------------------------------------------------------
Learn Windows Azure Live! Tuesday, Dec 13, 2011
Microsoft is holding a special Learn Windows Azure training event for
developers. It will provide a great way to learn Windows Azure and what it
provides. You can attend the event by watching it streamed LIVE online.
Learn more at http://p.sf.net/sfu/ms-windowsazure
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/