Backup Central

Hi,

Little annoying problem with errors en rsnapshot (log) verbosity.

I usually put verbose level on 2, and the loglevel on 3.
That way, I get emails only when an error occurs, and when neccessary I can check the log file for extra info.

For performing backups of some databases, I have a simple script that remotely logs in through SSH and dumps the said databases
(http://support.ginsys.be/wsvn/scripts/backupdump/backup_hostdb.sh if you're interested).
Note that I run that script one time per remote host, with the remote host as a parameter. So if I can see which instance of the script was called, I can know on which host it ran.

On one host, the mysqldump part of the script generates an error, which point to a problem on the mysql setup (corrupt tables).
That error is the only piece of text that is being sent to STDOUT, and hence gets mailed through cron. Having only that error text, I can't know at what point rsnapshot launched the backup script, and hence I can't figure out which host experiences this problem.

So far so good, that's logical given the "verbose 2" level. My problem now arises when looking into the log file, hoping to find that same error, and seeing the context to know on which host it happens. But I can't find anny report or mention of an error.

I tried putting using "loglevel 4" but all I get is something like

[12/Mar/2010:00:24:08] mkdir -m 0755 -p /srv/rsnap/shots/tmp/
[12/Mar/2010:00:24:08] cd /srv/rsnap/shots/tmp/
[12/Mar/2010:00:24:08] /usr/local/bin/backup_hostdb.sh -h rsnapshot < at > ilabolamptest.ibbt.be ([email]rsnapshot < at > ilabolamptest.ibbt.be[/email]) -d -m
[12/Mar/2010:00:24:11] cd /root/
[12/Mar/2010:00:24:11] sync_if_different("/srv/rsnap/shots/tmp/", "/srv/rsnap/shots/hourly.0/DUMPS/ilabolamptest/")
[12/Mar/2010:00:24:12] /bin/rm -rf /srv/rsnap/shots/tmp/

doing a
  grep -A1 backup_hostdb /var/log/rsnapshot
gives me the same'ish output for all called instances.

So the mysql error does not get back to the rsnapshot log.
rsnapshot doesn't even seem to detect an error state, but I guess that's normal, as I don't do proper error code handling in my script.

Could it be that rsnapshot doesn't pick up the script output (STDOUT and? STDERR) to be able to log it?

Any help on how to improve my script or my rsnapshot config is kindly appreciated.
--

    Met vriendelijke groet,

    Serge van Ginderachter

On Fri, Mar 12, 2010 at 11:37:14AM +0100, Serge van Ginderachter wrote:

For performing backups of some databases, I have a simple script that
remotely logs in through SSH and dumps the said databases
Note that I run that script one time per remote host, with the remote
host as a parameter. So if I can see which instance of the script was
called, I can know on which host it ran.
On one host, the mysqldump part of the script generates an error,
which point to a problem on the mysql setup (corrupt tables).
That error is the only piece of text that is being sent to STDOUT, and
hence gets mailed through cron. Having only that error text, I can't
know at what point rsnapshot launched the backup script, and hence I
can't figure out which host experiences this problem.
So far so good, that's logical given the "verbose 2" level. My problem
now arises when looking into the log file, hoping to find that same
error, and seeing the context to know on which host it happens. But I
can't find anny report or mention of an error.
So the mysql error does not get back to the rsnapshot log.
rsnapshot doesn't even seem to detect an error state, but I guess
that's normal, as I don't do proper error code handling in my script.
Could it be that rsnapshot doesn't pick up the script output (STDOUT
and? STDERR) to be able to log it?

That is correct.
rsnapshot does NOT duplicate the stdout or stderr of backup_scripts to
put a copy in the logfile.

If the backup_script returns with a non-zero exit status, then rsnapshot
will note that to STDERR (if verbose >= 2) and logfile (if loglevel => 2)
and syslog (if cmd_logger is defined). But it will never put a copy of
stdout (nor stderr) from the script into the logfile.

Rsnapshot will also do a rollback of the directory the backup_script was
supposed to produce output into (as if the backup_script had produced the
same results as the previous backup), and log that rollback also.

So this is what should happen if the script uses an exit status to signal
that there is a problem.

One way you might be able to tell which mysqldump is failling is to look
at the destination directories (in the snapshot root). If the script isn't
detecting the error, then I would expect the destination directory for the
failled host to be empty (or contain empty files).

Any help on how to improve my script or my rsnapshot config is kindly
appreciated.

My suggestions:

(1) Detect errors with mysql in your script (maybe add "|| ERR=$ERR+$ERR_MYSQL"
on the end of the mysqldump line)

(2) You could add something like
if [ $ERR -gt 0 ]; then echo "Error $ERR on host $REMOTEHOST"; fi
shortly before the exit line, so the script confirms on stdout which host
had the error.

(3) You could add a "umask 007" near the top of the script (before it creates
any files) so that dump files are not readable while the script runs

--
___________________________________________________________________________
David Keegel <djk < at > cybersource.com.au> http://www.cyber.com.au/users/djk/
Cybersource P/L: Linux/Unix Systems Administration Consulting/Contracting

------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

On Sun, 2010-03-14 at 15:49 +1100, David Keegel wrote:
rsnapshot does NOT duplicate the stdout or stderr of backup_scripts to
put a copy in the logfile.

Maybe it should? A log file that the user has affirmatively specified
seems like a better place to send output than a stdout/stderr that may
or may not go anywhere useful depending on the setup.

--
Matt


------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss