Backup Central

Dave Hawkes <daveh < at > cadlink.com>
wrote the following on Mon, 19 Apr 2004 08:56:43 -0400

I've been running rdiff-backup (ver. 13.4) reliably for quite some
time but now consistently get the following crash when trying to
backup up a particular machine (see below). Has anyone any ideas?
...
File "/usr/lib/python2.2/site-packages/rdiff_backup/log.py", line 134,
in log_to_term
termfp.write(self.format(message, self.term_verbosity))
IOError: [Errno 11] Resource temporarily unavailable

I managed to fix the problem by lowering the terminal-verbosity setting
to 4 from 5. It would appear that there was some contention for stdout,
though I'm not sure how that could happen. The problem can likely be
fixed with an exception handler around the code that writes to stdout
and then deals with this event in an intelligent way.

You're right---it seems stdout is temporarily unavailable. Does
anyone know how this can happen? The whole server is built around
writing to stdout, so checking for this error could add a lot of
overhead.

Perhaps some un*x gurus can tell me 1) when this error occurs and 2)
is it really necessary to check for EAGAIN every time data is written
to a file descriptor? The examples I see in, say, books covering C on
unix generally don't check.


--
Ben Escoto

On 16 May 2004, Ben Escoto <bescoto < at > stanford.edu> wrote:
Dave Hawkes <daveh < at > cadlink.com>
wrote the following on Mon, 19 Apr 2004 08:56:43 -0400

I've been running rdiff-backup (ver. 13.4) reliably for quite some
time but now consistently get the following crash when trying to
backup up a particular machine (see below). Has anyone any ideas?
...
File "/usr/lib/python2.2/site-packages/rdiff_backup/log.py", line 134,
in log_to_term
termfp.write(self.format(message, self.term_verbosity))
IOError: [Errno 11] Resource temporarily unavailable

I managed to fix the problem by lowering the terminal-verbosity setting
to 4 from 5. It would appear that there was some contention for stdout,
though I'm not sure how that could happen. The problem can likely be
fixed with an exception handler around the code that writes to stdout
and then deals with this event in an intelligent way.

You're right---it seems stdout is temporarily unavailable. Does
anyone know how this can happen? The whole server is built around
writing to stdout, so checking for this error could add a lot of
overhead.

Perhaps some un*x gurus can tell me 1) when this error occurs and 2)
is it really necessary to check for EAGAIN every time data is written
to a file descriptor? The examples I see in, say, books covering C on
unix generally don't check.

I haven't reproduced the bug or anything so this is kind of handwavey:

You should only see EAGAIN on a file descriptor in nonblocking mode.
This can sometimes happen on programs invoked over SSH. In this case
the fd is a pipe or unix domain socket connecting the rdiff-backup
server to the sshd child. So EAGAIN (aka EWOULDBLOCK) will happen
when rdiff-backup produces output faster than sshd can consume it. As
you might expect lowering the terminal verbosity fixes that.

So there are basically two things you can do:

1 - Check for EAGAIN; when you get it then select() on the socket and
retry. Presumably a good way in Python is to just use a little
decorator object around the real file.

2 - Make the socket nonblocking. But I don't know if that would upset
sshd.

(Congratulations on your dissertation submission and job.)

--
Martin

2 - Make the socket nonblocking. But I don't know if that would upset
sshd.

i think you meant to say make the socket blocking... blocking
sockets/pipes should never issue EAGAIN.

-dean

Martin Pool <mbp < at > samba.org>
wrote the following on Mon, 17 May 2004 16:58:59 +1000

You should only see EAGAIN on a file descriptor in nonblocking mode.
This can sometimes happen on programs invoked over SSH. In this case
the fd is a pipe or unix domain socket connecting the rdiff-backup
server to the sshd child. So EAGAIN (aka EWOULDBLOCK) will happen
when rdiff-backup produces output faster than sshd can consume it. As
you might expect lowering the terminal verbosity fixes that.

So there are basically two things you can do:

1 - Check for EAGAIN; when you get it then select() on the socket and
retry. Presumably a good way in Python is to just use a little
decorator object around the real file.

2 - Make the socket nonblocking. But I don't know if that would upset
sshd.

(As Dean Gaudet noted, presumably you mean "blocking" for #2.) But
why would stdout be in non-blocking mode to begin with? Clearly on
most systems it's in blocking mode, or else almost every instance of
rdiff-backup would fail (because the server side is often writing to
stdout as fast as it can).

When rsync is writing to stdout when run over a pipe, do you remember
if it fcntls stdout to make sure it is blocking?

Also, and this may be an important clue, sshd may not have started the
crashed process in Dave Hawkes's example. It was the client that
crashed when trying to write to stdout, so stdout was most likely
connected to a terminal.

If Dave is still interested, perhaps he could tell us what kind of
system he has and how rdiff-backup is run on it.


--
Ben Escoto