Backup Central

I mentioned the long filename problem before, and Chris Wilson
mentioned that I forgot one case: where the path as a whole is too
long, even if each file/directory name is short.

I thought this could be solved by changing directories, but after
thinking about it this would actually be a pain: I'd need to add
either a check or a chdir before every file operation. And I'm sure
I'd forget some places, and would be finding random (hard to
diagnose?) errors after this.

Also a path may be more than 2x as long as the limit. These long
paths would require 2 or more chdirs to reach.


So anyway, I'm curious when/how the long-path-problem is coming up. I
wish rdiff-backup could successfully mirror all the files in this
case, but don't see a way of doing this that's not error-prone.
(Treating them the same way as long filename files and sticking them
in a separate directory would be a bit easier.)


--
Ben Escoto

On Sat, Nov 19, 2005 at 08:52:30PM -0600, Ben Escoto wrote:
[...]
So anyway, I'm curious when/how the long-path-problem is coming up. I
wish rdiff-backup could successfully mirror all the files in this
case, but don't see a way of doing this that's not error-prone.
(Treating them the same way as long filename files and sticking them
in a separate directory would be a bit easier.)

This was my most recent error along these lines:

SpecialFileError
var/lib/apache/fastcgi/80292d047b4ad9c7faad6b676c4fe423 Socket
error: AF_UNIX path too long

I get these every now and then. Presumably these are temporary files
(sockets, actually, in this case) being created and destroyed rapidly,
and so I only see the error when the planets align and the backup
coincides with the temporary file. I don't worry about it in this
case. I don't recall ever seeing it on non-socket files for that
matter. Now that I think about it, shouldn't rdiff-backup be ignoring
sockets files?

Hi Ben,

So anyway, I'm curious when/how the long-path-problem is coming up. I
wish rdiff-backup could successfully mirror all the files in this
case, but don't see a way of doing this that's not error-prone.
(Treating them the same way as long filename files and sticking them
in a separate directory would be a bit easier.)

I used a simple shell script on a Linux ext3 filesystem:

while mkdir aaaaaaaa && cd aaaaaaaa; do true; done

The path got to nearly 8000 characters long, and although cd was giving
warnings about "getcwd: path name too long", it was still going when I
killed it.

When I try to back up the root of this huge tree with rdiff-backup, I get
the following error:

Processing changed file aaaaaaaa/aaaaaaaa/.../aaaaaaaa
Exception '[Errno 36] File name too long: 'aaaaaaaa/.../aaaaaaaaa'' raised
of class 'exceptions.OSError':
File "/usr/lib/python2.3/site-packages/rdiff_backup/robust.py", line 32,
in check_common_error
try: return function(*args)
File "/usr/lib/python2.3/site-packages/rdiff_backup/rpath.py", line 922,
in append
return self.__class__(self.conn, self.base, self.index + (ext,))
File "/usr/lib/python2.3/site-packages/rdiff_backup/rpath.py", line 669,
in __init__
else: self.setdata()
File "/usr/lib/python2.3/site-packages/rdiff_backup/rpath.py", line 693,
in setdata
self.data = self.conn.C.make_file_dict(self.path)

Hope this helps,

Cheers, Chris.
--
_ ___ __ _
/ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |

Randall Nortman <rdiffbackuplist < at > wonderclown.com>
wrote the following on Mon, 21 Nov 2005 09:50:51 -0500

This was my most recent error along these lines:

SpecialFileError
var/lib/apache/fastcgi/80292d047b4ad9c7faad6b676c4fe423 Socket
error: AF_UNIX path too long

Yes, for some reason (historical, presumably) sockets need to be
shorter than any other kind of file.

But those don't count since I think rdiff-backup already handles those
socket cases satisfactorily :)


--
Ben Escoto

Chris Wilson <chris+rdiff-backup < at > qwirx.com>
wrote the following on Tue, 22 Nov 2005 20:47:29 +0000 (GMT)

I used a simple shell script on a Linux ext3 filesystem:

while mkdir aaaaaaaa && cd aaaaaaaa; do true; done

The path got to nearly 8000 characters long, and although cd was giving
warnings about "getcwd: path name too long", it was still going when I
killed it.

May I ask why you did this? Is it ok if rdiff-backup just skipped
over a lot of those directories? (As long as it didn't crash of
course.) Or do you need that structure correctly mirrored?


--
Ben Escoto

Hi Ben,

The path got to nearly 8000 characters long, and although cd was giving
warnings about "getcwd: path name too long", it was still going when I
killed it.

May I ask why you did this?

For kicks? I wanted to see if I could make my kernel panic? And to see
if rdiff-backup really did have a problem with long path names?

Is it ok if rdiff-backup just skipped over a lot of those directories?
(As long as it didn't crash of course.) Or do you need that structure
correctly mirrored?

I don't need it mirrored as such, it was a kind of stress test for
rdiff-backup. But I think it shows that rdiff-backup may have problems
backing up some legitimate directory structures, and that the long path
problem is not just theoretical.

Cheers, Chris.
--
_ ___ __ _
/ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |

Chris Wilson <chris < at > qwirx.com>
wrote the following on Wed, 23 Nov 2005 00:23:01 +0000 (GMT)

I don't need it mirrored as such, it was a kind of stress test for
rdiff-backup. But I think it shows that rdiff-backup may have problems
backing up some legitimate directory structures, and that the long path
problem is not just theoretical.

Oh I see, I've done something similar on my system and know errors can
be produced. I'm not sure your example qualifies as a legitimate
directory structure though :-)

I think most people will be fine until their paths reach about 3950
chars. So I was mainly wondering if it's ok to skip paths longer than
that, or if someone actually has data they want to save in files whose
paths are 5kB long.


--
Ben Escoto

On Tue, 22 Nov 2005, Ben Escoto wrote:

I think most people will be fine until their paths reach about 3950
chars. So I was mainly wondering if it's ok to skip paths longer than
that, or if someone actually has data they want to save in files whose
paths are 5kB long.


If I may barge in, I think it actually does happen legitimately, when a
file-level crypto filesystem is being backed up (not the plaintext inside
but the encrypted tree). For example EncFS pads filenames/dirnames to
make it harder to guess the original filename. It is not uncommon to end
up with very long pathnames in this mode (which is the default).

True, shortly after the 3950 limit is reached, EncFS will itself fail due
to inability to write the encrypted files. However, some pathnames may
fall in between so they will be written correctly by EncFS but not backed
up by rdiff-backup.

Yoav