| View previous topic :: View next topic |
| Author |
Message |
njbailey
Guest
|
 Posted: Mon Apr 11, 2005 7:02 am Post subject: Integer too small on AMD64 machine |
 |
|
Hi,
I've been trying to use rdiff-backup to copy between two locally mounted disk
drives on a AMD64 box running Ubuntu. After creating about 16BG of files in the
destination directory, the command fails with an integer overflow error.
Is this a 64-bit problem? I don't see anything in the FAQ or Wiki which might
help out.
I'm using python 2.4.1, librsync1 0.9.7 and rsync 2.6.3-2ubuntu1 packages.
Interested to know if anyone else is experiencing this problem.
Nick/.
Stacktrace follows...
Exception 'signed integer is greater than maximum' raised of class
'exceptions.OverflowError':
File "/usr/lib/python2.4/site-packages/rdiff_backup/robust.py", line 32, in
check_common_error
try: return function(*args)
File "/usr/lib/python2.4/site-packages/rdiff_backup/rpath.py", line 905, in append
return self.__class__(self.conn, self.base, self.index + (ext,))
File "/usr/lib/python2.4/site-packages/rdiff_backup/rpath.py", line 667, in
__init__
else: self.setdata()
File "/usr/lib/python2.4/site-packages/rdiff_backup/rpath.py", line 692, in
setdata
if self.lstat(): self.conn.rpath.setdata_local(self)
File "/usr/lib/python2.4/site-packages/rdiff_backup/rpath.py", line 1173, in
setdata_local
rpath.data['uname'] = user_group.uid2uname(rpath.data['uid'])
File "/usr/lib/python2.4/site-packages/rdiff_backup/user_group.py", line 49,
in uid2uname
try: uname = pwd.getpwuid(uid)[0]
Traceback (most recent call last):
File "/usr/bin/rdiff-backup", line 23, in ?
rdiff_backup.Main.Main(sys.argv[1:])
File "/usr/lib/python2.4/site-packages/rdiff_backup/Main.py", line 259, in Main
take_action(rps)
File "/usr/lib/python2.4/site-packages/rdiff_backup/Main.py", line 229, in
take_action
elif action == "backup": Backup(rps[0], rps[1])
File "/usr/lib/python2.4/site-packages/rdiff_backup/Main.py", line 281, in Backup
backup.Mirror(rpin, rpout)
File "/usr/lib/python2.4/site-packages/rdiff_backup/backup.py", line 38, in Mirror
DestS.patch(dest_rpath, source_diffiter)
File "/usr/lib/python2.4/site-packages/rdiff_backup/backup.py", line 213, in patch
for diff in rorpiter.FillInIter(source_diffiter, dest_rpath):
File "/usr/lib/python2.4/site-packages/rdiff_backup/rorpiter.py", line 181, in
FillInIter
for rp in rpiter:
File "/usr/lib/python2.4/site-packages/rdiff_backup/backup.py", line 102, in
get_diffs
for dest_sig in dest_sigiter:
File "/usr/lib/python2.4/site-packages/rdiff_backup/backup.py", line 164, in
get_sigs
for src_rorp, dest_rorp in cls.CCPP:
File "/usr/lib/python2.4/site-packages/rdiff_backup/backup.py", line 298, in next
source_rorp, dest_rorp = self.iter.next()
File "/usr/lib/python2.4/site-packages/rdiff_backup/rorpiter.py", line 103, in
Collate2Iters
for relem1 in riter1:
File "/usr/lib/python2.4/site-packages/rdiff_backup/rorpiter.py", line 340, in
next
next_elem = self.iter.next()
File "/usr/lib/python2.4/site-packages/rdiff_backup/selection.py", line 132,
in Iterate_fast
try: rpath, val = diryield_stack[-1].next()
File "/usr/lib/python2.4/site-packages/rdiff_backup/selection.py", line 120,
in diryield
rpath.append, (filename,))
File "/usr/lib/python2.4/site-packages/rdiff_backup/robust.py", line 32, in
check_common_error
try: return function(*args)
File "/usr/lib/python2.4/site-packages/rdiff_backup/rpath.py", line 905, in append
return self.__class__(self.conn, self.base, self.index + (ext,))
File "/usr/lib/python2.4/site-packages/rdiff_backup/rpath.py", line 667, in
__init__
else: self.setdata()
File "/usr/lib/python2.4/site-packages/rdiff_backup/rpath.py", line 692, in
setdata
if self.lstat(): self.conn.rpath.setdata_local(self)
File "/usr/lib/python2.4/site-packages/rdiff_backup/rpath.py", line 1173, in
setdata_local
rpath.data['uname'] = user_group.uid2uname(rpath.data['uid'])
File "/usr/lib/python2.4/site-packages/rdiff_backup/user_group.py", line 49,
in uid2uname
try: uname = pwd.getpwuid(uid)[0]
OverflowError: signed integer is greater than maximum
Exception exceptions.TypeError: "'NoneType' object is not callable" in <bound
method GzipFile.__del__ of <gzip open file
'/var/backups/home/rdiff-backup-data/extended_attributes.2005-04-11T13:08:16+01:00.snapshot.gz',
mode 'wb' at 0x2a97c54210 0x2a97c4e440>> ignored
Exception exceptions.TypeError: "'NoneType' object is not callable" in <bound
method GzipFile.__del__ of <gzip open file
'/var/backups/home/rdiff-backup-data/file_statistics.2005-04-11T13:08:16+01:00.data.gz',
mode 'wb' at 0x2a97c54030 0x2a97c4e368>> ignored
Exception exceptions.TypeError: "'NoneType' object is not callable" in <bound
method GzipFile.__del__ of <gzip open file
'/var/backups/home/rdiff-backup-data/error_log.2005-04-11T13:08:16+01:00.data.gz',
mode 'wb' at 0x2a97c4acd8 0x2a97c4d2d8>> ignored
Exception exceptions.TypeError: "'NoneType' object is not callable" in <bound
method GzipFile.__del__ of <gzip open file
'/var/backups/home/rdiff-backup-data/mirror_metadata.2005-04-11T13:08:16+01:00.snapshot.gz',
mode 'wb' at 0x2a97c54198 0x2a97c4e3b0>> ignored
-------------------------------------------------
This mail sent through IMP: http://horde.org/imp/ |
|
| Back to top |
|
 |
Ben Escoto
Guest
|
| Posted: Mon Apr 11, 2005 10:05 am Post subject: Integer too small on AMD64 machine |
|
|
| Quote: | | Quote: | | Quote: | | Quote: | | Quote: | njbailey < at > elec.gla.ac.uk
wrote the following on Mon, 11 Apr 2005 13:51:28 +0100
|
|
|
|
|
| Quote: | I've been trying to use rdiff-backup to copy between two locally
mounted disk drives on a AMD64 box running Ubuntu. After creating
about 16BG of files in the destination directory, the command fails
with an integer overflow error.
Is this a 64-bit problem? I don't see anything in the FAQ or Wiki
which might help out.
I'm using python 2.4.1, librsync1 0.9.7 and rsync 2.6.3-2ubuntu1
packages.
|
...
| Quote: | try: uname = pwd.getpwuid(uid)[0]
|
I haven't seen this before, perhaps someone else has some insight.
The problem seems to be that rdiff-backup can't query the username of
some uid because the uid (as read from the file system) is too big.
Probably there is some file with a junk uid, but probably rdiff-backup
should handle this case without crashing. Is it traditional that uids
have to be less than 65536/32768 or something?
Or should this be a bug in python's getpwuid since very large uids
should be allowable.
--
Ben Escoto |
|
| Back to top |
|
|
dean gaudet
Guest
|
| Posted: Mon Apr 11, 2005 1:03 pm Post subject: Integer too small on AMD64 machine |
|
|
| Quote: | I haven't seen this before, perhaps someone else has some insight.
The problem seems to be that rdiff-backup can't query the username of
some uid because the uid (as read from the file system) is too big.
Probably there is some file with a junk uid, but probably rdiff-backup
should handle this case without crashing. Is it traditional that uids
have to be less than 65536/32768 or something?
Or should this be a bug in python's getpwuid since very large uids
should be allowable.
|
i'd look for a 16->64 bit sign extension which should have been 16->32
bit... or shouldn't have happenned at all. i don't know enough about
python typesize / sign extensions to figure it out myself. but the ubuntu
system probably has 32-bit uids, and you might be using 64-bit ints in
rdiff-backup.
16-bit uids are more traditional, but 32-bits is common in new distros.
-dean |
|
| Back to top |
|
|
dean gaudet
Guest
|
| Posted: Mon Apr 11, 2005 7:10 pm Post subject: Integer too small on AMD64 machine |
|
|
On Mon, 11 Apr 2005, dean gaudet wrote:
| Quote: | | Quote: | I haven't seen this before, perhaps someone else has some insight.
The problem seems to be that rdiff-backup can't query the username of
some uid because the uid (as read from the file system) is too big.
Probably there is some file with a junk uid, but probably rdiff-backup
should handle this case without crashing. Is it traditional that uids
have to be less than 65536/32768 or something?
Or should this be a bug in python's getpwuid since very large uids
should be allowable.
|
i'd look for a 16->64 bit sign extension which should have been 16->32
bit... or shouldn't have happenned at all. i don't know enough about
python typesize / sign extensions to figure it out myself. but the ubuntu
system probably has 32-bit uids, and you might be using 64-bit ints in
rdiff-backup.
16-bit uids are more traditional, but 32-bits is common in new distros.
|
hmm i wonder if you'd get away with (uid&0xffffffff) as a hack... but the
annoying thing is i wonder what you're supposed to do to figure out the
max uid... for most folks (uid&0xffff) is correct.
-dean |
|
| Back to top |
|
|
Nick Bailey
Guest
|
| Posted: Tue Apr 12, 2005 1:00 am Post subject: Integer too small on AMD64 machine |
|
|
Ben Escoto wrote:
| Quote: | | Quote: | try: uname = pwd.getpwuid(uid)[0]
|
I haven't seen this before, perhaps someone else has some insight.
The problem seems to be that rdiff-backup can't query the username of
some uid because the uid (as read from the file system) is too big.
|
It seems it has indeed bitten some other people. Google found this for me:
http://marc.free.net.ph/message/20050317.092005.79c1a132.en.html
| Quote: | Probably there is some file with a junk uid, but probably rdiff-backup
should handle this case without crashing. Is it traditional that uids
have to be less than 65536/32768 or something?
|
I thought they could be up to 4294967294? That's what the above link
implies. But I do seem to remember being told it was 65535 when I was an
undergrad because UIDs were unsigned shorts (but that was in the 80s :)
| Quote: | Or should this be a bug in python's getpwuid since very large uids
should be allowable.
|
There are no silly entries in /etc/passwd, and the biggest uid/gid is
65534. Of course, there might be silly uids/gids on files, for example
if someone did a tar xp or something. The users of the file server in
this case are a bunch of Postgrads, all with root access to their local
boxes, who just love stressing things (in a good way :)
I hacked in a print to find out the value of UID when that routine was
called. I'm calling it via sudo and my uid is 1000 on that box. It was
called first as 0, then 2001 (those are users of the fs). Then the stuff
about filesystem capabilities gets printed, then it's called with 2000,
2002, and, bingo, with uid=4294967294.
That gets called from setdata_local, so let's print out the rpath.data
dictionary (sorry, I know, I should learn how to use a debugger :)
{'uid': 4294967294, 'perms': 493, 'type': 'reg', 'ctime': 1097059287,
'devloc': 2065L, 'nlink': 1, 'gid': 4294967294, 'mtime': 1097059287,
'atime': 1097059287, 'inode': 22890417, 'size': 24620}
uid= 4294967294
Hmm.
nick < at > strauss:/exports/home$ sudo ls -lnR | awk '$3>3000 {print $0}'
./niall/demos/4k-2004/src/sw/pirro - unnamed:
-rwxr-xr-x 1 4294967294 4294967294 4509 2005-03-18 11:38
config_basic_example
-rw-r--r-- 1 4294967294 4294967294 20276 2005-03-18 11:38
config_basic_example.o
(approx 300 such examples).
So rdiff-backup comes out smelling of roses, with the slight caveat that
it would have been nice if there were some indication of which files
were causing the problems, or at least continued the backup so other
users weren't penalised. The offending user will be ritually taunted and
have his laboratory fridge rights temporarily revoked.
I guess he's been using Windoze, Apple, or some other strange
architecture which has tar'd up a ball with uid=gid=-1, then this has
been untar'd with the p flag or something like that. I hope this thread
will be of some use to others coming across the same thing.
Niall, can I chmod your entire home directory to niall:niall? Is that
OK? (think carefully before you answer...) Then we can all have
incremental backups.
Nick/ |
|
| Back to top |
|
|
Ben Escoto
Guest
|
| Posted: Tue Apr 12, 2005 6:36 pm Post subject: Integer too small on AMD64 machine |
|
|
Well I'm not sure there is a bug in python or not, but I think this
patch should fix your case.
--
Ben Escoto
--- rdiff_backup/user_group.py 18 Sep 2003 02:10:14 -0000 1.4
+++ rdiff_backup/user_group.py 13 Apr 2005 02:58:57 -0000 1.5
< at > < at > -47,7 +47,7 < at > < at >
try: return uid2uname_dict[uid]
except KeyError:
try: uname = pwd.getpwuid(uid)[0]
- except KeyError: uname = None
+ except (KeyError, OverflowError), e: uname = None
uid2uname_dict[uid] = uname
return uname
< at > < at > -56,7 +56,7 < at > < at >
try: return gid2gname_dict[gid]
except KeyError:
try: gname = grp.getgrgid(gid)[0]
- except KeyError: gname = None
+ except (KeyError, OverflowError), e: gname = None
gid2gname_dict[gid] = gname
return gname |
|
| Back to top |
|
|
|
FAQ
Search
Register
Log in to check your private messages
Log in
