SearchFAQMemberlist Log in
Reply to topic Page 1 of 1
Error: Too many open files
Author Message
Post Error: Too many open files 
Hi,

I've been using rdiff-backup for a month, but since a few days,
it fails with that error:

<<<<<
Traceback (most recent call last):
File "/usr/bin/rdiff-backup", line 23, in ?
File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 259, in Main
File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 229, in take_action
File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 278, in Backup
File "/usr/lib/python2.3/site-packages/rdiff_backup/backup.py", line 51, in Mirror_and_increment
File "/usr/lib/python2.3/site-packages/rdiff_backup/backup.py", line 226, in patch_and_increment
File "/usr/lib/python2.3/site-packages/rdiff_backup/rorpiter.py", line 279, in __call__
File "/usr/lib/python2.3/site-packages/rdiff_backup/backup.py", line 607, in fast_process
File "/usr/lib/python2.3/site-packages/rdiff_backup/backup.py", line 593, in inc_with_checking
File "/usr/lib/python2.3/site-packages/rdiff_backup/increment.py", line 44, in Increment
File "/usr/lib/python2.3/site-packages/rdiff_backup/increment.py", line 73, in makesnapshot
File "/usr/lib/python2.3/site-packages/rdiff_backup/rpath.py", line 213, in copy_with_attribs
File "/usr/lib/python2.3/site-packages/rdiff_backup/rpath.py", line 96, in copy
File "/usr/lib/python2.3/site-packages/rdiff_backup/rpath.py", line 118, in copy_reg_file
File "/usr/lib/python2.3/site-packages/rdiff_backup/rpath.py", line 946, in write_from_fileobj
File "/usr/lib/python2.3/site-packages/rdiff_backup/rpath.py", line 925, in open
File "/usr/lib/python2.3/gzip.py", line 94, in __init__
IOError: [Errno 24] Too many open files: '/mnt/usb/rdiff-backup-data/increments/var/spool/news/mess
age.id/065/<slrncrqdpr.qcn.pontoize*NOSPAM* < at > choam.unix.efrei.fr>.2005-01-13T22:08:59+01:00.snapshot
.gz'
Exception exceptions.TypeError: "'NoneType' object is not callable" in <bound method GzipFile.__del
__ of <gzip open file '/mnt/usb/rdiff-backup-data/file_statistics.2005-01-24T23:16:32+01:00.data.gz
', mode 'wb' at 0x40491060 0x40841ccc>> ignored
Exception exceptions.TypeError: "'NoneType' object is not callable" in <bound method GzipFile.__del
__ of <gzip open file '/mnt/usb/rdiff-backup-data/error_log.2005-01-24T23:16:32+01:00.data.gz', mod
e 'wb' at 0x4048d4a0 0x409a276c>> ignored
Exception exceptions.TypeError: "'NoneType' object is not callable" in <bound method GzipFile.__del
__ of <gzip open file '/mnt/usb/rdiff-backup-data/mirror_metadata.2005-01-24T23:16:32+01:00.snapsho
t.gz', mode 'wb' at 0x404914e0 0x4084140c>> ignored


The concerned directory is the news spool, managed by Leafnode.
« /var/spool/news/message.id/065/ » currently contains only about
140 files.

I've monitored opened files during rdiff-backup run, with lsof.
These are the counts: (interval = 5 seconds)
51 52 53 52 92 83 117 186 364 522 717 876 1040 0 0 0 0
^
crash

Just before the crash, I can see about one thousand files opened
on the backup partition, in various directories:

<<<<<
/mnt/usb/var/spool/news/message.id/051/<cncrma$1h9$1 < at > sunnews.cern.ch> (deleted)
/mnt/usb/var/spool/news/message.id/051/<cr65og$hoa$1 < at > news.tiscali.fr> (deleted)
/mnt/usb/var/spool/news/message.id/051/<hmtb20lmiqoln777v15d1h2633eit6eg0m < at > 4ax.com> (deleted)
/mnt/usb/var/spool/news/message.id/051/<kLyld.61065$Kd1.1553819 < at > weber.videotron.net> (deleted)
/mnt/usb/var/spool/news/message.id/051/<v146mv47d67p7jo8akmf4lqao61uo376vm < at > 4ax.com> (deleted)
/mnt/usb/var/spool/news/message.id/052/<1gn7any.1jp107g5mv8akN%------ < at > -----.com> (deleted)
/mnt/usb/var/spool/news/message.id/052/<20041017161223.205e1505 < at > norbert.inapg.inra.fr> (deleted)
/mnt/usb/var/spool/news/message.id/052/<20041107144741.3015f101 < at > norbert.inapg.inra.fr> (deleted)
/mnt/usb/var/spool/news/message.id/052/<20041111124713.312293d1 < at > norbert.inapg.inra.fr> (deleted)
/mnt/usb/var/spool/news/message.id/052/<2us1hcF2d37c3U1 < at > uni-berlin.de> (deleted)
/mnt/usb/var/spool/news/message.id/052/<2v72iiF2a1f3lU1 < at > uni-berlin.de> (deleted)
[...]
/mnt/usb/var/spool/news/message.id/056/<ckiu8c$1bq$1 < at > sunnews.cern.ch> (deleted)
/mnt/usb/var/spool/news/message.id/056/<clqd5o$ng6$1 < at > news.tiscali.fr> (deleted)
/mnt/usb/var/spool/news/message.id/056/<cnfhuk$a56$1 < at > sunnews.cern.ch> (deleted)
/mnt/usb/var/spool/news/message.id/056/<cnvnh0$8fh$1 < at > news.tiscali.fr> (deleted)
/mnt/usb/var/spool/news/message.id/056/<j5s5m0hah7micvfabl53n1l5qp749679td < at > 4ax.com> (deleted)
/mnt/usb/var/spool/news/message.id/056/<klgi82-vfh.ln1 < at > news.labourdette.homelinux.com> (deleted)
/mnt/usb/var/spool/news/message.id/057/<1gkmrd9.qvpuy51qjfhdlN%francois < at > despammed.com> (deleted)
/mnt/usb/var/spool/news/message.id/057/<2s04miF1f2i1oU1 < at > uni-berlin.de> (deleted)
/mnt/usb/var/spool/news/message.id/057/<2ul810F2ca8mjU1 < at > uni-berlin.de> (deleted)
/mnt/usb/var/spool/news/message.id/057/<2us22bF2d0h3pU1 < at > uni-berlin.de> (deleted)


I'm using rdiff-backup 0.13.4 with Python 2.3.
My ulimit for opened files is set at 1024.


Is that a known problem of the development version? Should I downgrade
to 0.12.7? (but it seems that backup formats are incompatible)


Cheers.

--
Jérémy JUST <jeremy_just < at > netcourrier.com>

Post Error: Too many open files 
On Thu, 27 Jan 2005 16:31:41 +0100
Jérémy JUST <jeremy_just < at > netcourrier.com> wrote:

IOError: [Errno 24] Too many open files:
[...]
I'm using rdiff-backup 0.13.4 with Python 2.3.
My ulimit for opened files is set at 1024.

As a workaround, I have set the ulimit to 100000.
The backup is OK, but the number of open files reached 27420!!

--
Jérémy JUST <jeremy_just < at > netcourrier.com>

Post Error: Too many open files 
Jérémy JUST <jeremy_just < at > netcourrier.com> writes:
I've monitored opened files during rdiff-backup run, with lsof.
These are the counts: (interval = 5 seconds)
51 52 53 52 92 83 117 186 364 522 717 876 1040 0 0 0 0
^
crash

Just before the crash, I can see about one thousand files opened
on the backup partition, in various directories:
[...]
I'm using rdiff-backup 0.13.4 with Python 2.3.
My ulimit for opened files is set at 1024.


Is that a known problem of the development version?

I guess it is as of now...

... Should I downgrade to 0.12.7? (but it seems that backup formats are incompatible)

If it were me, I'd raise the ulimit.
0.12.7 is indeed incompatible, and kind of old besides,
and might have the same problem anyway.

Display posts from previous:
Reply to topic Page 1 of 1
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
  


Magic SEO URL for phpBB