Error 500 in "Admin -> Settings -> Full Text Search" (too many open files)


#1

Hi, I have this problem in my RhodeCode server since long time ago.

When I click on “Admin -> Settings -> Full Text Search” I get an Error 500.

I found “error: [Errno 24] Too many open files” in .rccontrol/community-1/community.log

my limits :
ulimit -n
555000

My server:
Ubuntu Linux 16.04 64 bits
8 cores
32 Gb RAM
RhodeCode Enterprise 4.10.5

Full Log:

2017-11-28 19:16:16.526 ERROR [rhodecode.config.middleware] error occurred handling this request for path: /_admin/settings/search
Traceback (most recent call last):
  File "/opt/rhodecode/store/34kjg3ids03zr9z8yza1f87mwrf0air8-python2.7-rhodecode-enterprise-ce-4.10.5/lib/python2.7/site-packages/rhodecode/lib/middleware/error_handling.py", line 73, in handle_request
    response = request.get_response(self.app)
  File "/opt/rhodecode/store/7cfsy6av786xrafqr9snmlgs2aflwwyf-python2.7-WebOb-1.7.3/lib/python2.7/site-packages/webob/request.py", line 1316, in send
    application, catch_exc_info=False)
  File "/opt/rhodecode/store/7cfsy6av786xrafqr9snmlgs2aflwwyf-python2.7-WebOb-1.7.3/lib/python2.7/site-packages/webob/request.py", line 1280, in call_application
    app_iter = application(self.environ, start_response)
  File "/opt/rhodecode/store/8glgrrbbbfhqc1s6nf9inbd456jjvv9w-python2.7-Paste-2.0.3/lib/python2.7/site-packages/paste/registry.py", line 379, in __call__
    app_iter = self.application(environ, start_response)
  File "/opt/rhodecode/store/pwsbjn91wzxbac3c8srr3nc3dibwh7jb-python2.7-Pylons-1.0.2.rhodecode-patch1/lib/python2.7/site-packages/pylons/wsgiapp.py", line 103, in __call__
    response = self.dispatch(controller, environ, start_response)
  File "/opt/rhodecode/store/pwsbjn91wzxbac3c8srr3nc3dibwh7jb-python2.7-Pylons-1.0.2.rhodecode-patch1/lib/python2.7/site-packages/pylons/wsgiapp.py", line 313, in dispatch
    return controller(environ, start_response)
  File "/opt/rhodecode/store/34kjg3ids03zr9z8yza1f87mwrf0air8-python2.7-rhodecode-enterprise-ce-4.10.5/lib/python2.7/site-packages/rhodecode/lib/base.py", line 576, in __call__
    return WSGIController.__call__(self, environ, start_response)
  File "/opt/rhodecode/store/pwsbjn91wzxbac3c8srr3nc3dibwh7jb-python2.7-Pylons-1.0.2.rhodecode-patch1/lib/python2.7/site-packages/pylons/controllers/core.py", line 214, in __call__
    response = self._dispatch_call()
  File "/opt/rhodecode/store/pwsbjn91wzxbac3c8srr3nc3dibwh7jb-python2.7-Pylons-1.0.2.rhodecode-patch1/lib/python2.7/site-packages/pylons/controllers/core.py", line 164, in _dispatch_call
    response = self._inspect_call(func)
  File "/opt/rhodecode/store/pwsbjn91wzxbac3c8srr3nc3dibwh7jb-python2.7-Pylons-1.0.2.rhodecode-patch1/lib/python2.7/site-packages/pylons/controllers/core.py", line 107, in _inspect_call
    result = self._perform_call(func, args)
  File "/opt/rhodecode/store/pwsbjn91wzxbac3c8srr3nc3dibwh7jb-python2.7-Pylons-1.0.2.rhodecode-patch1/lib/python2.7/site-packages/pylons/controllers/core.py", line 57, in _perform_call
    return func(**args)
  File "/opt/rhodecode/store/34kjg3ids03zr9z8yza1f87mwrf0air8-python2.7-rhodecode-enterprise-ce-4.10.5/lib/python2.7/site-packages/rhodecode/lib/auth.py", line 2188, in local_wrapper
    return wrapper(func, *args, **kwds)
  File "/opt/rhodecode/store/34kjg3ids03zr9z8yza1f87mwrf0air8-python2.7-rhodecode-enterprise-ce-4.10.5/lib/python2.7/site-packages/rhodecode/lib/auth.py", line 1569, in __wrapper
    return func(*fargs, **fkwargs)
  File "/opt/rhodecode/store/34kjg3ids03zr9z8yza1f87mwrf0air8-python2.7-rhodecode-enterprise-ce-4.10.5/lib/python2.7/site-packages/rhodecode/controllers/admin/settings.py", line 543, in settings_search
    c.statistics = searcher.statistics()
  File "/opt/rhodecode/store/34kjg3ids03zr9z8yza1f87mwrf0air8-python2.7-rhodecode-enterprise-ce-4.10.5/lib/python2.7/site-packages/rhodecode/lib/index/whoosh.py", line 168, in statistics
    'value': self.file_index.doc_count()},
  File "/opt/rhodecode/store/q3ipr1vmdyaya7l32g2kxbx8rb1cp9n0-python2.7-Whoosh-2.7.4/lib/python2.7/site-packages/whoosh/index.py", line 304, in doc_count
    r = self.reader()
  File "/opt/rhodecode/store/q3ipr1vmdyaya7l32g2kxbx8rb1cp9n0-python2.7-Whoosh-2.7.4/lib/python2.7/site-packages/whoosh/index.py", line 548, in reader
    info.generation, reuse=reuse)
  File "/opt/rhodecode/store/q3ipr1vmdyaya7l32g2kxbx8rb1cp9n0-python2.7-Whoosh-2.7.4/lib/python2.7/site-packages/whoosh/index.py", line 535, in _reader
    readers = [segreader(segment) for segment in segments]
  File "/opt/rhodecode/store/q3ipr1vmdyaya7l32g2kxbx8rb1cp9n0-python2.7-Whoosh-2.7.4/lib/python2.7/site-packages/whoosh/index.py", line 524, in segreader
    generation=generation)
  File "/opt/rhodecode/store/q3ipr1vmdyaya7l32g2kxbx8rb1cp9n0-python2.7-Whoosh-2.7.4/lib/python2.7/site-packages/whoosh/reading.py", line 610, in __init__
    files = segment.open_compound_file(storage)
  File "/opt/rhodecode/store/q3ipr1vmdyaya7l32g2kxbx8rb1cp9n0-python2.7-Whoosh-2.7.4/lib/python2.7/site-packages/whoosh/codec/base.py", line 570, in open_compound_file
    return CompoundStorage(dbfile, use_mmap=storage.supports_mmap)
  File "/opt/rhodecode/store/q3ipr1vmdyaya7l32g2kxbx8rb1cp9n0-python2.7-Whoosh-2.7.4/lib/python2.7/site-packages/whoosh/filedb/compound.py", line 75, in __init__
    self._source = mmap.mmap(fileno, 0, access=mmap.ACCESS_READ)
error: [Errno 24] Too many open files
2017-11-28 19:16:16.769 INFO  [rhodecode.lib.middleware.request_wrapper] IP: 172.168.1.6 Request to /_admin/settings/search time: 14.860s

Any Idea?


#2

Hi,

Yes it means the searcher is too fragmented. Please run the indexer with --optimize flag. We’d also recommend using this method every 5-10th index build.