Rhodecode worker VS VCSServer worker

Since rhodecode installation (community in my case) and VCSServer use Gunicorn WSGI server we read this comment on both configuration files:

Sets the number of process workers. You must set `instance_id = *`
when this option is set to more than one worker, recommended
value is (2 * NUMBER_OF_CPUS + 1), eg 2CPU = 5 workers

In old version (and documentation https://docs.rhodecode.com/RhodeCode-Enterprise/admin/vcs-server.html#vcs-server) we have this comment:

number of worker threads, this should be set based on a formula threadpool=N*6
where N is number of RhodeCode Enterprise workers, eg. running 2 instances
8 gunicorn workers each would be 2 * 8 * 6 = 96, threadpool_size = 96

In our installation we have 2 CPU, that done 5 rhodecode workers and 5 vscserver workers, is it the best configuration?

We have some performances issues with mercurial and I review the configuration.



Sorry the threadpool_size was a valid option for RhodeCode < 4.5.0, we’ll update the docs to change the configuration docs.

In the case you desribed 5 workers for each vcsserver and rhodecode is fine. There are more tunning options available, can you describe the problems you’re having ?

Every times we create a pull request, change the status of a pull request or close a pull request, we have a freeze time of ~5 minutes for every users. Rhodecode doesn’t respond.

According to the log, I think that is a problem with the mySQL database, we have many time this error:

2017-04-27 18:46:12.266 ERROR [rhodecode.lib.middleware.error_handling] (OperationalError) (1205, 'Lock wait timeout exceeded; try restarting transaction') 'UPDATE users SET user_data=%s WHERE users.user_id = %s' ('{"language": "en", "last_activity": 1493318721.682601, "notification_status": true}', 5L)
Traceback (most recent call last):
  File "/opt/rhodecode/store/cgg0ln7k2xdn96xi5kp8phzl5320aay9-python2.7-rhodecode-enterprise-ce-4.5.2/lib/python2.7/site-packages/rhodecode/lib/middleware/error_handling.py", line 72, in handle_request
    response = request.get_response(self.app)
  File "/opt/rhodecode/store/cbcb7xvl643c22r0k153ggyhvzkb87z5-python2.7-WebOb-1.3.1/lib/python2.7/site-packages/webob/request.py", line 1320, in send
    application, catch_exc_info=False)
  File "/opt/rhodecode/store/cbcb7xvl643c22r0k153ggyhvzkb87z5-python2.7-WebOb-1.3.1/lib/python2.7/site-packages/webob/request.py", line 1284, in call_application
    app_iter = application(self.environ, start_response)
  File "/opt/rhodecode/store/52v7z5kbn6fnbwjxcwcn9z8wdd0w66vh-python2.7-Paste-2.0.2/lib/python2.7/site-packages/paste/registry.py", line 379, in __call__
    app_iter = self.application(environ, start_response)
  File "/opt/rhodecode/store/pkp5jr8c2gk53plaq37rxpqrcdxgs1g5-python2.7-Pylons-1.0.1-patch1/lib/python2.7/site-packages/pylons/wsgiapp.py", line 103, in __call__
    response = self.dispatch(controller, environ, start_response)
  File "/opt/rhodecode/store/pkp5jr8c2gk53plaq37rxpqrcdxgs1g5-python2.7-Pylons-1.0.1-patch1/lib/python2.7/site-packages/pylons/wsgiapp.py", line 313, in dispatch
    return controller(environ, start_response)
  File "/opt/rhodecode/store/cgg0ln7k2xdn96xi5kp8phzl5320aay9-python2.7-rhodecode-enterprise-ce-4.5.2/lib/python2.7/site-packages/rhodecode/lib/base.py", line 514, in __call__
    return WSGIController.__call__(self, environ, start_response)
  File "/opt/rhodecode/store/pkp5jr8c2gk53plaq37rxpqrcdxgs1g5-python2.7-Pylons-1.0.1-patch1/lib/python2.7/site-packages/pylons/controllers/core.py", line 214, in __call__
    response = self._dispatch_call()
  File "/opt/rhodecode/store/pkp5jr8c2gk53plaq37rxpqrcdxgs1g5-python2.7-Pylons-1.0.1-patch1/lib/python2.7/site-packages/pylons/controllers/core.py", line 164, in _dispatch_call
    response = self._inspect_call(func)
  File "/opt/rhodecode/store/pkp5jr8c2gk53plaq37rxpqrcdxgs1g5-python2.7-Pylons-1.0.1-patch1/lib/python2.7/site-packages/pylons/controllers/core.py", line 107, in _inspect_call
    result = self._perform_call(func, args)
  File "/opt/rhodecode/store/pkp5jr8c2gk53plaq37rxpqrcdxgs1g5-python2.7-Pylons-1.0.1-patch1/lib/python2.7/site-packages/pylons/controllers/core.py", line 57, in _perform_call
    return func(**args)
  File "/opt/rhodecode/store/cgg0ln7k2xdn96xi5kp8phzl5320aay9-python2.7-rhodecode-enterprise-ce-4.5.2/lib/python2.7/site-packages/rhodecode/lib/auth.py", line 1904, in local_wrapper
    return wrapper(func, *args, **kwds)
  File "/opt/rhodecode/store/cgg0ln7k2xdn96xi5kp8phzl5320aay9-python2.7-rhodecode-enterprise-ce-4.5.2/lib/python2.7/site-packages/rhodecode/lib/auth.py", line 1226, in __wrapper
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 788, in commit
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 384, in commit
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 364, in _prepare_impl
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1985, in flush
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2103, in _flush
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2067, in _flush
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 372, in execute
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 526, in execute
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 60, in save_obj
    mapper, table, update)
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 518, in _emit_update_statements
    execute(statement, params)
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 841, in execute
    return meth(self, multiparams, params)
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 322, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 938, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1070, in _execute_context
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1271, in _handle_dbapi_exception
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1063, in _execute_context
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 442, in do_execute
    cursor.execute(statement, parameters)
  File "/opt/rhodecode/store/cgqzxbpg5v6bzw3xrk6axhzqsqqq88dc-python2.7-MySQL-python-1.2.5/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
    self.errorhandler(self, exc, value)
  File "/opt/rhodecode/store/cgqzxbpg5v6bzw3xrk6axhzqsqqq88dc-python2.7-MySQL-python-1.2.5/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
OperationalError: (OperationalError) (1205, 'Lock wait timeout exceeded; try restarting transaction') 'UPDATE users SET user_data=%s WHERE users.user_id = %s' ('{"language": "en", "last_activity": 1493318721.682601, "notification_status": true}', 5L)

I don’t kwon why, but database is locked.

We have few questions:

Is this database shared with other system?
Did you checked mysql logs ?
Also do you use gevent as worker type?

Please try upgrade to 4.7.2, maybe it will fix it ?

The database is local and dedicate to rhodecode only.
No, we don’t use gevent. We use gunicorn with sync worker class.

There are nothing to see in mysql logs, but we use an old version of mySQL (5.1), is that the problem? I’m connecting to mySQL console, when rhodecode freeze there are no activity to show with query SHOW FULL PROCESSLIST; or SHOW OPEN TABLES;.

We planed rhodecode update, but we doesn’t good experience with update.

Please also check if ALL your tables in Mysql are using INNODB.

Upgrade should be very easy did you encoutered any other problems ?

No, there are two tables myISAM, beaker_cache and db_session.

The last time we try to update to 4.6.0, nobody could push on our most biggest repository (but every others operations works well). When we update from 4.4.2 to 4.5.2, we have a setting have change on the rhodecode.ini, someone from your slack channel help us with that. I thing we need to keep rhodecode updated, but it’s now more complicated to convain whole people.

Please try to change those tables to InnoDB, also do you use celery ? if yes try to disable it and check again.

You can run many instances in pararell so maybe do a snapshot of database and run upgrade on the copy to 4.7.2 and test there ?

No, we don’t use celery.

Could we delete everything on beaker_cache (~3M rows) and db_session (~145K rows) tables?

Yes it’s ok to delete those (just restart rhodecode after)

In 4.7 + we added interface to clean obsolete sessions, and also improved large push support I enoucrage you try that.

I have convert table myISAM to innoDB and we have the same problem.

We will try 3 things (this weekend and monday):

  1. Move repository on better disk (I’m pretty sur that will not fix this problem, but it’s a good idea for better performance)
  2. Update mySQL to 5.7 (because working with old tool from 2008 it’s always a bad idea)
  3. Update rhodecode


Please let us know if we can help in any case, also maybe you can share the use case, the team size, what is your biggest repository ? (also can share privatly if that’s an option)

We actually have one of the bigger customers running RhodeCode with almost 2000 devs, using mysql, and having repos of ~10GB in size commonly.
There are things we can still optimize if there’s need. But faster storage would always help.