Rhodecode worker VS VCSServer worker

harcher81 · April 27, 2017, 2:14pm

Hi,
Since rhodecode installation (community in my case) and VCSServer use Gunicorn WSGI server we read this comment on both configuration files:

Sets the number of process workers. You must set `instance_id = *`
when this option is set to more than one worker, recommended
value is (2 * NUMBER_OF_CPUS + 1), eg 2CPU = 5 workers

In old version (and documentation https://docs.rhodecode.com/RhodeCode-Enterprise/admin/vcs-server.html#vcs-server) we have this comment:

number of worker threads, this should be set based on a formula threadpool=N*6
where N is number of RhodeCode Enterprise workers, eg. running 2 instances
8 gunicorn workers each would be 2 * 8 * 6 = 96, threadpool_size = 96

In our installation we have 2 CPU, that done 5 rhodecode workers and 5 vscserver workers, is it the best configuration?

We have some performances issues with mercurial and I review the configuration.

Thanks

rhodecode-support · April 27, 2017, 6:30pm

Hi,

Sorry the threadpool_size was a valid option for RhodeCode < 4.5.0, we’ll update the docs to change the configuration docs.

In the case you desribed 5 workers for each vcsserver and rhodecode is fine. There are more tunning options available, can you describe the problems you’re having ?

harcher81 · April 28, 2017, 1:23pm

Hi,
Every times we create a pull request, change the status of a pull request or close a pull request, we have a freeze time of ~5 minutes for every users. Rhodecode doesn’t respond.

According to the log, I think that is a problem with the mySQL database, we have many time this error:

2017-04-27 18:46:12.266 ERROR [rhodecode.lib.middleware.error_handling] (OperationalError) (1205, 'Lock wait timeout exceeded; try restarting transaction') 'UPDATE users SET user_data=%s WHERE users.user_id = %s' ('{"language": "en", "last_activity": 1493318721.682601, "notification_status": true}', 5L)
Traceback (most recent call last):
  File "/opt/rhodecode/store/cgg0ln7k2xdn96xi5kp8phzl5320aay9-python2.7-rhodecode-enterprise-ce-4.5.2/lib/python2.7/site-packages/rhodecode/lib/middleware/error_handling.py", line 72, in handle_request
    response = request.get_response(self.app)
  File "/opt/rhodecode/store/cbcb7xvl643c22r0k153ggyhvzkb87z5-python2.7-WebOb-1.3.1/lib/python2.7/site-packages/webob/request.py", line 1320, in send
    application, catch_exc_info=False)
  File "/opt/rhodecode/store/cbcb7xvl643c22r0k153ggyhvzkb87z5-python2.7-WebOb-1.3.1/lib/python2.7/site-packages/webob/request.py", line 1284, in call_application
    app_iter = application(self.environ, start_response)
  File "/opt/rhodecode/store/52v7z5kbn6fnbwjxcwcn9z8wdd0w66vh-python2.7-Paste-2.0.2/lib/python2.7/site-packages/paste/registry.py", line 379, in __call__
    app_iter = self.application(environ, start_response)
  File "/opt/rhodecode/store/pkp5jr8c2gk53plaq37rxpqrcdxgs1g5-python2.7-Pylons-1.0.1-patch1/lib/python2.7/site-packages/pylons/wsgiapp.py", line 103, in __call__
    response = self.dispatch(controller, environ, start_response)
  File "/opt/rhodecode/store/pkp5jr8c2gk53plaq37rxpqrcdxgs1g5-python2.7-Pylons-1.0.1-patch1/lib/python2.7/site-packages/pylons/wsgiapp.py", line 313, in dispatch
    return controller(environ, start_response)
  File "/opt/rhodecode/store/cgg0ln7k2xdn96xi5kp8phzl5320aay9-python2.7-rhodecode-enterprise-ce-4.5.2/lib/python2.7/site-packages/rhodecode/lib/base.py", line 514, in __call__
    return WSGIController.__call__(self, environ, start_response)
  File "/opt/rhodecode/store/pkp5jr8c2gk53plaq37rxpqrcdxgs1g5-python2.7-Pylons-1.0.1-patch1/lib/python2.7/site-packages/pylons/controllers/core.py", line 214, in __call__
    response = self._dispatch_call()
  File "/opt/rhodecode/store/pkp5jr8c2gk53plaq37rxpqrcdxgs1g5-python2.7-Pylons-1.0.1-patch1/lib/python2.7/site-packages/pylons/controllers/core.py", line 164, in _dispatch_call
    response = self._inspect_call(func)
  File "/opt/rhodecode/store/pkp5jr8c2gk53plaq37rxpqrcdxgs1g5-python2.7-Pylons-1.0.1-patch1/lib/python2.7/site-packages/pylons/controllers/core.py", line 107, in _inspect_call
    result = self._perform_call(func, args)
  File "/opt/rhodecode/store/pkp5jr8c2gk53plaq37rxpqrcdxgs1g5-python2.7-Pylons-1.0.1-patch1/lib/python2.7/site-packages/pylons/controllers/core.py", line 57, in _perform_call
    return func(**args)
  File "/opt/rhodecode/store/cgg0ln7k2xdn96xi5kp8phzl5320aay9-python2.7-rhodecode-enterprise-ce-4.5.2/lib/python2.7/site-packages/rhodecode/lib/auth.py", line 1904, in local_wrapper
    return wrapper(func, *args, **kwds)
  File "/opt/rhodecode/store/cgg0ln7k2xdn96xi5kp8phzl5320aay9-python2.7-rhodecode-enterprise-ce-4.5.2/lib/python2.7/site-packages/rhodecode/lib/auth.py", line 1226, in __wrapper
    Session().commit()
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 788, in commit
    self.transaction.commit()
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 384, in commit
    self._prepare_impl()
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 364, in _prepare_impl
    self.session.flush()
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1985, in flush
    self._flush(objects)
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2103, in _flush
    transaction.rollback(_capture_exception=True)
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2067, in _flush
    flush_context.execute()
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 372, in execute
    rec.execute(self)
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 526, in execute
    uow
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 60, in save_obj
    mapper, table, update)
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 518, in _emit_update_statements
    execute(statement, params)
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 841, in execute
    return meth(self, multiparams, params)
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 322, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 938, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1070, in _execute_context
    context)
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1271, in _handle_dbapi_exception
    exc_info
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1063, in _execute_context
    context)
  File "/opt/rhodecode/store/7whz4rq77cg2az9ywlay6myk72rcpizg-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 442, in do_execute
    cursor.execute(statement, parameters)
  File "/opt/rhodecode/store/cgqzxbpg5v6bzw3xrk6axhzqsqqq88dc-python2.7-MySQL-python-1.2.5/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
    self.errorhandler(self, exc, value)
  File "/opt/rhodecode/store/cgqzxbpg5v6bzw3xrk6axhzqsqqq88dc-python2.7-MySQL-python-1.2.5/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
OperationalError: (OperationalError) (1205, 'Lock wait timeout exceeded; try restarting transaction') 'UPDATE users SET user_data=%s WHERE users.user_id = %s' ('{"language": "en", "last_activity": 1493318721.682601, "notification_status": true}', 5L)

I don’t kwon why, but database is locked.

rhodecode-support · April 28, 2017, 1:38pm

We have few questions:

Is this database shared with other system?
Did you checked mysql logs ?
Also do you use gevent as worker type?

Please try upgrade to 4.7.2, maybe it will fix it ?

harcher81 · April 28, 2017, 2:12pm

The database is local and dedicate to rhodecode only.
No, we don’t use gevent. We use gunicorn with sync worker class.

There are nothing to see in mysql logs, but we use an old version of mySQL (5.1), is that the problem? I’m connecting to mySQL console, when rhodecode freeze there are no activity to show with query SHOW FULL PROCESSLIST; or SHOW OPEN TABLES;.

We planed rhodecode update, but we doesn’t good experience with update.

rhodecode-support · April 28, 2017, 2:17pm

Please also check if ALL your tables in Mysql are using INNODB.

Upgrade should be very easy did you encoutered any other problems ?

harcher81 · April 28, 2017, 2:47pm

No, there are two tables myISAM, beaker_cache and db_session.

The last time we try to update to 4.6.0, nobody could push on our most biggest repository (but every others operations works well). When we update from 4.4.2 to 4.5.2, we have a setting have change on the rhodecode.ini, someone from your slack channel help us with that. I thing we need to keep rhodecode updated, but it’s now more complicated to convain whole people.

rhodecode-support · April 28, 2017, 2:56pm

Please try to change those tables to InnoDB, also do you use celery ? if yes try to disable it and check again.

You can run many instances in pararell so maybe do a snapshot of database and run upgrade on the copy to 4.7.2 and test there ?

harcher81 · April 28, 2017, 3:56pm

No, we don’t use celery.

Could we delete everything on beaker_cache (~3M rows) and db_session (~145K rows) tables?

rhodecode-support · April 28, 2017, 5:03pm

Yes it’s ok to delete those (just restart rhodecode after)

In 4.7 + we added interface to clean obsolete sessions, and also improved large push support I enoucrage you try that.

harcher81 · April 28, 2017, 8:09pm

I have convert table myISAM to innoDB and we have the same problem.

We will try 3 things (this weekend and monday):

Move repository on better disk (I’m pretty sur that will not fix this problem, but it’s a good idea for better performance)
Update mySQL to 5.7 (because working with old tool from 2008 it’s always a bad idea)
Update rhodecode

rhodecode-support · April 28, 2017, 9:09pm

HI,

Please let us know if we can help in any case, also maybe you can share the use case, the team size, what is your biggest repository ? (also can share privatly if that’s an option)

We actually have one of the bigger customers running RhodeCode with almost 2000 devs, using mysql, and having repos of ~10GB in size commonly.
There are things we can still optimize if there’s need. But faster storage would always help.