New High Performance HTTP backend

1

We identified that pyro4 doesn’t work well with large repos, and big teams. We worked for some time to implemented a new better backend using HTTP. It works in both CE and EE editions.
It’s recommended to switch to get a better performance and scalability for RhodeCode. It’s available in release 4.3+

If you’re using version 4.5.0+ this is enabled by default and no need to change anything.

To switch to this backend you need to do few things.

  • update to latest version minimum 4.4.0

  • on enterprise/community you need to enable http backend. Please find such options (if not present add them) inside rhodecode.ini:

    vcs.server.protocol = http
vcs.scm_app_implementation = rhodecode.lib.middleware.utils.scm_app_http
vcs.hooks.protocol = http

  • Backup your old configuration from /home/<user>/.rccontrol/vcsserver-1/vcsserver.ini

  • Copy the content of https://code.rhodecode.com/rhodecode-vcsserver/raw/default/configs/production_http.ini
    into a file called vcsserver.ini and store it inside /home/<user>/.rccontrol/vcsserver-1 directory.

    curl https://code.rhodecode.com/rhodecode-vcsserver/raw/default/configs/production_http.ini > vcsserver.ini

  • Adjust the port inside vcsserver.ini to a different one than in the one in old vcsserver.ini, for example set it to: 10008

  • Change the rhodecode.ini file inside ~.rccontrol/enterprise-N/rhodecode.ini to connect vcsserver to the new port

    vcs.server = localhost:10008

  • Unlock supervisord auto-config overwrite. Open a file ~/.rccontrol.ini and set for vcsserver-1 self_managed_supervisor=True. Save the file:

    [instance:enterprise-1]
start_at_boot = True
[instance:vcsserver-1]
start_at_boot = True
self_managed_supervisor = True

    This would allow to change supervisord.ini and not loose the changes

  • Change inside .rccontrol/supervisor/supervisord.ini command to start vcsserver from:

command = /home/USER/.rccontrol/vcsserver-1/profile/bin/vcsserver --config=/home/USER/.rccontrol/vcsserver-1/vcsserver.ini

- You should also change the vcsserver supervisord group `_port` variables to reflect the new port (eg 10008)
- into a new command that doesn't use vcsserver anymore, and replaces it with Gunicorn:

command = /home/USER/.rccontrol/vcsserver-1/profile/bin/gunicorn --error-logfile=- --paster=/home/USER/.rccontrol/vcsserver-1/vcsserver.ini


You must restart whole supervisord to get new changes to be applied. Please run:

rccontrol self-stop --force
rccontrol self-init


Now it should use http with streaming support and handle big repositories, as well as being faster. 

You can also now use the async workers which should give you great concurrency, and allow
having much more concurrent pulls/pushes without need of additional CPUs

Simple change inside ~.rccontrol/enterprise-N/rhodecode.ini

worker_class = gevent