VCS Server is extremly slow

Support
1

Hi there,

We recently migrated our few HG repositories to RhodeCode community server. It’s very small installation: 5 repositories and 2 registered users.

After repositories migration we also change VCS roots in our TeamCity build server.

After that we were observing that RhodeCode is almost unusable: webpage was unresponsive (sometimes didn’t load at all), push/pull requests sometimes took tens of seconds…

By checking machine health, I found out that VCS Server process almost constantly hands one CPU. It also ate all memory and swap. (see screenshot attached)

Hardware configuration: Intel Xeon @ 2 GHz, 2 GB RAM, 2 GB swap, SAS disk (all virtualized), OS Ubuntu Server 16.04
RhodeCode configuration:

  • we use sqllite (tried postgre, same issue)
  • we use anonymous access to repositories (it’s inside of VPN network)
  • 5 repositories (2 are quite big - 850 and 300 megabytes)

TeamCity server checks 4 of this repositories, every 60 seconds.

Is that normal behavior? Can I do anything about it? Can I send some logs or something?

2

Hi,

It’s not normal. I think it’s easily fixable by changing the workers configuration. Please check inside rhodecode.ini how many workers you have ? I’d recommend using 5-6 workers, for you setup.

Then you have to increase threadpool on vcsserver config based on the formula specified there.

ps, on a side note we’d be shipping much more efficient backend in 4.3 release that would additionally speed things up. Until then the configuration i posted should help.

Please let us know if it does.

3

Hi,

Thanks for quick answer.

However, I was not very successful with setting up configs you mentioned. I set number of workers to 5 and thread pool to 240 (I was experimenting with this number, but seems it does not affect my problem).

See last screenshot from htop. It ate all available memory and almost both CPU (see ‘Load average’ part).

I also noticed that supervisord takes a lot of CPU…

Is there anything else I can try?

4

We identified that pyro4 doesn’t work well with large repos, thus we implemented a new better backend using HTTP. It works in both CE and EE editions.

To switch to this backend you need to do few things.

  • on enterprise/community you need to enable http backend. Please find such options (if not present add them) inside rhodecode.ini:

    vcs.server.protocol = http
vcs.scm_app_implementation = rhodecode.lib.middleware.utils.scm_app_http
vcs.hooks.protocol = http

  • Copy the content of https://code.rhodecode.com/rhodecode-vcsserver/raw/default/configs/production_http.ini
    into a file called vcsserver_http.ini and store it inside /home/<user>/.rccontrol/vcsserver-1

    curl https://code.rhodecode.com/rhodecode-vcsserver/raw/default/configs/production_http.ini > vcsserver_http.ini

  • Adjust the port inside vcsserver_http.ini to the one that is in old vcsserver.ini

  • Unlock supervisord auto-config overwirte for vcsserver-1 by changing file ~/.rccontrol.ini
    Set:

    [instance:enterprise-1]
start_at_boot = True
[instance:vcsserver-1]
start_at_boot = True
self_managed_supervisor = True

    This would allow to change supervisord.ini and not loose the changes

  • you need to change in .rccontrol/supervisor/supervisord.ini a call how to start vcsserver:
    from

    command = /home/<user>/.rccontrol/vcsserver-1/profile/bin/vcsserver --config=/home/<user>/.rccontrol/vcsserver-1/vcsserver.ini

    to

    command = /home/<user>/.rccontrol/vcsserver-1/profile/bin/gunicorn --error-logfile=- --paster=/home/<user>/.rccontrol/vcsserver-1/vcsserver_http.ini

Restart the enterprise and vcsserver by running rccontrol restart '*'

Now it should use http with streaming support and handle big repositories, as well as beeing faster.

5

Hi,

I did all steps, but in supervisord.ini config for VCS server is vcsserver_http.ini (I suppose you meant that).

What I get now is HTTP Error: 500 (Internal Server Error).

I can see from logs that community service is not able to connect to VCS server. And in vcsserver.log I see that is listening on port 9900, instead of 10001 (defined in config file).

Anyway, I’m sending config files and logs (cleared and then machine was rebooted). Please find it here: https://1drv.ms/u/s!AhKm-nEB-YrthcNlLEd-BLJHh18AXA

Am I missing something?

6

make sure the port of vcsserver matches the one from rhodecode.ini inside the vcsserver. config variables.

7

rhodecode.ini contains following line:

vcs.server = 127.0.0.1:10001

and vcsserver_http.ini this lines:

[server:main]
## COMMON ##
host = 127.0.0.1
port = 10001

Is it correct? Where else should I change it?

8

Try using different port on both, like 10010 for example