Rcstack - unable to clone large hg repositories

Support
,
1

Hi,

I’ve migrated RhodeCode from a rccontrol instance to a rcstack one, following the migration documentation at Migration to Docker & Rcstack - RhodeCode rcstack 5.13.0 documentation .

Everything seems to be working, but I am not able to clone large hg repositories.
I tried from the 4.28.0 image to the beta one and the issue remains.

It usually takes more than 30 seconds to clone those large repositories. The 30 seconds seems to be a hard limit in my current configuration.

The command fails with:

% hg clone --verbose https://rhodecode.example.com/a/bc/d "S:\Test\Config"
requesting all changes
adding changesets
adding manifests
adding file changes
transaction abort!
rollback completed
(sent 4 HTTP requests and 2242 bytes; received 336271052 bytes in responses)
abort: stream ended unexpectedly  (got 9788 bytes, expected 32768)
[command returned code 255 Tue Jul  9 15:58:46 2024]

From the vccserver logs we can see a worker timeout:

{"timestamp": "2024-07-09T13:58:14.379815+00:00", "levelname": "INFO", "name": "vcsserver.tweens.request_wrapper", "message": "Req[   8] IP: 127.0.0.1 POST 
Request to a/bc/d@/a/bc/d/localrepository time: 0.0325s [mercurial/proto-1.0 (Mercurial 6.5.1)], VCSServer 5.1.0
", "req_id": null, "time": 0.032531023025512695, "ver": "5.1.0", "code": 200, "path": "/a/bc/d", "view_name": "stream_hg", "user_agent
": "mercurial/proto-1.0 (Mercurial 6.5.1)", "vcs_method": "localrepository", "repo_name": "a/bc/d"}
2024-07-09 13:58:44 [11] [CRITICAL] WORKER TIMEOUT (pid:13)
2024-07-09 13:58:44 [13] [INFO] [13        ] worker received SIGABRT signal
2024-07-09 13:58:44 [13] [ERROR] Error handling request
Traceback (most recent call last):
  File "/home/rhodecode/venv/lib/python3.11/site-packages/gunicorn/workers/sync.py", line 183, in handle_request
    for item in respiter:
  File "/home/rhodecode/venv/lib/python3.11/site-packages/mercurial/hgweb/hgweb_mod.py", line 363, in run_wsgi
    for r in self._runwsgi(req, res, repo):
  File "/home/rhodecode/venv/lib/python3.11/site-packages/mercurial/hgweb/request.py", line 581, in sendresponse
    for chunk in self._bodygen:
  File "/home/rhodecode/venv/lib/python3.11/site-packages/mercurial/wireprotoserver.py", line 291, in genversion2
    for chunk in gen:
  File "/home/rhodecode/venv/lib/python3.11/site-packages/mercurial/utils/compression.py", line 688, in compressstream
    for chunk in it:
  File "/home/rhodecode/venv/lib/python3.11/site-packages/mercurial/bundle2.py", line 740, in getchunks
    for chunk in self._compengine.compressstream(
  File "/home/rhodecode/venv/lib/python3.11/site-packages/mercurial/bundle2.py", line 763, in _getcorechunk
    for chunk in part.getchunks(ui=self.ui):
  File "/home/rhodecode/venv/lib/python3.11/site-packages/mercurial/bundle2.py", line 1222, in getchunks
    pycompat.raisewithtb(exc, tb)
  File "/home/rhodecode/venv/lib/python3.11/site-packages/mercurial/pycompat.py", line 348, in raisewithtb
    raise exc.with_traceback(tb)
  File "/home/rhodecode/venv/lib/python3.11/site-packages/mercurial/bundle2.py", line 1194, in getchunks
    for chunk in self._payloadchunks():
  File "/home/rhodecode/venv/lib/python3.11/site-packages/mercurial/bundle2.py", line 1239, in _payloadchunks
    chunk = buff.read(preferedchunksize)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rhodecode/venv/lib/python3.11/site-packages/mercurial/util.py", line 2722, in read
    for chunk in self.iter:
  File "/home/rhodecode/venv/lib/python3.11/site-packages/mercurial/util.py", line 2693, in splitbig
    for chunk in chunks:
  File "/home/rhodecode/venv/lib/python3.11/site-packages/mercurial/changegroup.py", line 1643, in generate
    for delta in deltas:
  File "/home/rhodecode/venv/lib/python3.11/site-packages/mercurial/changegroup.py", line 1197, in deltagroup
    for i, revision in enumerate(revisions):
  File "/home/rhodecode/venv/lib/python3.11/site-packages/mercurial/utils/storageutil.py", line 536, in emitrevisions
    revision = store.rawdata(node)
               ^^^^^^^^^^^^^^^^^^^
  File "/home/rhodecode/venv/lib/python3.11/site-packages/mercurial/revlog.py", line 2784, in rawdata
    return self._revisiondata(nodeorrev, raw=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rhodecode/venv/lib/python3.11/site-packages/mercurial/revlog.py", line 2746, in _revisiondata
    rev, rawtext, validated = self._rawtext(node, rev)
                              ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rhodecode/venv/lib/python3.11/site-packages/mercurial/revlog.py", line 2729, in _rawtext
    return self._inner.raw_text(node, rev)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rhodecode/venv/lib/python3.11/site-packages/mercurial/revlog.py", line 1031, in raw_text
    bins = self._chunks(chain, targetsize=targetsize)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rhodecode/venv/lib/python3.11/site-packages/mercurial/revlog.py", line 960, in _chunks
    offset, data = self.get_segment_for_revs(firstrev, lastrev)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rhodecode/venv/lib/python3.11/site-packages/mercurial/revlog.py", line 872, in get_segment_for_revs
    return start, self._segmentfile.read_chunk(start, length)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rhodecode/venv/lib/python3.11/site-packages/mercurial/revlogutils/randomaccessfile.py", line 194, in read_chunk
    return self._read_and_update_cache(offset, length)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rhodecode/venv/lib/python3.11/site-packages/mercurial/revlogutils/randomaccessfile.py", line 205, in _read_and_update_cache
    with self._read_handle() as file_obj:
  File "/usr/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/home/rhodecode/venv/lib/python3.11/site-packages/mercurial/revlogutils/randomaccessfile.py", line 151, in _read_handle
    with self._open() as fp:
         ^^^^^^^^^^^^
  File "/home/rhodecode/venv/lib/python3.11/site-packages/mercurial/revlogutils/randomaccessfile.py", line 134, in _open
    return self.opener(self.filename, mode=mode)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rhodecode/venv/lib/python3.11/site-packages/mercurial/store.py", line 1104, in __call__
    return self.vfs(encoded, mode, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rhodecode/venv/lib/python3.11/site-packages/mercurial/vfs.py", line 495, in __call__
    f = self.join(path)
        ^^^^^^^^^^^^^^^
  File "/home/rhodecode/venv/lib/python3.11/site-packages/mercurial/vfs.py", line 592, in join
    return self._join(*parts)
           ^^^^^^^^^^^^^^^^^^
  File "/home/rhodecode/venv/lib/python3.11/site-packages/mercurial/vfs.py", line 165, in _join
    for idx, p in enumerate(paths):
                  ^^^^^^^^^^^^^^^^
  File "/home/rhodecode/venv/lib/python3.11/site-packages/gunicorn/workers/base.py", line 203, in handle_abort
    sys.exit(1)
SystemExit: 1

The vcsserver.ini is the one provided by the ./rcstack init and rhodecode.ini is also the one provided by this command, minus the backported values pointed by the migration documentation.

Did this happen before, or did I miss anything in my migration?
Also, I’m not sure how to tweak this gunicorn timeout from the rhodecode.ini or the vcsserver.ini file to see if this can solve the clone issue.

Thanks for any help.

2

I can confirm that the issue is due to a too short living gunicorn worker from the vcsserver and rhodecode services.

By adding a --timeout=0 to the GUNICORN_CMD_ARGS environment, found from the docker-compose-apps.yaml, any clone operation is successful. If we are updating this command only for vcsserver or rhodecode the issue remains.

The .custom/docker-compose-overrides.apps.yaml was updated as follows:

services:

  rhodecode:

    deploy:
      # set number of instances of services to be spawned on stack start
      replicas: 1

    environment:
      RC_COMPOSE_OVERRIDE: 1
      GUNICORN_CMD_ARGS: "--bind=0.0.0.0:10020 --name=gunicorn-rhodecode-1 --workers=2 --timeout=0"
      #DB_UPGRADE: 0  # 1 run the DB upgrade, 0 disable db upgrade at startup
      #SETUP_APP: 0   # run the application default settings setup, can be turned off after initial run

  vcsserver:

    deploy:
      # set number of instances of services to be spawned on stack start
      replicas: 1

    environment:
      RC_COMPOSE_OVERRIDE: 1
      GUNICORN_CMD_ARGS: "--bind=0.0.0.0:10010 --name=gunicorn-vcsserver-1 --workers=3 --timeout=0"

Beware that the timeout=0 says “no timeout”. If you are facing the same issue, you may want other value. The old rccontrol values were 21600 seconds for the timeout, and 3600 seconds for the graceful_timeout ( see rhodecode-enterprise-ce Files · configs/production.ini · RhodeCode Free Hosting ).

@rhodecode-support is there any other way to change the timeout?

1 Like
3

Hello @aji,

We see your request and our dev team is already checking it. They will come back shortly with update.

Thanks for raising this issue!

Regards, Artem

4

Hi! Thanks for providing feedback!
Gunicorn timeouts could be configured in

config/_shared/gunicorn_conf_rc.py
config/_shared/gunicorn_conf_vcs.py

We have some planned performance improvements specifically for big mercurial repos for 5.1.0 fyi

1 Like
5

Good to know.
Thanks.

1 Like