Long Repo Creation Time then Failure

Hi,

First, a quick recap of the setup, I recently downloaded the rhodecode_ubuntu1404_ver8.ova, converted the image to run on Hyper-V and switched it from Enterprise to the Community version per this doc. I also enabled SSL and LDAPS.

Now the issue, I’m unable to create a new repository. When I try to create it (logged in with the default Admin account) it seems to be working properly but if you look at the list of repositories it shows Creating for around 20 minutes then eventually fails and shows the notice: Error creating repository repositoryname.

I should also mention that I can’t seem to access the System Info page but I’m not sure if that’s a related issue or not. I’ve checked

The status results and what look to be the relevant community.log lines below. I confirmed in rhodecode.ini that vcs.server is set to localhost:10004.

I’d appreciate any suggestions, thanks.

RHODECODE CONTROL VERSION: 1.13.1
 - NAME: community-1
 - STATUS: RUNNING
   logs:/home/rcdev/.rccontrol/community-1/community.log
 - VERSION: 4.4.2 Community
 - VCS: None
 - URL: http://127.0.0.1:10008
 - CONFIG: /home/rcdev/.rccontrol/community-1/rhodecode.ini
 - NAME: vcsserver-1
 - STATUS: RUNNING
   logs:/home/rcdev/.rccontrol/vcsserver-1/vcsserver.log
 - VERSION: 4.4.2 VCSServer
 - URL: http://127.0.0.1:10004
 - CONFIG: /home/rcdev/.rccontrol/vcsserver-1/vcsserver.ini
[22/Mar/2017:13:59:06 -0500] GNCRN <1527>   127.0.0.1       rqt:0.306062 200 34514 "GET:/_admin/repos " usr:- "https://repo.domain.com/_admin/create_repository" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.98 Safari/537.36"
2017-03-22 13:59:14.820 INFO  [rhodecode.lib.base] IP: xxx.xxx.xxx.xxx User: <AuthUser('id:2[admin] ip:xxx.xxx.xxx.xxx auth:True')> accessed /_admin/repos [admin/repos.index]
2017-03-22 13:59:14.823 INFO  [rhodecode.lib.auth] user <AuthUser('id:2[admin] ip:xxx.xxx.xxx.xxx auth:True')> authenticating with:RHODECODE_AUTH IS authenticated on func ReposController:__before__
2017-03-22 13:59:14.990 INFO  [rhodecode.lib.middleware.request_wrapper] IP: xxx.xxx.xxx.xxx Request to /_admin/repos time: 0.172s
[22/Mar/2017:13:59:14 -0500] GNCRN <1527>   127.0.0.1       rqt:0.261817 200 34514 "GET:/_admin/repos " usr:- "https://repo.domain.com/_admin/create_repository" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.98 Safari/537.36"
2017-03-22 14:46:43.557 WARNI [rhodecode.lib.celerylib.tasks] Exception Unable to connect to remote pyro server <Pyro4.core.Proxy at 0x7fb0cb7521d0, not connected, for PYRO:git_remote@localhost:10004> occurred when creating repository, doing cleanup...
2017-03-22 14:46:43.728 INFO  [rhodecode.model.repo] Removing repository /home/rcdev/repositories_store/Test2017-03-22-14-46
2017-03-22 14:46:43.737 ERROR [rhodecode.lib.vcs.backends] Perhaps this repository is in db and not in filesystem run rescan repositories with "destroy old data" option from admin panel
Traceback (most recent call last):
  File "/opt/rhodecode/store/183r08ibb7z2ibkvkn0wsb8fdgp4a5b5-python2.7-rhodecode-enterprise-ce-4.4.2/lib/python2.7/site-packages/rhodecode/lib/vcs/backends/__init__.py", line 46, in get_vcs_instance
    vcs_alias = get_scm(repo_path)[0]
  File "/opt/rhodecode/store/183r08ibb7z2ibkvkn0wsb8fdgp4a5b5-python2.7-rhodecode-enterprise-ce-4.4.2/lib/python2.7/site-packages/rhodecode/lib/vcs/utils/helpers.py", line 49, in get_scm
    raise VCSError("Given path %s is not a directory" % path)
VCSError: Given path /home/rcdev/repositories_store/Test2017-03-22-14-46 is not a directory
2017-03-22 14:46:43.738 ERROR [rhodecode.controllers.admin.repos] Exception creating a repository
Traceback (most recent call last):
  File "/opt/rhodecode/store/183r08ibb7z2ibkvkn0wsb8fdgp4a5b5-python2.7-rhodecode-enterprise-ce-4.4.2/lib/python2.7/site-packages/rhodecode/controllers/admin/repos.py", line 171, in create
    task = RepoModel().create(form_result, c.rhodecode_user.user_id)
  File "/opt/rhodecode/store/183r08ibb7z2ibkvkn0wsb8fdgp4a5b5-python2.7-rhodecode-enterprise-ce-4.4.2/lib/python2.7/site-packages/rhodecode/model/repo.py", line 556, in create
    return run_task(tasks.create_repo, form_data, cur_user)
  File "/opt/rhodecode/store/183r08ibb7z2ibkvkn0wsb8fdgp4a5b5-python2.7-rhodecode-enterprise-ce-4.4.2/lib/python2.7/site-packages/rhodecode/lib/celerylib/__init__.py", line 160, in run_task
    return ResultWrapper(task(*args, **kwargs))
  File "/opt/rhodecode/store/183r08ibb7z2ibkvkn0wsb8fdgp4a5b5-python2.7-rhodecode-enterprise-ce-4.4.2/lib/python2.7/site-packages/rhodecode/lib/celerylib/__init__.py", line 107, in __call__
    return super(RhodecodeCeleryTask, self).__call__(*args, **kwargs)
  File "/opt/rhodecode/store/81633wm50ppgmb5dm7nij7qa4pjpf387-python2.7-celery-2.2.10/lib/python2.7/site-packages/celery/task/base.py", line 241, in __call__
    return self.run(*args, **kwargs)
  File "/opt/rhodecode/store/81633wm50ppgmb5dm7nij7qa4pjpf387-python2.7-celery-2.2.10/lib/python2.7/site-packages/celery/app/__init__.py", line 141, in run
    return fun(*args, **kwargs)
  File "<string>", line 2, in create_repo
  File "/opt/rhodecode/store/183r08ibb7z2ibkvkn0wsb8fdgp4a5b5-python2.7-rhodecode-enterprise-ce-4.4.2/lib/python2.7/site-packages/rhodecode/lib/celerylib/__init__.py", line 200, in __wrapper
    ret = func(*fargs, **fkwargs)
  File "<string>", line 2, in create_repo
  File "/opt/rhodecode/store/183r08ibb7z2ibkvkn0wsb8fdgp4a5b5-python2.7-rhodecode-enterprise-ce-4.4.2/lib/python2.7/site-packages/rhodecode/lib/celerylib/__init__.py", line 222, in __wrapper
    ret = func(*fargs, **fkwargs)
  File "/opt/rhodecode/store/183r08ibb7z2ibkvkn0wsb8fdgp4a5b5-python2.7-rhodecode-enterprise-ce-4.4.2/lib/python2.7/site-packages/rhodecode/lib/celerylib/tasks.py", line 171, in create_repo
    clone_uri=clone_uri,
  File "/opt/rhodecode/store/183r08ibb7z2ibkvkn0wsb8fdgp4a5b5-python2.7-rhodecode-enterprise-ce-4.4.2/lib/python2.7/site-packages/rhodecode/model/repo.py", line 849, in _create_filesystem_repo
    if is_valid_repo_group(repo_path, self.repos_path):
  File "/opt/rhodecode/store/183r08ibb7z2ibkvkn0wsb8fdgp4a5b5-python2.7-rhodecode-enterprise-ce-4.4.2/lib/python2.7/site-packages/rhodecode/lib/utils.py", line 339, in is_valid_repo_group
    scm_ = get_scm(os.path.dirname(full_path))
  File "/opt/rhodecode/store/183r08ibb7z2ibkvkn0wsb8fdgp4a5b5-python2.7-rhodecode-enterprise-ce-4.4.2/lib/python2.7/site-packages/rhodecode/lib/vcs/utils/helpers.py", line 51, in get_scm
    found_scms = [(scm, path) for scm in get_scms_for_path(path)]
  File "/opt/rhodecode/store/183r08ibb7z2ibkvkn0wsb8fdgp4a5b5-python2.7-rhodecode-enterprise-ce-4.4.2/lib/python2.7/site-packages/rhodecode/lib/vcs/utils/helpers.py", line 91, in get_scms_for_path
    if backend.is_valid_repository(path):
  File "/opt/rhodecode/store/183r08ibb7z2ibkvkn0wsb8fdgp4a5b5-python2.7-rhodecode-enterprise-ce-4.4.2/lib/python2.7/site-packages/rhodecode/lib/vcs/backends/git/repository.py", line 141, in is_valid_repository
    GitRepository(path)
  File "/opt/rhodecode/store/183r08ibb7z2ibkvkn0wsb8fdgp4a5b5-python2.7-rhodecode-enterprise-ce-4.4.2/lib/python2.7/site-packages/rhodecode/lib/vcs/backends/git/repository.py", line 70, in __init__
    self._init_repo(create, src_url, update_after_clone, bare)
  File "/opt/rhodecode/store/183r08ibb7z2ibkvkn0wsb8fdgp4a5b5-python2.7-rhodecode-enterprise-ce-4.4.2/lib/python2.7/site-packages/rhodecode/lib/vcs/backends/git/repository.py", line 166, in _init_repo
    self._remote.assert_correct_path()
  File "/opt/rhodecode/store/183r08ibb7z2ibkvkn0wsb8fdgp4a5b5-python2.7-rhodecode-enterprise-ce-4.4.2/lib/python2.7/site-packages/rhodecode/lib/vcs/client.py", line 278, in __getattr__
    func = _get_proxy_method(self._remote_proxy, name)
  File "/opt/rhodecode/store/183r08ibb7z2ibkvkn0wsb8fdgp4a5b5-python2.7-rhodecode-enterprise-ce-4.4.2/lib/python2.7/site-packages/rhodecode/lib/vcs/client.py", line 309, in _get_proxy_method
    'Unable to connect to remote pyro server %s' % proxy)
PyroVCSCommunicationError: Unable to connect to remote pyro server <Pyro4.core.Proxy at 0x7fb0cb7521d0, not connected, for PYRO:git_remote@localhost:10004>    
[21/Mar/2017:23:47:34 -0500] GNCRN <1528>   127.0.0.1       rqt:0.323285 200 32513 "GET:/ " usr:- "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.98 Safari/537.36"
2017-03-21 23:47:36.450 INFO  [rhodecode.lib.base] IP: xxx.xxx.xxx.xxx User: <AuthUser('id:2[admin] ip:xxx.xxx.xxx.xxx auth:True')> accessed /_admin/settings/system [admin/settings.settings_system]
2017-03-21 23:47:36.453 INFO  [rhodecode.lib.auth] user <AuthUser('id:2[admin] ip:xxx.xxx.xxx.xxx auth:True')> authenticating with:RHODECODE_AUTH IS authenticated on func SettingsController:__before__
2017-03-22 00:47:37.189 WARNI [rhodecode.lib.vcs.backends.git] Failed to discover the Git version
Traceback (most recent call last):
  File "/opt/rhodecode/store/183r08ibb7z2ibkvkn0wsb8fdgp4a5b5-python2.7-rhodecode-enterprise-ce-4.4.2/lib/python2.7/site-packages/rhodecode/lib/vcs/backends/git/__init__.py", line 44, in discover_git_version
    return connection.Git.discover_git_version()
  File "/opt/rhodecode/store/183r08ibb7z2ibkvkn0wsb8fdgp4a5b5-python2.7-rhodecode-enterprise-ce-4.4.2/lib/python2.7/site-packages/rhodecode/lib/vcs/client.py", line 190, in __getattr__
    func = _get_proxy_method(remote_proxy, name)
  File "/opt/rhodecode/store/183r08ibb7z2ibkvkn0wsb8fdgp4a5b5-python2.7-rhodecode-enterprise-ce-4.4.2/lib/python2.7/site-packages/rhodecode/lib/vcs/client.py", line 309, in _get_proxy_method
    'Unable to connect to remote pyro server %s' % proxy)
PyroVCSCommunicationError: Unable to connect to remote pyro server <Pyro4.core.Proxy at 0x7fb0cb681d10, not connected, for PYRO:git_remote@localhost:10004>
2017-03-22 00:47:37.520 INFO  [rhodecode.lib.middleware.request_wrapper] IP: xxx.xxx.xxx.xxx Request to /_admin/settings/system time: 3601.072s

HI,

Please check if you have enable_celery turned inside .ini file. If celery is used then repositories are created via async worker of celery. If it’s enabled and not running then it might be that repositories aren’t created.

Hi,

Thanks, but it appears that celery is disabled in rhodecode.ini (use_celery = false) unless there is somewhere else I need to check.

Hi,

please check, if your vcsserver isn’t misconfigured. Meaning
inside rhodecode.ini if

vcs.server.enable = true
vcs.server = localhost:10004

are set.

Also, please upgrade to latest version by simply running
rccontrol self-update && rccontrol upgrade "*"

Thanks, I thought I had checked for updates after the initial setup but I must have done something incorrectly since there were still updates available nad after running the update/upgrade I was able to create the repositories without issue.

Thanks again for the help.

Great, enjoy RhodeCode :slight_smile: