Cloning from git repo - "No module named dulwich.errors"

Support
1

When creating a git repo with cloning option, I have following erro pasted below. There is no problem creating empty git repo.

2016-09-28 11:16:29.233 INFO  [rhodecode.lib.base] IP: 192.168.40.56 User: <AuthUser('id:2[admin] ip:192.168.40.56 auth:True')> accessed / [home.index]
2016-09-28 11:16:29.278 INFO  [rhodecode.lib.auth] user <AuthUser('id:2[admin] ip:192.168.40.56 auth:True')> authenticating with:RHODECODE_AUTH IS authenticated on func HomeController:index
2016-09-28 11:16:29.483 INFO  [rhodecode.lib.middleware.request_wrapper] IP: 192.168.40.56 Request to / time: 0.252s
2016-09-28 11:16:31.855 INFO  [rhodecode.lib.base] IP: 192.168.40.56 User: <AuthUser('id:2[admin] ip:192.168.40.56 auth:True')> accessed /Avatar [home.index_repo_group]
2016-09-28 11:16:31.888 INFO  [rhodecode.lib.auth] user <AuthUser('id:2[admin] ip:192.168.40.56 auth:True')> authenticating with:RHODECODE_AUTH IS authenticated on func HomeController:index_repo_group
2016-09-28 11:16:32.080 INFO  [rhodecode.lib.middleware.request_wrapper] IP: 192.168.40.56 Request to /Avatar time: 0.228s
2016-09-28 11:16:34.019 INFO  [rhodecode.lib.base] IP: 192.168.40.56 User: <AuthUser('id:2[admin] ip:192.168.40.56 auth:True')> accessed /_admin/create_repository [admin/repos.create_repository]
2016-09-28 11:16:34.023 INFO  [rhodecode.lib.auth] user <AuthUser('id:2[admin] ip:192.168.40.56 auth:True')> authenticating with:RHODECODE_AUTH IS authenticated on func ReposController:__before__
2016-09-28 11:16:34.382 INFO  [rhodecode.lib.middleware.request_wrapper] IP: 192.168.40.56 Request to /_admin/create_repository time: 0.366s
2016-09-28 11:16:58.083 INFO  [rhodecode.lib.base] IP: 192.168.40.56 User: <AuthUser('id:2[admin] ip:192.168.40.56 auth:True')> accessed /_admin/repos [admin/repos.create]
2016-09-28 11:16:58.108 INFO  [rhodecode.lib.auth] user <AuthUser('id:2[admin] ip:192.168.40.56 auth:True')>     authenticating with:RHODECODE_AUTH IS authenticated on func ReposController:__before__
2016-09-28 11:16:58.888 INFO  [rhodecode.lib.utils] Logging action:`started_following_repo` on repo:`<RepoTemp('id:9')>` by user:<User('id:2:admin')> ip:192.168.40.56
2016-09-28 11:16:58.922 INFO  [rhodecode.lib.utils] Logging action:`user_created_repo` on repo:`Avatar/openrpd` by user:<User('id:2:admin')> ip:192.168.40.56
2016-09-28 11:16:58.948 INFO  [rhodecode.model.repo] creating repo openrpd in /home/vcs/repos/Avatar/openrpd from url: `http://192.168.2.192:5000/AVATAR/openrpd`
2016-09-28 11:17:00.057 WARNI [rhodecode.lib.celerylib.tasks] Exception No module named dulwich.errors occurred when creating repository, doing cleanup...
2016-09-28 11:17:00.155 INFO  [rhodecode.model.repo] Removing repository /home/vcs/repos/Avatar/openrpd
2016-09-28 11:17:00.178 ERROR [rhodecode.controllers.admin.repos] Exception creating a repository
Traceback (most recent call last):
  File "/opt/rhodecode/store/2nrqs5ylil69bsjb09j699ydahcxvvfn-python2.7-rhodecode-enterprise-ce-4.4.0/lib/python2.7/site-packages/rhodecode/controllers/admin/repos.py", line 171, in create
task = RepoModel().create(form_result, c.rhodecode_user.user_id)
  File "/opt/rhodecode/store/2nrqs5ylil69bsjb09j699ydahcxvvfn-python2.7-rhodecode-enterprise-ce-4.4.0/lib/python2.7/site-packages/rhodecode/model/repo.py", line 556, in create
return run_task(tasks.create_repo, form_data, cur_user)
  File "/opt/rhodecode/store/2nrqs5ylil69bsjb09j699ydahcxvvfn-python2.7-rhodecode-enterprise-ce-4.4.0/lib/python2.7/site-packages/rhodecode/lib/celerylib/__init__.py", line 160, in run_task
return ResultWrapper(task(*args, **kwargs))
  File "/opt/rhodecode/store/2nrqs5ylil69bsjb09j699ydahcxvvfn-python2.7-rhodecode-enterprise-ce-4.4.0/lib/python2.7/site-packages/rhodecode/lib/celerylib/__init__.py", line 107, in __call__
return super(RhodecodeCeleryTask, self).__call__(*args, **kwargs)
  File "/opt/rhodecode/store/32ay02r6m59qvxk7gvybpfsqmmhd810v-python2.7-celery-2.2.10/lib/python2.7/site-packages/celery/task/base.py", line 241, in __call__
return self.run(*args, **kwargs)
  File "/opt/rhodecode/store/32ay02r6m59qvxk7gvybpfsqmmhd810v-python2.7-celery-2.2.10/lib/python2.7/site-packages/celery/app/__init__.py", line 141, in run
return fun(*args, **kwargs)
  File "<string>", line 2, in create_repo
  File "/opt/rhodecode/store/2nrqs5ylil69bsjb09j699ydahcxvvfn-python2.7-rhodecode-enterprise-ce-4.4.0/lib/python2.7/site-packages/rhodecode/lib/celerylib/__init__.py", line 200, in __wrapper
ret = func(*fargs, **fkwargs)
  File "<string>", line 2, in create_repo
  File "/opt/rhodecode/store/2nrqs5ylil69bsjb09j699ydahcxvvfn-python2.7-rhodecode-enterprise-ce-4.4.0/lib/python2.7/site-packages/rhodecode/lib/celerylib/__init__.py", line 222, in __wrapper
ret = func(*fargs, **fkwargs)
  File "/opt/rhodecode/store/2nrqs5ylil69bsjb09j699ydahcxvvfn-python2.7-rhodecode-enterprise-ce-4.4.0/lib/python2.7/site-packages/rhodecode/lib/celerylib/tasks.py", line 171, in create_repo
clone_uri=clone_uri,
  File "/opt/rhodecode/store/2nrqs5ylil69bsjb09j699ydahcxvvfn-python2.7-rhodecode-enterprise-ce-4.4.0/lib/python2.7/site-packages/rhodecode/model/repo.py", line 873, in _create_filesystem_repo
bare=True)
  File "/opt/rhodecode/store/2nrqs5ylil69bsjb09j699ydahcxvvfn-python2.7-rhodecode-enterprise-ce-4.4.0/lib/python2.7/site-packages/rhodecode/lib/vcs/backends/git/repository.py", line 70, in __init__
self._init_repo(create, src_url, update_after_clone, bare)
  File "/opt/rhodecode/store/2nrqs5ylil69bsjb09j699ydahcxvvfn-python2.7-rhodecode-enterprise-ce-4.4.0/lib/python2.7/site-packages/rhodecode/lib/vcs/backends/git/repository.py", line 157, in _init_repo
self.clone(src_url, update_after_clone, bare)
  File "/opt/rhodecode/store/2nrqs5ylil69bsjb09j699ydahcxvvfn-python2.7-rhodecode-enterprise-ce-4.4.0/lib/python2.7/site-packages/rhodecode/lib/vcs/backends/git/repository.py", line 637, in clone
url, deferred, valid_refs, update_after_clone)
  File "/opt/rhodecode/store/2nrqs5ylil69bsjb09j699ydahcxvvfn-python2.7-rhodecode-enterprise-ce-4.4.0/lib/python2.7/site-packages/rhodecode/lib/vcs/exceptions.py", line 175, in wrapper
return func(*args, **kwargs)
  File "/opt/rhodecode/store/2nrqs5ylil69bsjb09j699ydahcxvvfn-python2.7-rhodecode-enterprise-ce-4.4.0/lib/python2.7/site-packages/rhodecode/lib/vcs/client.py", line 319, in caller
return func(*all_args, **kwargs)
  File "/opt/rhodecode/store/ss0apdfdw13jgjplhgxqqxv7pmwxfz5d-python2.7-Pyro4-4.35/lib/python2.7/site-packages/Pyro4/core.py", line 168, in __call__
return self.__send(self.__name, args, kwargs)
  File "/opt/rhodecode/store/ss0apdfdw13jgjplhgxqqxv7pmwxfz5d-python2.7-Pyro4-4.35/lib/python2.7/site-packages/Pyro4/core.py", line 391, in _pyroInvoke
data = serializer.deserializeData(msg.data, compressed=msg.flags & message.FLAGS_COMPRESSED)
  File "/opt/rhodecode/store/ss0apdfdw13jgjplhgxqqxv7pmwxfz5d-python2.7-Pyro4-4.35/lib/python2.7/site-packages/Pyro4/util.py", line 162, in deserializeData
return self.loads(data)
  File "/opt/rhodecode/store/ss0apdfdw13jgjplhgxqqxv7pmwxfz5d-python2.7-Pyro4-4.35/lib/python2.7/site-packages/Pyro4/util.py", line 412, in loads
return pickle.loads(data)
ImportError: No module named dulwich.errors

My env:
$ rccontrol status

 - NAME: community-1
 - STATUS: RUNNING
 - TYPE: Community
 - VERSION: 4.4.0
 - VCS: vcsserver-1
 - URL: http://0.0.0.0:8080

 - NAME: vcsserver-1
 - STATUS: RUNNING
 - TYPE: VCSServer
 - VERSION: 4.4.0
 - URL: http://127.0.0.1:10002

$ find /opt/rhodecode/ -name "*dulwich*"
/opt/rhodecode/store/kbkkbpalhw5cdjsnly7hihhx8p4h1w15-python2.7-dulwich-0.13.0
/opt/rhodecode/store/kbkkbpalhw5cdjsnly7hihhx8p4h1w15-python2.7-dulwich-0.13.0/lib/python2.7/site-packages/dulwich-0.13.0.dist-info
/opt/rhodecode/store/kbkkbpalhw5cdjsnly7hihhx8p4h1w15-python2.7-dulwich-0.13.0/lib/python2.7/site-packages/dulwich
/opt/rhodecode/store/kbkkbpalhw5cdjsnly7hihhx8p4h1w15-python2.7-dulwich-0.13.0/bin/dulwich
/opt/rhodecode/store/kbkkbpalhw5cdjsnly7hihhx8p4h1w15-python2.7-dulwich-0.13.0/bin/.dulwich-wrapped
2

Hi,

Maybe it looks like your celery workers is running “old” snapshot of the code, before the upgrade. Please try to restart it.

3

I’ve restarted both vcs and community instances. I’ve also upgraded both, since there is 4.4.1 available. Nothing work. The error message is still the same.

4

Please check if “celery” process is running. RhodeCode uses as async worker and it seems that this might be still running on old code. It has nothing to do with vcsserver and community instance. It’s CELERY process that i’m thinking of here.

5

I installed and configured celery w/ rabbitmq as in [1] (I’ve had no celery before), run it with paster, restart community instance.
Now, I can see in logs, that it uses celery, but celery is thowing the very same exception:

[2016-09-30 11:56:53,965: ERROR/MainProcess] Task rhodecode.lib.celerylib.tasks.create_repo[942ddeac-918c-4572-a05f-6852fd3aa4cc] raised exception: ImportError('No module named dulwich.errors',)
Traceback (most recent call last):
  File "/opt/rhodecode/store/32ay02r6m59qvxk7gvybpfsqmmhd810v-python2.7-celery-2.2.10/lib/python2.7/site-packages/celery/execute/trace.py", line 34, in trace
return cls(states.SUCCESS, retval=fun(*args, **kwargs))
  File "/opt/rhodecode/store/zdz5m20x9vhlp3ibbq14l95mcshhja3l-python2.7-rhodecode-enterprise-ce-4.4.1/lib/python2.7/site-packages/rhodecode/lib/celerylib/__init__.py", line 127, in __call__
return super(RhodecodeCeleryTask, self).__call__(*args, **kwargs)
  File "/opt/rhodecode/store/32ay02r6m59qvxk7gvybpfsqmmhd810v-python2.7-celery-2.2.10/lib/python2.7/site-packages/celery/task/base.py", line 241, in __call__
return self.run(*args, **kwargs)
  File "/opt/rhodecode/store/32ay02r6m59qvxk7gvybpfsqmmhd810v-python2.7-celery-2.2.10/lib/python2.7/site-packages/celery/app/__init__.py", line 141, in run
return fun(*args, **kwargs)
  File "<string>", line 2, in create_repo
  File "/opt/rhodecode/store/zdz5m20x9vhlp3ibbq14l95mcshhja3l-python2.7-rhodecode-enterprise-ce-4.4.1/lib/python2.7/site-packages/rhodecode/lib/celerylib/__init__.py", line 200, in __wrapper
ret = func(*fargs, **fkwargs)
  File "<string>", line 2, in create_repo
  File "/opt/rhodecode/store/zdz5m20x9vhlp3ibbq14l95mcshhja3l-python2.7-rhodecode-enterprise-ce-4.4.1/lib/python2.7/site-packages/rhodecode/lib/celerylib/__init__.py", line 222, in __wrapper
ret = func(*fargs, **fkwargs)
  File "/opt/rhodecode/store/zdz5m20x9vhlp3ibbq14l95mcshhja3l-python2.7-rhodecode-enterprise-ce-4.4.1/lib/python2.7/site-packages/rhodecode/lib/celerylib/tasks.py", line 171, in create_repo
clone_uri=clone_uri,
  File "/opt/rhodecode/store/zdz5m20x9vhlp3ibbq14l95mcshhja3l-python2.7-rhodecode-enterprise-ce-4.4.1/lib/python2.7/site-packages/rhodecode/model/repo.py", line 873, in _create_filesystem_repo
bare=True)
  File "/opt/rhodecode/store/zdz5m20x9vhlp3ibbq14l95mcshhja3l-python2.7-rhodecode-enterprise-ce-4.4.1/lib/python2.7/site-packages/rhodecode/lib/vcs/backends/git/repository.py", line 70, in __init__
self._init_repo(create, src_url, update_after_clone, bare)
  File "/opt/rhodecode/store/zdz5m20x9vhlp3ibbq14l95mcshhja3l-python2.7-rhodecode-enterprise-ce-4.4.1/lib/python2.7/site-packages/rhodecode/lib/vcs/backends/git/repository.py", line 157, in _init_repo
self.clone(src_url, update_after_clone, bare)
  File "/opt/rhodecode/store/zdz5m20x9vhlp3ibbq14l95mcshhja3l-python2.7-rhodecode-enterprise-ce-4.4.1/lib/python2.7/site-packages/rhodecode/lib/vcs/backends/git/repository.py", line 637, in clone
url, deferred, valid_refs, update_after_clone)
  File "/opt/rhodecode/store/zdz5m20x9vhlp3ibbq14l95mcshhja3l-python2.7-rhodecode-enterprise-ce-4.4.1/lib/python2.7/site-packages/rhodecode/lib/vcs/exceptions.py", line 175, in wrapper
return func(*args, **kwargs)
  File "/opt/rhodecode/store/zdz5m20x9vhlp3ibbq14l95mcshhja3l-python2.7-rhodecode-enterprise-ce-4.4.1/lib/python2.7/site-packages/rhodecode/lib/vcs/client.py", line 319, in caller
return func(*all_args, **kwargs)
  File "/opt/rhodecode/store/ss0apdfdw13jgjplhgxqqxv7pmwxfz5d-python2.7-Pyro4-4.35/lib/python2.7/site-packages/Pyro4/core.py", line 168, in __call__
return self.__send(self.__name, args, kwargs)
  File "/opt/rhodecode/store/ss0apdfdw13jgjplhgxqqxv7pmwxfz5d-python2.7-Pyro4-4.35/lib/python2.7/site-packages/Pyro4/core.py", line 391, in _pyroInvoke
data = serializer.deserializeData(msg.data, compressed=msg.flags & message.FLAGS_COMPRESSED)
  File "/opt/rhodecode/store/ss0apdfdw13jgjplhgxqqxv7pmwxfz5d-python2.7-Pyro4-4.35/lib/python2.7/site-packages/Pyro4/util.py", line 162, in deserializeData
return self.loads(data)
  File "/opt/rhodecode/store/ss0apdfdw13jgjplhgxqqxv7pmwxfz5d-python2.7-Pyro4-4.35/lib/python2.7/site-packages/Pyro4/util.py", line 412, in loads
return pickle.loads(data)
ImportError: No module named dulwich.errors

Do you need some additional logs? How can I help debugging this? I have python master in my team, you can send me some python-charms, we will try to debug.

[1] https://docs.rhodecode.com/RhodeCode-Enterprise/install/install-celery.html

6

Hi, so you had NO celery before ? That’s puzzling. I tested few imports from github etc and it works ok. Is there something special with the server you’re importing ?

Ps. did you try to restart the machine maybe there’s some zombie processes still running with the old code ?

7

You were right! Importing from github does work.

Importing from rhodecode 1.7.1 doesn’t ]:->

8

Ahh, thanks for the hint. We’ll try to reproduce this then !