Error when running "Remap and rescan"

Support
1

I’m trying to manually create some Git repositories on my filesystem. I just create empty ones, using git init --bare.

When I press “Remap and rescan” / “Rescan filesystem” (without any other option checked), I get an exception, although the repositories seem to be added to RhodeCode correctly, but not sure if something might be broken after.

The exception is:

POST http://myserver/_admin/settings/mapping/update

Traceback (most recent call last):
  File "/nix/store/s43k2r9rysfbzmsjdqnxgzvvb7zjhkxb-python2.7-pyramid-1.10.4/lib/python2.7/site-packages/pyramid/tweens.py", line 41, in excview_tween
    response = handler(request)
  File "/nix/store/s43k2r9rysfbzmsjdqnxgzvvb7zjhkxb-python2.7-pyramid-1.10.4/lib/python2.7/site-packages/pyramid/router.py", line 148, in handle_request
    registry, request, context, context_iface, view_name
  File "/nix/store/s43k2r9rysfbzmsjdqnxgzvvb7zjhkxb-python2.7-pyramid-1.10.4/lib/python2.7/site-packages/pyramid/view.py", line 667, in _call_view
    response = view_callable(context, request)
  File "/nix/store/s43k2r9rysfbzmsjdqnxgzvvb7zjhkxb-python2.7-pyramid-1.10.4/lib/python2.7/site-packages/pyramid/config/views.py", line 188, in attr_view
    return view(context, request)
  File "/nix/store/s43k2r9rysfbzmsjdqnxgzvvb7zjhkxb-python2.7-pyramid-1.10.4/lib/python2.7/site-packages/pyramid/config/views.py", line 214, in predicate_wrapper
    return view(context, request)
  File "/nix/store/s43k2r9rysfbzmsjdqnxgzvvb7zjhkxb-python2.7-pyramid-1.10.4/lib/python2.7/site-packages/pyramid/viewderivers.py", line 436, in rendered_view
    result = view(context, request)
  File "/nix/store/s43k2r9rysfbzmsjdqnxgzvvb7zjhkxb-python2.7-pyramid-1.10.4/lib/python2.7/site-packages/pyramid/viewderivers.py", line 132, in _class_view
    response = getattr(inst, attr)()
  File "/nix/store/liqcjfzp4qrjsw0fmv88pmlbd7jg8l26-python2.7-rhodecode-enterprise-ce-4.28.0/lib/python2.7/site-packages/rhodecode/lib/auth.py", line 2512, in local_wrapper
    return wrapper(func, *args, **kwds)
  File "/nix/store/liqcjfzp4qrjsw0fmv88pmlbd7jg8l26-python2.7-rhodecode-enterprise-ce-4.28.0/lib/python2.7/site-packages/rhodecode/lib/auth.py", line 1828, in __wrapper
    return func(*fargs, **fkwargs)
  File "/nix/store/liqcjfzp4qrjsw0fmv88pmlbd7jg8l26-python2.7-rhodecode-enterprise-ce-4.28.0/lib/python2.7/site-packages/rhodecode/lib/auth.py", line 2512, in local_wrapper
    return wrapper(func, *args, **kwds)
  File "/nix/store/liqcjfzp4qrjsw0fmv88pmlbd7jg8l26-python2.7-rhodecode-enterprise-ce-4.28.0/lib/python2.7/site-packages/rhodecode/lib/auth.py", line 1902, in __wrapper
    return func(*fargs, **fkwargs)
  File "/nix/store/liqcjfzp4qrjsw0fmv88pmlbd7jg8l26-python2.7-rhodecode-enterprise-ce-4.28.0/lib/python2.7/site-packages/rhodecode/lib/auth.py", line 2512, in local_wrapper
    return wrapper(func, *args, **kwds)
  File "/nix/store/liqcjfzp4qrjsw0fmv88pmlbd7jg8l26-python2.7-rhodecode-enterprise-ce-4.28.0/lib/python2.7/site-packages/rhodecode/lib/auth.py", line 1721, in __wrapper
    return func(*fargs, **fkwargs)
  File "/nix/store/liqcjfzp4qrjsw0fmv88pmlbd7jg8l26-python2.7-rhodecode-enterprise-ce-4.28.0/lib/python2.7/site-packages/rhodecode/apps/admin/views/settings.py", line 249, in settings_mapping_update
    added, removed = repo2db_mapper(filesystem_repos, rm_obsolete)
  File "/nix/store/liqcjfzp4qrjsw0fmv88pmlbd7jg8l26-python2.7-rhodecode-enterprise-ce-4.28.0/lib/python2.7/site-packages/rhodecode/lib/utils.py", line 565, in repo2db_mapper
    git_repo._update_server_info()
  File "/nix/store/liqcjfzp4qrjsw0fmv88pmlbd7jg8l26-python2.7-rhodecode-enterprise-ce-4.28.0/lib/python2.7/site-packages/rhodecode/lib/vcs/backends/git/repository.py", line 702, in _update_server_info
    self._remote.update_server_info()
  File "/nix/store/liqcjfzp4qrjsw0fmv88pmlbd7jg8l26-python2.7-rhodecode-enterprise-ce-4.28.0/lib/python2.7/site-packages/rhodecode/lib/vcs/client_http.py", line 242, in repo_remote_attr
    return self._call(name, *args, **kwargs)
  File "/nix/store/liqcjfzp4qrjsw0fmv88pmlbd7jg8l26-python2.7-rhodecode-enterprise-ce-4.28.0/lib/python2.7/site-packages/rhodecode/lib/vcs/exceptions.py", line 187, in wrapper
    return func(*args, **kwargs)
  File "/nix/store/liqcjfzp4qrjsw0fmv88pmlbd7jg8l26-python2.7-rhodecode-enterprise-ce-4.28.0/lib/python2.7/site-packages/rhodecode/lib/vcs/client_http.py", line 303, in _call
    result = remote_call(cache_key)
  File "/nix/store/liqcjfzp4qrjsw0fmv88pmlbd7jg8l26-python2.7-rhodecode-enterprise-ce-4.28.0/lib/python2.7/site-packages/rhodecode/lib/rc_cache/utils.py", line 85, in decorate
    return creator()
  File "/nix/store/liqcjfzp4qrjsw0fmv88pmlbd7jg8l26-python2.7-rhodecode-enterprise-ce-4.28.0/lib/python2.7/site-packages/rhodecode/lib/rc_cache/utils.py", line 82, in creator
    return fn(*arg, **kw)
  File "/nix/store/liqcjfzp4qrjsw0fmv88pmlbd7jg8l26-python2.7-rhodecode-enterprise-ce-4.28.0/lib/python2.7/site-packages/rhodecode/lib/vcs/client_http.py", line 301, in remote_call
    return _remote_call(url, payload, EXCEPTIONS_MAP, self._session)
  File "/nix/store/liqcjfzp4qrjsw0fmv88pmlbd7jg8l26-python2.7-rhodecode-enterprise-ce-4.28.0/lib/python2.7/site-packages/rhodecode/lib/vcs/client_http.py", line 103, in _remote_call
    raise exc
Exception
2

can you see something on vcsserver logs side ?

3

I a having this same error when I clone a git repository from the command line and run “Remap and Rescan”. Two exceptions are logged in the Exceptions Tracker. This is one of them, but the revealing one is this:

Traceback (most recent call last):
  File "/nix/store/p1qi4lc1927i68b8nf5zlmryydjjqpza-python2.7-rhodecode-vcsserver-4.28.0/lib/python2.7/site-packages/vcsserver/http_main.py", line 347, in vcs_view
    resp = getattr(remote, method)(*args, **kwargs)
  File "/nix/store/p1qi4lc1927i68b8nf5zlmryydjjqpza-python2.7-rhodecode-vcsserver-4.28.0/lib/python2.7/site-packages/vcsserver/git.py", line 72, in wrapper
    return func(*args, **kwargs)
  File "/nix/store/p1qi4lc1927i68b8nf5zlmryydjjqpza-python2.7-rhodecode-vcsserver-4.28.0/lib/python2.7/site-packages/vcsserver/git.py", line 1227, in install_hooks
    return install_git_hooks(path, bare, force_create=force)
  File "/nix/store/p1qi4lc1927i68b8nf5zlmryydjjqpza-python2.7-rhodecode-vcsserver-4.28.0/lib/python2.7/site-packages/vcsserver/hook_utils/__init__.py", line 79, in install_git_hooks
    os.chmod(_hook_file, 0o755)
OSError: [Errno 1] Operation not permitted: '/var/opt/rhodecode_repo_store/causeway-app-simpleapp/.git/hooks/pre-receive'

This equates to this line: rhodecode-vcsserver Files · vcsserver/hook_utils/__init__.py · RhodeCode Free Hosting

This code is trying to run chmod 755 pre-receive in the hooks directory of the new git repository. This works fine if I run it from the command line, but not when I run it as www-data with:

sudo -u www-data chmod 755 pre-receive

I get:

chmod: changing permissions of 'pre-receive': Operation not permitted

I was unable to clone the repository from github via the web interface. There were other errors involved with that one. I have the repository store volume mapped to folder on my Windows workstation. This is running on WSL, so that location is /mnt/e/rhodecode-repository_store/ .

Interesting to note, all hook files have 777 permissions. Running chmod 755 as the main WSL user renders no error and is ignored. As I said, running it as the user that is likely executing the python code gets that “Operation not permitted” error.

4

I believe this might be due to host / container user mismatch. The docker container user ID /group id is 999:999

Could it be that on WSL has it setup differently and this is causing the permissions denied ?

5

if I run:
sudo -u \#999 chmod 755 pre-receive
I get that same error. I see that a bunch of processes are run under user is 999, including apache2, so I guess that is the user id docker is running as. User 999 is not identified.

root     20999 20997  0 Nov16 pts/0    00:00:10 /usr/sbin/apache2 -D FOREGROUND
999      21000 20999  0 Nov16 pts/0    00:00:00 /usr/sbin/apache2 -D FOREGROUND
999      21001 20999  0 Nov16 pts/0    00:00:01 /usr/sbin/apache2 -D FOREGROUND

sudo -u \#1000 chmod 755 pre-receive gives no error. 1000 is the user id of the primary WSL user.

6

Bear in mind that this is a mounted volume. When the repositories are not a mounted volume this specific error does not happen. I actually was trying to sidestep another error when I was trying to clone a remote repository when I mounted the repository folder this way. I am not familiar with why we use 999 for the docker user id. Can we use 1000 instead?

7

The Docker user-id is part of the image build, i think this is not possible to change unless you rebuild the image.

Could you still try to do on the host (so windows)

sudo chown -R 999:999  /path/to/mounted/directory

So it matches host id with container id ?

8

That did not work. The owner tries it and it says operation not supported. Root tries it and it gives no error but has no effect. Too bad there is no permission to run chmod.on a file. I read that it has to be root or the owner. Can the script maybe check for the 755 permission before running chmod on it? It looks like every windows file is assigned the 777 permission set. If you are looking to increase to 755 then 777 will do. Still requires a code modification but not as big as changing the user id of the docker processes.

9

Hi,

the code for hooks does this:

  • creates a hooks dir with mode=777
  • write a hooks file
  • does chmod(hooks_file, mode=755)

the hooks file needs to have those permissions and be executable.

Not sure what logic change we’d need to do to make it work, we believe this is somehow stil WSL host / container issue with permissions to making an executable file

10

If that is the requirement, then this error can be avoided by either checking the permissions of the file before doing chmod on it or catching the error and reporting it as a warning. All files, besides System Volume Information,DumpStack.log.tmp and pagefile.sys
look to have 777 permissions on them when accessed as a windows mount from within WSL:

ls -alt
ls: cannot access 'DumpStack.log.tmp': Permission denied
ls: cannot access 'pagefile.sys': Permission denied
ls: 'System Volume Information': Permission denied
total 4
drwxrwxrwx 1 brian brian  512 Nov 22 09:22  .
drwxrwxrwx 1 brian brian  512 Nov 16 15:09  rhodecode-repo_store
drwxr-xr-x 6 root  root  4096 Nov 13 13:45  ..
drwxrwxrwx 1 brian brian  512 Sep 24 08:57  WSL2

I am on:

WSL version: 2.0.9.0
Kernel version: 5.15.133.1-1
WSLg version: 1.0.59
MSRDC version: 1.2.4677
Direct3D version: 1.611.1-81528511
DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows version: 10.0.19042.2965
11

We believe this should be an error, because without those permissions to write a hooks file, with executable flag it’s a functionality that is essential to work.

12

Why not check that the file is not already executable before attempting chmod on it? It might be a bug for the code to assume that the docker user owns the repository directory, especially when it could be a mounted share.

13

But this is a requirement, the hooks needs to have specific permissions and be executable. If we skip this check things might not work properly. The permissions aligned with host and docker are crucial.

A push command modifies the lines on the docker, and you need to have the proper permissions setup, to first properly save the pushed data, and execute hooks properly.

14

The code I see is not checking for executable permission. If it did, it would see the 777 permissions are set already. It is calling the chmod operation on it regardless, and this is causing the error when it is a mounted windows volume in WSL2.

15

Ok then, we could change the logic is such way:

    def set_permissions_if_needed(path_to_check, perms):
        # Get current permissions
        current_permissions = os.stat(path_to_check).st_mode & 0o777  # Extract permission bits

        if current_permissions != perms:
            # Change the permissions if they are not already set
            os.chmod(path_to_check, perms)

# ....

    with open(_hook_file, 'wb') as f:
        f.write('...')
    set_permissions_if_needed(_hook_file, perms=0o755)

do you think that would fix the WSL problem ?

Because executable is one thing but generally hooks file should be 777 permissions regardless

16

That looks like it would solve the issue on WSL. You need at least 755 permissions, right? So I guess you’d do the AND bit comparison on 755 instead of 777.

17

Hi,

we’ve pushed this to vcsserver: rhodecode-vcsserver Commit - r1186:652285bb · RhodeCode Free Hosting

Should be a part of beta build anytime soon.

Thanks for making our product better !!