RhodeCode on Windows 10 Creators Update

rg_software · April 19, 2017, 5:11am

As many people may know, Microsoft announced the support of Linux tools on Windows starting from Anniversary Update (August 2016). This functionality is based on Ubuntu 14.04 code. Back then I tried to instll RhodeCode on Windows 10, but with no success. However, recently they have updated Linux support in the Creators Update, released just a week ago or so. This time it worked like a charm! So I hope that soon there will be no need to use a virtual machine. Instead one can simply run bash.exe and install RhodeCode as if running Ubuntu.

There are still issues, however:

There are some weird problems related (as I believe) to different file permissions models on Windows/Linux. In particular, I had some fun with rhodecode.ini being not found/deleted/corrupted on my instance.
It seems that the database will be eventually corrupted if you use sqlite. MySQL seems to work fine, however.

I can live with them, but there is a third issue that I cannot solve yet. It seems that hg push calls (we use Mercurial) often return “500 Internal server error” with no reason: just try the second time, and it might work. I traced down this problem to the following log fragment:

[2017-04-19 06:06:52 +0000] [9421] [ERROR] Socket error processing request.
Traceback (most recent call last):
File “/opt/rhodecode/store/bfzfyaf580smdc9g7gx8a09hc820b4vb-python2.7-gunicorn-19.6.0/lib/python2.7/site-packages/gunicorn/workers/sync.py”, line 135, in handle
self.handle_request(listener, req, client, addr)
File “/opt/rhodecode/store/bfzfyaf580smdc9g7gx8a09hc820b4vb-python2.7-gunicorn-19.6.0/lib/python2.7/site-packages/gunicorn/workers/sync.py”, line 191, in handle_request
six.reraise(*sys.exc_info())
File “/opt/rhodecode/store/bfzfyaf580smdc9g7gx8a09hc820b4vb-python2.7-gunicorn-19.6.0/lib/python2.7/site-packages/gunicorn/workers/sync.py”, line 176, in handle_request
respiter = self.wsgi(environ, resp.start_response)
File “/opt/rhodecode/store/v8v0xjg7sl1mis7bkliavihdmnch4alz-python2.7-rhodecode-enterprise-ce-4.7.1/lib/python2.7/site-packages/rhodecode/config/middleware.py”, line 386, in pyramid_app_with_cleanup
return pyramid_app(environ, start_response)
File “/opt/rhodecode/store/v8v0xjg7sl1mis7bkliavihdmnch4alz-python2.7-rhodecode-enterprise-ce-4.7.1/lib/python2.7/site-packages/rhodecode/config/middleware.py”, line 85, in call
environ, start_response)
File “/opt/rhodecode/store/929c46ghng2v8b08nlk7wfjhpan90hnm-python2.7-Routes-1.13/lib/python2.7/site-packages/routes/middleware.py”, line 131, in call
response = self.app(environ, start_response)
File “/opt/rhodecode/store/v8v0xjg7sl1mis7bkliavihdmnch4alz-python2.7-rhodecode-enterprise-ce-4.7.1/lib/python2.7/site-packages/rhodecode/lib/middleware/https_fixup.py”, line 46, in call
return self.application(environ, custom_start_response)
File “/opt/rhodecode/store/w6kjp4h2szaa42f7yksgi3lqnpp2ax51-python2.7-pyramid-1.7.4/lib/python2.7/site-packages/pyramid/router.py”, line 236, in call
response = self.invoke_subrequest(request, use_tweens=True)
File “/opt/rhodecode/store/w6kjp4h2szaa42f7yksgi3lqnpp2ax51-python2.7-pyramid-1.7.4/lib/python2.7/site-packages/pyramid/router.py”, line 211, in invoke_subrequest
response = handle_request(request)
File “/opt/rhodecode/store/v8v0xjg7sl1mis7bkliavihdmnch4alz-python2.7-rhodecode-enterprise-ce-4.7.1/lib/python2.7/site-packages/rhodecode/tweens.py”, line 46, in vcs_detection_tween
return handler(request)
File “/opt/rhodecode/store/v8v0xjg7sl1mis7bkliavihdmnch4alz-python2.7-rhodecode-enterprise-ce-4.7.1/lib/python2.7/site-packages/rhodecode/lib/middleware/request_wrapper.py”, line 43, in call
response = self.handler(request)
File “/opt/rhodecode/store/w6kjp4h2szaa42f7yksgi3lqnpp2ax51-python2.7-pyramid-1.7.4/lib/python2.7/site-packages/pyramid/tweens.py”, line 51, in excview_tween
request_iface=request_iface.combined
File “/opt/rhodecode/store/w6kjp4h2szaa42f7yksgi3lqnpp2ax51-python2.7-pyramid-1.7.4/lib/python2.7/site-packages/pyramid/view.py”, line 547, in _call_view
response = view_callable(context, request)
File “/opt/rhodecode/store/w6kjp4h2szaa42f7yksgi3lqnpp2ax51-python2.7-pyramid-1.7.4/lib/python2.7/site-packages/pyramid/config/views.py”, line 182, in call
return view(context, request)
File “/opt/rhodecode/store/w6kjp4h2szaa42f7yksgi3lqnpp2ax51-python2.7-pyramid-1.7.4/lib/python2.7/site-packages/pyramid/viewderivers.py”, line 413, in viewresult_to_response
result = view(context, request)
File “/opt/rhodecode/store/w6kjp4h2szaa42f7yksgi3lqnpp2ax51-python2.7-pyramid-1.7.4/lib/python2.7/site-packages/pyramid/wsgi.py”, line 37, in decorator
return request.get_response(wrapped)
File “/opt/rhodecode/store/cbcb7xvl643c22r0k153ggyhvzkb87z5-python2.7-WebOb-1.3.1/lib/python2.7/site-packages/webob/request.py”, line 1320, in send
application, catch_exc_info=False)
File “/opt/rhodecode/store/cbcb7xvl643c22r0k153ggyhvzkb87z5-python2.7-WebOb-1.3.1/lib/python2.7/site-packages/webob/request.py”, line 1287, in call_application
output.extend(app_iter)
File “/opt/rhodecode/store/v8v0xjg7sl1mis7bkliavihdmnch4alz-python2.7-rhodecode-enterprise-ce-4.7.1/lib/python2.7/site-packages/rhodecode/lib/middleware/simplevcs.py”, line 458, in _generate_vcs_response
response = app(environ, start_response)
File “/opt/rhodecode/store/v8v0xjg7sl1mis7bkliavihdmnch4alz-python2.7-rhodecode-enterprise-ce-4.7.1/lib/python2.7/site-packages/rhodecode/lib/middleware/utils/scm_app_http.py”, line 112, in call
stream=True)
File “/opt/rhodecode/store/7hb16gn75b8pdhygs79nd1x1dlwsyvpz-python2.7-requests-2.9.1/lib/python2.7/site-packages/requests/sessions.py”, line 468, in request
resp = self.send(prep, **send_kwargs)
File “/opt/rhodecode/store/7hb16gn75b8pdhygs79nd1x1dlwsyvpz-python2.7-requests-2.9.1/lib/python2.7/site-packages/requests/sessions.py”, line 576, in send
r = adapter.send(request, **kwargs)
File “/opt/rhodecode/store/7hb16gn75b8pdhygs79nd1x1dlwsyvpz-python2.7-requests-2.9.1/lib/python2.7/site-packages/requests/adapters.py”, line 426, in send
raise ConnectionError(err, request=request)
ConnectionError: (‘Connection aborted.’, BadStatusLine(“‘’”,))

Is there any hope for me to try to do something?.. I would greatly appreciate any advices.

rhodecode-support · April 19, 2017, 9:51am

I believe this is due to different newlines ending on windows vs linux.

As for the 3rd error it’s really odd. The BadStatusLine("''",) is because some data is sent in a format python cannot understand (some added info: Python/Django "BadStatusLine" error - Stack Overflow)

Do you have any proxy in between RhodeCode and your clients ?

rg_software · April 19, 2017, 11:12am

I believe this is due to different newlines ending on windows vs linux.

I doubt that: I was careful enough to edit rhodecode.ini with nano (and not Notepad.exe), so the file had to be OK, and RhodeCode should have processed it as if we are on Linux. There were some really odd things, but well, this is not our primary concern for now

The BadStatusLine exception is raised when you call urllib2.urlopen(url) and the remote server responds with a status code that python cannot understand.

No, no proxies, just a router that does port forwarding. I thought it might be related to some gunicorn issue (this looks similar). Do you think I might try to update some packages like gunicorn or even the whole Ubuntu to 16.04 or it makes no sense?..

rhodecode-support · April 19, 2017, 11:27am

I see, well then you did a really good thing Would it be possible you paste those errors maybe we can have an idea ? Just to check for others.

We use Gunicorn 19.6 (which is not latest) However there are few concearning things for 19.7 so we’re holding the release until it stabilizes, and other reported issues from users are addressed.

I’d try to upgrade Ubuntu itself.

I’m not sure how is your python knowledge, but you could put some prints here: /opt/rhodecode/store/v8v0xjg7sl1mis7bkliavihdmnch4alz-python2.7-rhodecode-enterprise-ce-4.7.1/lib/python2.7/site-packages/rhodecode/lib/middleware/utils/scm_app_http.py in line 112

When we use requests to do a call to VCSServer that sends mercurial data over.

rg_software · April 19, 2017, 12:06pm

Would it be possible you paste those errors maybe we can have an idea ? Just to check for others.

Well, nothing specific about the ini files – I bet it is a purely Windows issue. Once I copied the file from another folder using a Windows tool, and RhodeCode (as well as bash) didn’t recognize there is a file at all! However, cp file1 file2 then failed since it thought there IS a file there already. Crazy. Next time I did everything correctly, but RhodeCode destroyed the ini file upon restart. I still don’t know how could it be (I could repeat this behavior), so I ended up with marking the file as read-only.

With sqlite it was something similar: file not found or found and then corrupted (so the actual errors I got – sorry, I deleted that log already – were SQL errors probably caused with a bad/nonexistent database).

Thank you for the advice, I’ll try this and report… keeping my fingers crossed: hope this ubuntu update won’t destroy anything

rg_software · April 20, 2017, 6:27pm

It seems that I narrowed down the issue to the following scenario:

If I create a new repository from scratch, it works fine.
If I copy an old repository from a Windows disk with “scan & remap”, I get this error.

I tried to check the permissions, and there is a difference: by default Linux files created by RhodeCode are 755/644, while Windows copies are 777/666. I did chmod for Windows files to make them 755/644, but it didn’t help.

I wouldn’t mind to create a new repository and reimport all the files, but not sure what to do with my largefiles then.

rhodecode-support · April 20, 2017, 7:32pm

I think a hg clone would copy over everyting, and largefiles can be simply copied to the store as they are binaries (imho should work fine)

But this windows vs linux difference is odd. Maybe some config/extensions ? generally things in plaintext formats ?

rg_software · April 21, 2017, 2:25am

True, everything is odd, still experimenting…
Just one quick question for now: I know that largefiles are stored under repos/cache/largefiles, but the same files seem to be under /repos/my-repo/.hg/largefiles. Where should I copy them?

rhodecode-support · April 21, 2017, 8:18am

This section explains how largefiles works behind the scenes. If you're just adding/modifying/committing/pushing/pulling in a largefiles repo, you shouldn't have to read this section (although it can't hurt). But if you are setting up or administering Mercurial with largefiles, this is essential reading.

4.1. The local store

Each local repository has a local largefiles store in '.hg/largefiles'. When you add a new largefile to a repository, it is first stored here. When largefiles are downloaded from the central store (see below), a copy is saved there. Files in the local store are also hard-linked to the user cache.

4.2. The user cache

The user cache helps to avoid downloading and storing multiple copies of largefiles. When a largefile is needed but does not exist in the local store, Mercurial checks the user cache. If the needed largefile exists, a hard-link is created in the local store.

But because RhodeCode acts as a central server this also means:

In a typical setup with a central Mercurial server, the user who serves the central repositories will get a user cache that acts as a central store for all the repositories. This central largefiles store has every past revision of every largefile.

<!> Unlike other user caches, the central store should not be deleted! It may be the only cache that holds a largefile used by an old revision.

<!> When a client repository needs to download a largefile, it'll try to get it from the repository specified as default in the hgrc file. If not specified or incorrect repository is specified, the download will fail. As an alternative, a default path can be set for the specific hg update command:

more here: https://www.mercurial-scm.org/wiki/LargefilesExtension

rg_software · April 21, 2017, 12:48pm

Well, a victory and a defeat at the same time
I managed to get it working, but have no idea why it helped.

Bottom line:

Create a new repo as a clone of the old repo.
Go inside the new repo folder and run hg lfpull --rev "all()" <old-repo-url> to copy there all the large files.
Delete the old repo.

Works fine. Thank you very much!

rhodecode-support · April 21, 2017, 2:57pm

Great to hear! Please leave any feedback possible on how things work on windows

Cheers

rg_software · April 22, 2017, 9:08am

The bug returned So yeah, I will try to provide more info as soon as I can.

rg_software · April 28, 2017, 5:50am

Ok, here it’s me again. Yes, these socket errors appear very sporadically, but unfortunately, way too often. I followed your advice and tried to print out parameter values in scm_app_http.py (url/headers/method/data).

In the attached log file, the last request actually ended with an error. Is there anything suspicious?..
https://dl.dropboxusercontent.com/u/13527065/out1.zip

rg_software · May 1, 2017, 6:03pm

Well, it seems that another theory is finally made it working (I hope, no problems for ~2 days). The idea is to store all the files outside the Linux filesystem. They should be stored somewhere within ordinary Windows folders (e.g., “c:\repos”) and accessed via /mnt (e.g., “/mnt/c/repos”). Let’s see.

It looks like by default we should try to keep as much data as possible within Windows filesystem.

rhodecode-support · May 1, 2017, 7:33pm

Amazing, thanks for all the feedback We’ll share this to others to start using it natively under windows too.