Error: [Errno 104] Connection reset by peer

Support
1

Dear Support,

Sometimes we have issues pushing changes to our Rhodecode server through VPN connection.
If we work from the office, where the Rhodecode server is located, there are no problems.
If we work from home, and connected to the office network through AO VPN, sometimes there are issues with pushing files.
I was not really able to find a pattern about when does push fails. But it seems that the size of the changeset that is being pushed is an important factor.
For example, if I replace all line endings on a file of ~200kb, I am not able to push.
If I just make 1 line change on the same file, there are no issues.

Here is the log output from the server:

2021-01-07 19:00:26.615 ERROR [rhodecode.config.middleware] error occurred handling this request for path: /rhodecode/diagnostics/forks/kvgm_dotnet_protocolbuffers-gr_lza
Traceback (most recent call last):
File “/opt/rhodecode/store/da2il1f83gci4333j0sd256nji9nlhpa-python2.7-pyramid-1.10.4/lib/python2.7/site-packages/pyramid/router.py”, line 277, in default_execution_policy
return router.invoke_request(request)
File “/opt/rhodecode/store/da2il1f83gci4333j0sd256nji9nlhpa-python2.7-pyramid-1.10.4/lib/python2.7/site-packages/pyramid/router.py”, line 249, in invoke_request
response = handle_request(request)
File “/opt/rhodecode/store/cnpzq6z5xjxzxw2m3d1cfb5yhpazscn5-python2.7-rhodecode-enterprise-ce-4.22.0/lib/python2.7/site-packages/rhodecode/lib/middleware/request_wrapper.py”, line 44, in call
response = self.handler(request)
File “/opt/rhodecode/store/cnpzq6z5xjxzxw2m3d1cfb5yhpazscn5-python2.7-rhodecode-enterprise-ce-4.22.0/lib/python2.7/site-packages/rhodecode/tweens.py”, line 103, in sanity_check
return handler(request)
File “/opt/rhodecode/store/cnpzq6z5xjxzxw2m3d1cfb5yhpazscn5-python2.7-rhodecode-enterprise-ce-4.22.0/lib/python2.7/site-packages/rhodecode/tweens.py”, line 49, in vcs_detection_tween
return handler(request)
File “/opt/rhodecode/store/da2il1f83gci4333j0sd256nji9nlhpa-python2.7-pyramid-1.10.4/lib/python2.7/site-packages/pyramid/tweens.py”, line 43, in excview_tween
response = _error_handler(request, exc)
File “/opt/rhodecode/store/da2il1f83gci4333j0sd256nji9nlhpa-python2.7-pyramid-1.10.4/lib/python2.7/site-packages/pyramid/tweens.py”, line 13, in _error_handler
response = request.invoke_exception_view(exc_info)
File “/opt/rhodecode/store/da2il1f83gci4333j0sd256nji9nlhpa-python2.7-pyramid-1.10.4/lib/python2.7/site-packages/pyramid/view.py”, line 767, in invoke_exception_view
request_iface=request_iface.combined,
File “/opt/rhodecode/store/da2il1f83gci4333j0sd256nji9nlhpa-python2.7-pyramid-1.10.4/lib/python2.7/site-packages/pyramid/view.py”, line 667, in _call_view
response = view_callable(context, request)
File “/opt/rhodecode/store/da2il1f83gci4333j0sd256nji9nlhpa-python2.7-pyramid-1.10.4/lib/python2.7/site-packages/pyramid/config/views.py”, line 169, in call
return view(context, request)
File “/opt/rhodecode/store/da2il1f83gci4333j0sd256nji9nlhpa-python2.7-pyramid-1.10.4/lib/python2.7/site-packages/pyramid/viewderivers.py”, line 401, in viewresult_to_response
result = view(context, request)
File “/opt/rhodecode/store/da2il1f83gci4333j0sd256nji9nlhpa-python2.7-pyramid-1.10.4/lib/python2.7/site-packages/pyramid/viewderivers.py”, line 144, in _requestonly_view
response = view(request)
File “/opt/rhodecode/store/cnpzq6z5xjxzxw2m3d1cfb5yhpazscn5-python2.7-rhodecode-enterprise-ce-4.22.0/lib/python2.7/site-packages/rhodecode/config/middleware.py”, line 150, in not_found_view
return wsgiapp(vcs_app)(None, request)
File “/opt/rhodecode/store/da2il1f83gci4333j0sd256nji9nlhpa-python2.7-pyramid-1.10.4/lib/python2.7/site-packages/pyramid/wsgi.py”, line 38, in decorator
return request.get_response(wrapped)
File “/opt/rhodecode/store/p05sclzmx9zbmciggm367lfz8wrmh1h5-python2.7-webob-1.8.5/lib/python2.7/site-packages/webob/request.py”, line 1314, in send
application, catch_exc_info=False)
File “/opt/rhodecode/store/p05sclzmx9zbmciggm367lfz8wrmh1h5-python2.7-webob-1.8.5/lib/python2.7/site-packages/webob/request.py”, line 1281, in call_application
output.extend(app_iter)
File “/opt/rhodecode/store/cnpzq6z5xjxzxw2m3d1cfb5yhpazscn5-python2.7-rhodecode-enterprise-ce-4.22.0/lib/python2.7/site-packages/rhodecode/lib/middleware/simplevcs.py”, line 613, in _generate_vcs_response
response = app(environ, start_response)
File “/opt/rhodecode/store/cnpzq6z5xjxzxw2m3d1cfb5yhpazscn5-python2.7-rhodecode-enterprise-ce-4.22.0/lib/python2.7/site-packages/rhodecode/lib/middleware/utils/scm_app_http.py”, line 122, in call
data=_maybe_stream_request(environ),
File “/opt/rhodecode/store/cnpzq6z5xjxzxw2m3d1cfb5yhpazscn5-python2.7-rhodecode-enterprise-ce-4.22.0/lib/python2.7/site-packages/rhodecode/lib/middleware/utils/scm_app_http.py”, line 172, in _maybe_stream_request
return environ[‘wsgi.input’].read()
File “/opt/rhodecode/store/f3cphs7x07gnqgv9iiz6y2chglc63bxk-python2.7-gunicorn-19.9.0/lib/python2.7/site-packages/gunicorn/http/body.py”, line 212, in read
data = self.reader.read(1024)
File “/opt/rhodecode/store/f3cphs7x07gnqgv9iiz6y2chglc63bxk-python2.7-gunicorn-19.9.0/lib/python2.7/site-packages/gunicorn/http/body.py”, line 128, in read
data = self.unreader.read()
File “/opt/rhodecode/store/f3cphs7x07gnqgv9iiz6y2chglc63bxk-python2.7-gunicorn-19.9.0/lib/python2.7/site-packages/gunicorn/http/unreader.py”, line 38, in read
d = self.chunk()
File “/opt/rhodecode/store/f3cphs7x07gnqgv9iiz6y2chglc63bxk-python2.7-gunicorn-19.9.0/lib/python2.7/site-packages/gunicorn/http/unreader.py”, line 65, in chunk
return self.sock.recv(self.mxchunk)
error: [Errno 104] Connection reset by peer

And here is the debug push command’s output from the client laptop:

C:\Workspace_2120>hg push --debug
pushing to http://gr_tcs@10.31.4.49/rhodecode/_2120
using http://10.31.4.49/rhodecode/_2120
http auth: user gr_tcs, password not set
sending capabilities command
http authorization required for http://10.31.4.49/rhodecode/_2120
realm: RhodeCode
user: gr_tcs
password:
http auth: user gr_tcs, password *********
no changes made to subrepo kvgm_sub_protobufs since last push to http://10.31.4.49/rhodecode/_645_kvgm_sub_protobufs
query 1; heads
sending batch command
searching for changes
all remote heads known locally
preparing listkeys for “phases”
sending listkeys command
received listkey for “phases”: 15 bytes
checking for updated bookmarks
preparing listkeys for “bookmarks”
sending listkeys command
received listkey for “bookmarks”: 0 bytes
sending branchmap command
preparing listkeys for “bookmarks”
sending listkeys command
received listkey for “bookmarks”: 0 bytes
4 changesets found
list of changesets:
734358de74f4296e80712cf50a16db1d4574d58a
efe1fdf86b44f99a3f3f2d88bf3cd87111d50377
6b7dc3ce02d1ff25552fd2e9146dbf6966617bb3
6b498e55215c909f6fcb4ad8b307f43c10a73f0c
bundle2-output-bundle: “HG20”, 5 parts total
bundle2-output-part: “replycaps” 205 bytes payload
bundle2-output-part: “check:phases” 24 bytes payload
bundle2-output-part: “check:heads” streamed payload
bundle2-output-part: “changegroup” (params: 1 mandatory) streamed payload
bundle2-output-part: “phase-heads” 24 bytes payload
sending unbundle command
sending 433802 bytes
abort: error: An existing connection was forcibly closed by the remote host

2

It seems that the clients were connected to the server but they encountered with " [Errno 104] Connection reset by peer" exception when they tried to send data. For the first time, Python raises “[Errno 104] Connection reset by peer” exception, then for the second time and more you would get “[Errno 32] Broken pipe” exception on the client side.

This can mean that the server is up and listening on the port (otherwise, you would get “[Errno 111] Connection refused" exception on the client side”. This also means that the server is crashed before closing the connection since if the connection was closed on the server side before sending data on the client side, the client would encounter with “[Errno 32] Broken pipe” exception.

3

We think that VPN can add a latency/slower-network and you hit some sort of timeout. Please review the gunicorn timeout settings, or your reverse proxy timeouts.

4

Dear Support,

Here are the relevant settings from rhodecode.ini:

> ##########################
> ## GUNICORN WSGI SERVER ##
> ##########################
> ## run with gunicorn --log-config rhodecode.ini --paste rhodecode.ini>
> 
> use = egg:gunicorn#main
> ## Sets the number of process workers. You must set `instance_id = *`
> ## when this option is set to more than one worker, recommended
> ## value is (2 * NUMBER_OF_CPUS + 1), eg 2CPU = 5 workers
> ## The `instance_id = *` must be set in the [app:main] section below
> workers = 5
> ## number of threads for each of the worker, must be set to 1 for gevent
> ## generally recommened to be at 1
> #threads = 1
> ## process name
> proc_name = RhodeCodeCommunity
> ## type of worker class, one of sync, gevent
> ## recommended for bigger setup is using of of other than sync one
> worker_class = sync
> ## The maximum number of simultaneous clients. Valid only for Gevent
> #worker_connections = 10
> ## max number of requests that worker will handle before being gracefully
> ## restarted, could prevent memory leaks
> max_requests = 1000
> max_requests_jitter = 30
> ## amount of time a worker can spend with handling a request before it
> ## gets killed and restarted. Set to 6hrs
> timeout = 21600
> limit_request_line = 0
> limit_request_fields = 32768
> limit_request_field_size = 0
> graceful_timeout = 3600
> keepalive = 2
> memory_max_usage = 0
> memory_usage_check_interval = 60
> memory_usage_recovery_threshold = 0.8

What shall I change?

I configured some local clones to bypass the revers proxy by accessing the Rhodecode server directly on IP address, instead of domain name. I still have the same problem, so I guess we can take the reverse proxy out from the equation.

5

Thanks Warren for the reply!

Any tips on how to prevent this to happen?

Tamas

6

Dear Support,

Any updates on this ticket?

7

I comment, just to bring this topic a little attention…

8

Hi,

Sorry no updates on that, it’s hard to judge base on the data provided.
This would require some more in-depth testing for connectivity problems.

This isn’t something we could get sorted here I believe.