Timeout for large git repos

Support
,
#1

I have some large git repos (>2.7GB) that encounter timeouts during cloning. It looks like this question, but I just upgraded RhodeCode to 4.19.3 (before it was on 4.7) and the problem persists.

Output of git command:

git clone -v --progress https://username@host.example.com/git/reponame 2>&1 | ts -s
00:00:00 Cloning into 'reponame'...
Password for 'https://username@host.example.com':
00:00:05 POST git-upload-pack (365 bytes)
00:05:05 error: RPC failed; HTTP 502 curl 22 The requested URL returned error: 502 Proxy Error
00:05:05 fatal: The remote end hung up unexpectedly

The error occurs always exactly 5 minutes after the POST operation.

Error from the apache log:

[Wed Jun 17 15:03:32.916786 2020] [proxy_http:error] [pid 7190] (70007)The timeout specified has expired: [client 10.1.2.3:56240] AH01102: error reading status line from remote server 127.0.0.1:10002
[Wed Jun 17 15:03:32.916933 2020] [proxy:error] [pid 7190] [client 10.1.2.3:56240] AH00898: Error reading from remote server returned by /git/reponame/git-upload-pack

I already tried increasing the ProxyTimeout and connectiontimeout values on the apache side to 600 seconds, to no avail.

As far as I can tell all timeout values in rhodecode.ini are on their default values of 6 hours.

Is there any place where I can go on debugging this?

#2

Just did another test I read in the linked question. Cloning directly from RhodeCode without the reverse proxy via localhost:port works.
I guess it’s back to the Apache config then.

#3

I went through this documentation and modified the Apache config to reflect it (I inherited the server from a previous admin and it was missing some parts of that example).

The config now looks like this:

<VirtualHost host.example.com:443>

    ServerName host.example.com
    ServerAlias host.example.com

    ## serve static files by Apache, recommended for performance
    ProxyPass /_static !
    Alias /_static /home/rhodecode/.rccontrol/community-1/static
    <Directory /home/rhodecode/.rccontrol/community-1/static/>
        AllowOverride none
        Require all granted
    </Directory>

    RequestHeader set X-Forwarded-Proto "https"

    ## channelstream websocket handling
    ProxyPass /_channelstream ws://localhost:9800
    ProxyPassReverse /_channelstream ws://localhost:9800

    <Proxy *>
      Order allow,deny
      Allow from all
    </Proxy>

    # Directive to properly generate url (clone url) for RhodeCode
    ProxyPreserveHost On

    SetEnv proxy-sendchunked 1

    # Url to running RhodeCode instance. This is shown as `- URL:` when
    # running rccontrol status.
    ProxyPass / http://127.0.0.1:10002/ timeout=7200 Keepalive=On
    ProxyPassReverse / http://127.0.0.1:10002/

    # strict http prevents from https -> http downgrade
    Header always set Strict-Transport-Security "max-age=63072000; includeSubdomains; preload"

    # Set x-frame options
    Header always append X-Frame-Options SAMEORIGIN

    # To enable https use line below
    SetEnvIf X-Url-Scheme https HTTPS=1

    # SSL setup
    SSLEngine On
    SSLCertificateFile /etc/apache2/ssl/rhodecode/rc-voltaire.crt
    SSLCertificateKeyFile /etc/apache2/ssl/rhodecode/rc-voltaire.key

    SSLProtocol All -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
    SSLCipherSuite EECDH+AESGCM:EDH+AESGCM
    SSLHonorCipherOrder     on

    SSLCompression off
    SSLUseStapling on
    SSLStaplingCache "shmcb:logs/stapling-cache(150000)"
    SSLSessionTickets Off

    ErrorDocument 502 /home/rhodecode/.rccontrol/community-1/static/502.html

</VirtualHost>

Sadly, it doesn’t make any difference. The cloning still times out after exactly 5 minutes.

#4

Hi Gerald,

Can you please check with these options set for rhodecode.ini/vcsserver.ini ?

https://code.rhodecode.com/rhodecode-enterprise-ce/files/default/configs/production.ini?at=default#L71-98

#5

The values in the configuration files are set to the default values:

rhodecode.ini

timeout = 21600
limit_request_line = 0
limit_request_fields = 32768
limit_request_field_size = 0
graceful_timeout = 3600
keepalive = 2

vcsserver.ini

timeout = 21600

The other values are not set in vcsserver.ini. None of these values is anywhere near the 5 minute timeout.

#6

Gerald -

Can you try adding also connectiontimeout

ProxyPass / http://127.0.0.1:10002/ connectiontimeout=7200 timeout=7200 Keepalive=On
#7

These parameters worked. I don’t really get why, since I already tried setting connectiontimeout and timeout to 600, which is still way above the 300 seconds of the timeout, but it works now. Even with the largest repository of 11GB.

Thank you very much.