We’re running Community 4.12.4. Today we ran out of space on the repo volume. Since growing the disk (VM), all repo actions are ok EXCEPT push actions to master (84GB).
All we see is Abort HTTP 500 error on the client side. Am not seeing any errors in the server logs.
Any clues how to solve this one?
FYI, also seeing this when trying to upgrade to latest…
Running Upgrade of <CommunityInstance of (community-1)> at version 4.12.4 to 4.14.1
Traceback (most recent call last):
File “/opt/rhodecode/store/56kin2h8jbribv7wxd6a4gr6a3vj4f9h-python2.7-rhodecode-enterprise-ce-4.14.1/bin/.rc-upgrade-db-wrapped”, line 12, in
sys.exit(main())
File “/opt/rhodecode/store/bbsdjfz28lj6g8s90wbj6i746nwxhh2s-python2.7-click-6.6/lib/python2.7/site-packages/click/core.py”, line 716, in call
return self.main(*args, **kwargs)
File “/opt/rhodecode/store/bbsdjfz28lj6g8s90wbj6i746nwxhh2s-python2.7-click-6.6/lib/python2.7/site-packages/click/core.py”, line 696, in main
rv = self.invoke(ctx)
File “/opt/rhodecode/store/bbsdjfz28lj6g8s90wbj6i746nwxhh2s-python2.7-click-6.6/lib/python2.7/site-packages/click/core.py”, line 889, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/opt/rhodecode/store/bbsdjfz28lj6g8s90wbj6i746nwxhh2s-python2.7-click-6.6/lib/python2.7/site-packages/click/core.py”, line 534, in invoke
return callback(*args, **kwargs)
File “/opt/rhodecode/store/56kin2h8jbribv7wxd6a4gr6a3vj4f9h-python2.7-rhodecode-enterprise-ce-4.14.1/lib/python2.7/site-packages/rhodecode/lib/rc_commands/upgrade_db.py”, line 37, in main
return command(ini_path, force_yes)
File “/opt/rhodecode/store/56kin2h8jbribv7wxd6a4gr6a3vj4f9h-python2.7-rhodecode-enterprise-ce-4.14.1/lib/python2.7/site-packages/rhodecode/lib/rc_commands/upgrade_db.py”, line 43, in command
with bootstrap(ini_path, env={‘RC_CMD_UPGRADE_DB’: ‘1’}) as env:
File “/opt/rhodecode/store/56kin2h8jbribv7wxd6a4gr6a3vj4f9h-python2.7-rhodecode-enterprise-ce-4.14.1/lib/python2.7/site-packages/rhodecode/lib/pyramid_utils.py”, line 53, in bootstrap
return pyramid_bootstrap(config_uri, request=request, options=options)
File “/opt/rhodecode/store/vipsx3nr7395jq0s9qrk1lhzalyg6nna-python2.7-pyramid-1.9.2/lib/python2.7/site-packages/pyramid/paster.py”, line 107, in bootstrap
app = get_app(config_uri, options=options)
File “/opt/rhodecode/store/vipsx3nr7395jq0s9qrk1lhzalyg6nna-python2.7-pyramid-1.9.2/lib/python2.7/site-packages/pyramid/paster.py”, line 28, in get_app
return loader.get_wsgi_app(name, options)
File “/opt/rhodecode/store/ip98x35alh0h0g60qfgb5ysc8m6nvzs8-python2.7-plaster-pastedeploy-0.6/lib/python2.7/site-packages/plaster_pastedeploy/init.py”, line 131, in get_wsgi_app
global_conf=defaults)
File “/opt/rhodecode/store/lxizr6wwy8ncr2zk52lzmysm44agv6mw-python2.7-pastedeploy-1.5.2/lib/python2.7/site-packages/paste/deploy/loadwsgi.py”, line 247, in loadapp
return loadobj(APP, uri, name=name, **kw)
File “/opt/rhodecode/store/lxizr6wwy8ncr2zk52lzmysm44agv6mw-python2.7-pastedeploy-1.5.2/lib/python2.7/site-packages/paste/deploy/loadwsgi.py”, line 272, in loadobj
return context.create()
File “/opt/rhodecode/store/lxizr6wwy8ncr2zk52lzmysm44agv6mw-python2.7-pastedeploy-1.5.2/lib/python2.7/site-packages/paste/deploy/loadwsgi.py”, line 710, in create
return self.object_type.invoke(self)
File “/opt/rhodecode/store/lxizr6wwy8ncr2zk52lzmysm44agv6mw-python2.7-pastedeploy-1.5.2/lib/python2.7/site-packages/paste/deploy/loadwsgi.py”, line 146, in invoke
return fix_call(context.object, context.global_conf, **context.local_conf)
File “/opt/rhodecode/store/lxizr6wwy8ncr2zk52lzmysm44agv6mw-python2.7-pastedeploy-1.5.2/lib/python2.7/site-packages/paste/deploy/util.py”, line 55, in fix_call
val = callable(*args, **kw)
File “/opt/rhodecode/store/56kin2h8jbribv7wxd6a4gr6a3vj4f9h-python2.7-rhodecode-enterprise-ce-4.14.1/lib/python2.7/site-packages/rhodecode/config/middleware.py”, line 85, in make_pyramid_app
settings = _substitute_values(settings, environ)
File “/opt/rhodecode/store/56kin2h8jbribv7wxd6a4gr6a3vj4f9h-python2.7-rhodecode-enterprise-ce-4.14.1/lib/python2.7/site-packages/rhodecode/config/middleware.py”, line 575, in _substitute_values
for key, value in mapping.items()
File “/opt/rhodecode/store/56kin2h8jbribv7wxd6a4gr6a3vj4f9h-python2.7-rhodecode-enterprise-ce-4.14.1/lib/python2.7/site-packages/rhodecode/config/middleware.py”, line 575, in
for key, value in mapping.items()
ValueError: Single ‘}’ encountered in format string
Upgrade of RhodeCode Community failed.
Rolling back to previous version.
HTTP 500 errors were corrected with hg recover.
Community upgrade still failing.
this must be an issue with rhodecode.ini
in 4.14 there’s an ENV subsitute replace that searches for “{ENV_NAME}” in .ini file, maybe it’s somehow used there ?
I also tried an upgrade to just the next version, 4.13.0, and get the same failed result. Does that align with your idea here?
if it’s the same error, then yes. Please search for
}
Character, inside rhodecode.ini to see if it could produce a problem
The server has been upgraded every few months for over a year. Original was 4.7.2 I think, with few modifications made since then.
I see no entry for {ENV_NAME} in our rhodecode.ini file.
The only mention of the } character in non-commented lines is within the rcadmin password for postgresql. Could that be the cause?
Hmm, potentially yes. We need to check this case. Would it be possible to change that password to not use } for now ?
We’ve tested on our cloned VM and the results are good. We changed the password to exclude that character, rebooted the server, and ran the upgrade. No issues. We’ll work out a maintenance window to apply the same in production.
Thanks!