Hello,
I have 2 installations of RhodeCode with rcstack.
A production install with many repositories, featuring rcstack 5.3.0 and rhodecode 5.0.0.beta21
A test install with only 2 repositories, featuring rcstack 5.4.0 and rhodecode 5.0.0.beta22
If I do rcstack backup-data on the second, it says that it will pack an archive containing what it founds in /etc/rhodecode/conf/ (configuration files), which it eventually does./etc/rhodecode/conf/ while running, but the final archive only contains what is inside /var/opt/rhodecode_repo_store/. So I tried to self-update to rcstack 5.4.0 but the result is the same! Same thing for the search index, which is listed but not in the archive.
It seems to be a bug, but I donāt know how it is related to the number of repos or size of the backup.
Justin
Hi Justin,
let us start that we really appreciate your feedback on rcstack !! Thank you for helping us improve our project.
Indeed this looks like a bug, weāll check this and fix if needed.
Great, thank you.
Did you also notice that rcstack status and rcstack stack-status are broken since version 5.3.0?
Please add (and adjust SSH port if different than 9022)
# SSH PORT
RC_SSH_PORT=9022
into .custom/.runtime.env file for a quick fix. The 5.5.0 release would expose this as a warning, so the error is not silent like in the 5.3.0
rhodecode-support:
Please add (and adjust SSH port if different than 9022)
# SSH PORT
RC_SSH_PORT=9022
into .custom/.runtime.env file for a quick fix. The 5.5.0 release would expose this as a warning, so the error is not silent like in the 5.3.0
OK, this one is fixed now.
It does write it by default, odd that 5.4.0 didnāt itās part of our test suite now. Could it be that there was already existing .runtime.env file so the new init didnāt overwrite it ?
My bad, I installed RhodeCode with rcstack 5.4.0 but overwrote my .runtime.env file with one coming from an install with rcstack 5.2.1.
Things are getting worse: over the last run, /var/opt/rhodecode_repo_store/ was backed up first and /etc/rhodecode/conf/ last./etc/rhodecode/conf/ and not my repositories
Here is the console output (extract):
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/format
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/hooks/
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/hooks/pre-revprop-change.tmpl
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/hooks/post-lock.tmpl
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/hooks/pre-commit
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/hooks/post-unlock.tmpl
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/hooks/pre-lock.tmpl
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/hooks/post-commit
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/hooks/start-commit.tmpl
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/hooks/pre-revprop-change
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/hooks/post-commit.tmpl
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/hooks/pre-commit.tmpl
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/hooks/pre-unlock.tmpl
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/hooks/post-revprop-change.tmpl
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/conf/
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/conf/hooks-env.tmpl
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/conf/authz
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/conf/svnserve.conf
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/conf/passwd
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/rep-cache.db
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/format
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/5
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/0
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/18
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/15
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/9
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/10
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/13
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/11
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/14
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/1
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/17
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/2
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/4
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/6
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/19
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/16
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/12
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/8
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/7
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/3
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/txn-current
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/uuid
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/write-lock
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/fsfs.conf
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/rep-cache.db-journal
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/5
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/0
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/18
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/15
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/9
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/10
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/13
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/11
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/14
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/1
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/17
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/2
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/4
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/6
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/19
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/16
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/12
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/8
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/7
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/3
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/txn-protorevs/
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/txn-current-lock
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/locks/
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/locks/9fd/
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/locks/129/
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/locks/19e/
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/locks/666/
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/fs-type
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/current
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/min-unpacked-rev
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/transactions/
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/locks/
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/locks/db-logs.lock
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/locks/db.lock
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/README.txt
backup: /etc/rhodecode/conf
t/ar: Removing leading `/' from member names
etc/rhodecode/conf/
/etc/rhodecode/conf/vcsserver.ini.replaced
/etc/rhodecode/conf/rhoderc_indexer.conf
/etc/rhodecode/conf/rhodecode.ini.replaced
/etc/rhodecode/conf/rhodecode_enterprise.license
/etc/rhodecode/conf/ca-bundle.crt
/etc/rhodecode/conf/rhodecode.ini
/etc/rhodecode/conf/gunicorn_conf_vcs.py
/etc/rhodecode/conf/svn/
/etc/rhodecode/conf/svn/mod_dav_svn.conf
/etc/rhodecode/conf/gunicorn_conf_rc.py
/etc/rhodecode/conf/.rcmetadata.json
/etc/rhodecode/conf/vcsserver.ini
/etc/rhodecode/conf/ssh/
/etc/rhodecode/conf/ssh/ssh_host_rsa_key.pub
/etc/rhodecode/conf/ssh/authorized_keys_rhodecode
/etc/rhodecode/conf/ssh/authorized_keys
/etc/rhodecode/conf/ssh/ssh_host_ed25519_key
/etc/rhodecode/conf/ssh/ssh_host_rsa_key
/etc/rhodecode/conf/ssh/ssh_host_ed25519_key.pub
/etc/rhodecode/conf/ssh/ssh_host_ecdsa_key.pub
/etc/rhodecode/conf/ssh/ssh_host_ecdsa_key
Backup created in /tmp: rc_data_dump-2023-12-22.tar.gz
We fixed this on 5.5.0 release, itās going to be released today.
Great!
In the meantime, could you confirm that the upgrade procedure described in the docs is still up-to-date?
Here is what I plan to do:
# update the rcstack script
sudo ./rcstack self-update
# update RhodeCode components
sudo ./rcstack stack-upgrade router
sudo ./rcstack stack-upgrade services
sudo ./rcstack stack-upgrade rhodecode
sudo ./rcstack stack-upgrade metrics
# clean-up
sudo docker image prune -f
Secondly, if I allow a service downtime, is there a way to not duplicate the instances during the upgrade? sudo ./rcstack stack all down then sudo ./rcstack stack-upgrade rhodecode do this by default? --no-recreate supposed to handle this?
Please see: Upgrade rcstack - RhodeCode rcstack 5.5.0 documentation
When we explained the option without scale 2x. Note: this only works on 5.5.0
Very nice!
Thank you very much for the updated documentation.
1 Like
Hi,
Also 5.5.0 is out as well
1 Like
./rcstack backup-data now works as expected with rcstack 5.5.0: it gives an archive file for each directory (as soon as I provide the new required ādestinationā parameter).
By the way, to me this āDESTINATIONā argument is a non-sense for the command backup-db
./rcstack backup-db
missing required argument: DESTINATION
usage: rcstack backup-db DESTINATION
1 Like
Hi,
can you clarify about DESTINATION ? this is used to define where the backup would be written. Instead of specifying current dir, this is now configurable via DESTINATION
OK, sorry for not being clear.
Running sudo ./rcstack backup-db /tmp gives:
creating backup: rc_db_dump-2024-01-01.sql.gz
Backup created in container volume under: /var/rc-data-dump/rc_db_dump-2024-01-01.sql.gz !
to copy this file into your host machine run:
docker cp 029d65994372:/var/rc-data-dump/rc_db_dump-2024-01-01.sql.gz /home/ubuntu/docker-rhodecode
As you may know, the backup file is also available from the host inside .custom/db_dump/, since this is a mapping on /var/rc-data-dump.
But the backup file is never made available inside the DESTINATION directory (/tmp in my example), which is a nonsense for this particular command.
Hmm, but it should push the archives to DESTINATION because of this mount actually. So under the hood it mounts the $destination_dir:/backup and the pushes archives to /backup which should create them at destination.
Isnāt that the case for you ?
Ahh sorry backup-db works different that backup-data ! Weāve just missed that you were talking about the backup-db
Thatās my point: the DESTINATION parameter for the backup-db is not used, so there shouldnāt be any parameter for this command, should there?
the backup-db we need to fix to actually run the cp into DESTINATION, so thereās no user action required as a 2nd step after backup is done
1 Like
