Issue with `rcstack backup-data`: config files not backed up

Hello,

I have 2 installations of RhodeCode with rcstack.

  1. A production install with many repositories, featuring rcstack 5.3.0 and rhodecode 5.0.0.beta21
  2. A test install with only 2 repositories, featuring rcstack 5.4.0 and rhodecode 5.0.0.beta22

If I do rcstack backup-data on the second, it says that it will pack an archive containing what it founds in /etc/rhodecode/conf/ (configuration files), which it eventually does.
If I do the same on my first install, it lists the files from /etc/rhodecode/conf/ while running, but the final archive only contains what is inside /var/opt/rhodecode_repo_store/. So I tried to self-update to rcstack 5.4.0 but the result is the same! Same thing for the search index, which is listed but not in the archive.

It seems to be a bug, but I don’t know how it is related to the number of repos or size of the backup.
Any ideas to track this down are welcome.

Justin

Hi Justin,

let us start that we really appreciate your feedback on rcstack !! Thank you for helping us improve our project.

Indeed this looks like a bug, we’ll check this and fix if needed.

Great, thank you.

Did you also notice that rcstack status and rcstack stack-status are broken since version 5.3.0?

Please add (and adjust SSH port if different than 9022)

# SSH PORT
RC_SSH_PORT=9022

into .custom/.runtime.env file for a quick fix. The 5.5.0 release would expose this as a warning, so the error is not silent like in the 5.3.0

OK, this one is fixed now.
May I suggest the installer should add the default port by itself? Version 5.4.0 apparently didn’t.

It does write it by default, odd that 5.4.0 didn’t it’s part of our test suite now. Could it be that there was already existing .runtime.env file so the new init didn’t overwrite it ?

My bad, I installed RhodeCode with rcstack 5.4.0 but overwrote my .runtime.env file with one coming from an install with rcstack 5.2.1.
Sorry for that :innocent:

Things are getting worse: over the last run, /var/opt/rhodecode_repo_store/ was backed up first and /etc/rhodecode/conf/ last.
In the end, I got an archive containing only /etc/rhodecode/conf/ and not my repositories :dizzy_face:

Here is the console output (extract):

/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/format
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/hooks/
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/hooks/pre-revprop-change.tmpl
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/hooks/post-lock.tmpl
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/hooks/pre-commit
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/hooks/post-unlock.tmpl
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/hooks/pre-lock.tmpl
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/hooks/post-commit
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/hooks/start-commit.tmpl
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/hooks/pre-revprop-change
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/hooks/post-commit.tmpl
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/hooks/pre-commit.tmpl
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/hooks/pre-unlock.tmpl
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/hooks/post-revprop-change.tmpl
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/conf/
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/conf/hooks-env.tmpl
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/conf/authz
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/conf/svnserve.conf
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/conf/passwd
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/rep-cache.db
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/format
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/5
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/0
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/18
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/15
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/9
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/10
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/13
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/11
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/14
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/1
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/17
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/2
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/4
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/6
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/19
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/16
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/12
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/8
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/7
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revprops/0/3
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/txn-current
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/uuid
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/write-lock
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/fsfs.conf
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/rep-cache.db-journal
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/5
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/0
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/18
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/15
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/9
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/10
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/13
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/11
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/14
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/1
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/17
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/2
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/4
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/6
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/19
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/16
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/12
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/8
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/7
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/revs/0/3
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/txn-protorevs/
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/txn-current-lock
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/locks/
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/locks/9fd/
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/locks/129/
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/locks/19e/
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/locks/666/
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/fs-type
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/current
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/min-unpacked-rev
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/db/transactions/
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/locks/
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/locks/db-logs.lock
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/locks/db.lock
/var/opt/rhodecode_repo_store/cg1_proto-shield_uno-r3/README.txt
backup: /etc/rhodecode/conf
t/ar: Removing leading `/' from member names
etc/rhodecode/conf/
/etc/rhodecode/conf/vcsserver.ini.replaced
/etc/rhodecode/conf/rhoderc_indexer.conf
/etc/rhodecode/conf/rhodecode.ini.replaced
/etc/rhodecode/conf/rhodecode_enterprise.license
/etc/rhodecode/conf/ca-bundle.crt
/etc/rhodecode/conf/rhodecode.ini
/etc/rhodecode/conf/gunicorn_conf_vcs.py
/etc/rhodecode/conf/svn/
/etc/rhodecode/conf/svn/mod_dav_svn.conf
/etc/rhodecode/conf/gunicorn_conf_rc.py
/etc/rhodecode/conf/.rcmetadata.json
/etc/rhodecode/conf/vcsserver.ini
/etc/rhodecode/conf/ssh/
/etc/rhodecode/conf/ssh/ssh_host_rsa_key.pub
/etc/rhodecode/conf/ssh/authorized_keys_rhodecode
/etc/rhodecode/conf/ssh/authorized_keys
/etc/rhodecode/conf/ssh/ssh_host_ed25519_key
/etc/rhodecode/conf/ssh/ssh_host_rsa_key
/etc/rhodecode/conf/ssh/ssh_host_ed25519_key.pub
/etc/rhodecode/conf/ssh/ssh_host_ecdsa_key.pub
/etc/rhodecode/conf/ssh/ssh_host_ecdsa_key
Backup created in /tmp: rc_data_dump-2023-12-22.tar.gz

We fixed this on 5.5.0 release, it’s going to be released today.

Great!

In the meantime, could you confirm that the upgrade procedure described in the docs is still up-to-date?

Here is what I plan to do:

# update the rcstack script
sudo ./rcstack self-update

# update RhodeCode components
sudo ./rcstack stack-upgrade router
sudo ./rcstack stack-upgrade services
sudo ./rcstack stack-upgrade rhodecode
sudo ./rcstack stack-upgrade metrics

# clean-up
sudo docker image prune -f

Secondly, if I allow a service downtime, is there a way to not duplicate the instances during the upgrade?
Does sudo ./rcstack stack all down then sudo ./rcstack stack-upgrade rhodecode do this by default?
Or is the CLI switch --no-recreate supposed to handle this?

Please see: Upgrade rcstack - RhodeCode rcstack 5.5.0 documentation

When we explained the option without scale 2x. Note: this only works on 5.5.0

Very nice!

Thank you very much for the updated documentation.

1 Like

Hi,

Also 5.5.0 is out as well :slight_smile:

1 Like

./rcstack backup-data now works as expected with rcstack 5.5.0: it gives an archive file for each directory (as soon as I provide the new required “destination” parameter).

By the way, to me this ‘DESTINATION’ argument is a non-sense for the command backup-db because it is never used…

./rcstack backup-db
missing required argument: DESTINATION
usage: rcstack backup-db DESTINATION
1 Like

Hi,

can you clarify about DESTINATION ? this is used to define where the backup would be written. Instead of specifying current dir, this is now configurable via DESTINATION

OK, sorry for not being clear.

Running sudo ./rcstack backup-db /tmp gives:

creating backup: rc_db_dump-2024-01-01.sql.gz
Backup created in container volume under: /var/rc-data-dump/rc_db_dump-2024-01-01.sql.gz !
to copy this file into your host machine run:
docker cp 029d65994372:/var/rc-data-dump/rc_db_dump-2024-01-01.sql.gz /home/ubuntu/docker-rhodecode

As you may know, the backup file is also available from the host inside .custom/db_dump/, since this is a mapping on /var/rc-data-dump.

But the backup file is never made available inside the DESTINATION directory (/tmp in my example), which is a nonsense for this particular command.

Hmm, but it should push the archives to DESTINATION because of this mount actually. So under the hood it mounts the $destination_dir:/backup and the pushes archives to /backup which should create them at destination.

Isn’t that the case for you ?

Ahh sorry backup-db works different that backup-data ! We’ve just missed that you were talking about the backup-db

That’s my point: the DESTINATION parameter for the backup-db is not used, so there shouldn’t be any parameter for this command, should there?

the backup-db we need to fix to actually run the cp into DESTINATION, so there’s no user action required as a 2nd step after backup is done

1 Like