Migration from Ubuntu 16.04 Rhodecode 4.17 to Docker & Rcstack on Ubuntu 24.04:
The instructions do not mention how to properly restore the mysql database inside the docker container.
After making changes to ./docker-rhodecode/config/_shared/rhodecode.ini and uploading the dump via “source”, next “./cstack stack all up -d” container “rc_cluster_apps-rhodecode-1” is constantly being restarted. The wording in the instructions - to change the files, for example, this one does not make it clear what exactly will have to be changed
Hi,
so mysql is fully supported via rcstack cli:
Here’s how to do it:
# restore backup for mysql
# copy the backup sql file, and execute cli db to run the backup restore
1) cp your_sql_backup.sql .custom/db_dump/your_sql_backup.sql
2) ./rcstack cli db
3) use rhodecode
4) source /var/rc-data-dump/your_sql_backup.sql
Have you checked why containers are being restarted in the logs ?
most probably there’s a password mismatch. Please check both
rhodecode.ini and .custom/.runtime.env file to add same password as in the backup sql
source /var/rc-data-dump/your_sql_backup.sql
this command writes copies from the archive to the current rhodecode database, respectively, I left the data for connecting the database from the new installation
how to check the possible reasons for restarting the rc_cluster_apps-rhodecode-1 container
I also found problems in other containers
rc_cluster_services-nginx-statics-1 nginx:1.27.0 Up 13 minutes (unhealthy) 80/tcp
[error] 20#20: *23 open() "/var/opt/rhodecode_static_data/robots.txt" failed (2: No such file or directory), client: 127.0.0.1, server: localhost, request: "GET /_health_static HTTP/1.1", host: "127.0.0.1"
/docker-rhodecode/.custom/static_files - there are no files here
Best way to check the problem is:
./rcstack stack rhodecode down
./rcstack stack rhodecode up (without --detached)
This way you could see the restart problems in console
sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (2058, 'Plugin caching_sha2_password could not be loaded: /nix/store/1spnd42aw8q3ab41c304gv2q4xk4v8nl-mariadb-connector-c-2.3.4/lib/plugin/caching_sha2_password.so: cannot open shared object file: No such file or directory')
Hi,
it looks like sha2 auth is not supported for old 4.X installation.
Please see to change auth type: mysql - Authentication plugin 'caching_sha2_password' cannot be loaded - Stack Overflow
or switch to 5.X which should support that ootb
Unfortunately, the migration instructions indicate that you need to upgrade to version 4.28, but how to upgrade to the latest version is not specified. I was already beginning to doubt that we would be able to switch from 4.17 to rcstack 5.X 
root@rhodecode:/opt/docker-rhodecode# ./rcstack cli set-image 5.24
setting runtime version to 5.24 into /opt/docker-rhodecode/.custom/.runtime.env
New version set to 5.24
root@rhodecode:/opt/docker-rhodecode# ./rcstack stack rhodecode up
[+] Running 8/8
! rhodecode Warning manifest for rhodecode/rhodecode-ce:5.24 not found: manifest unknown: manifest unknown 2.0s
✘ svn Error manifest for rhodecode/rhodecode-ce:5.24 not found: manifest unknown: manifest unknown 1.9s
✘ celery Error manifest for rhodecode/rhodecode-ce:5.24 not found: manifest unknown: manifest unknown 2.0s
✘ sshd Error manifest for rhodecode/rhodecode-ce:5.24 not found: manifest unknown: manifest unknown 2.0s
✘ vcsserver Error manifest for rhodecode/rhodecode-ce:5.24 not found: manifest unknown: manifest unknown 2.1s
✘ common_base Error manifest for rhodecode/rhodecode-ce:5.24 not found: manifest unknown: manifest unknown 1.9s
! rhodecode_base Warning manifest for rhodecode/rhodecode-ce:5.24 not found: manifest unknown: manifest unknown 2.1s
✘ celery-beat Error manifest for rhodecode/rhodecode-ce:5.24 not found: manifest unknown: manifest unknown 1.9s
[+] Running 0/0
⠋ Container rc_cluster_apps-vcsserver-1 Creating 0.0s
⠋ Container rc_cluster_apps-rhodecode-1 Creating 0.0s
Error response from daemon: No such image: rhodecode/rhodecode-ce:5.24
root@rhodecode:/opt/docker-rhodecode# ./rcstack cli db
*Volumes*:
attaching mysql_data under: /var/lib/mysql
attaching $PWD/.custom/db_dump under: /var/rc-data-dump
mysql: [ERROR] Found option without preceding group in config file /etc/mysql/conf.d/config-file.cnf at line 1.
mysql: [ERROR] Stopped processing the 'includedir' directive in file /etc/my.cnf at line 36.
mysql: [Warning] Using a password on the command line interface can be insecure.
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 8.0.37 MySQL Community Server - GPL
Copyright (c) 2000, 2024, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> CREATE USER 'sha2user'@'localhost' IDENTIFIED BY 'password';
ERROR 1227 (42000): Access denied; you need (at least one of) the CREATE USER privilege(s) for this operation
5.24 is not a valid version for RhodeCode. Please use 5.3.1
By design db shell runs with safe user.
Please temporary change in .custom/.runtime.env
DB_USER=root
and run the cli shell again
At the moment, I observe the following in the logs:
vcsserver-1
vcsserver-1 | File "/etc/rhodecode/conf/gunicorn_conf_vcs.py", line 352, in atoms
vcsserver-1 | 't': self.now(),
vcsserver-1 | ^^^^^^^^^^
vcsserver-1 | File "/etc/rhodecode/conf/gunicorn_conf_vcs.py", line 339, in now
vcsserver-1 | msecs = int((now - long(now)) * 1000)
vcsserver-1 | ^^^^
vcsserver-1 | NameError: name 'long' is not defined
celery-1
celery-1 | - ** ---------- .> transport: amqp://guest:**@localhost:5672//
celery-1 | - ** ---------- .> results: disabled://
celery-1 | - *** --- * --- .> concurrency: {min=2, max=10} (prefork)
celery-1 | -- ******* ---- .> task events: ON
celery-1 | --- ***** -----
celery-1 | -------------- [queues]
celery-1 | .> celery exchange=celery(direct) key=celery
celery-1 |
celery-1 |
celery-1 | [tasks]
celery-1 | . celery.accumulate
celery-1 | . celery.backend_cleanup
celery-1 | . celery.chain
celery-1 | . celery.chord
celery-1 | . celery.chord_unlock
celery-1 | . celery.chunks
celery-1 | . celery.group
celery-1 | . celery.map
celery-1 | . celery.starmap
celery-1 |
celery-1 | {"timestamp": "2024-11-13T14:13:39.202982+00:00", "levelname": "WARNING", "name": "py.warnings", "message": "/home/rhodecode/venv/lib/python3.11/site-packages/celery/worker/consumer/consumer.py:507: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine\nwhether broker connection retries are made during startup in Celery 6.0 and above.\nIf you wish to retain the existing behavior for retrying connections on startup,\nyou should set broker_connection_retry_on_startup to True.\n warnings.warn(\n", "req_id": "00000000-0000-0000-0000-000000000000"}
celery-1 | {"timestamp": "2024-11-13T14:13:39.204842+00:00", "levelname": "ERROR", "name": "celery.worker.consumer.consumer", "message": "consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.\nTrying again in 2.00 seconds... (1/100)\n", "req_id": "00000000-0000-0000-0000-000000000000"}
celery-1 | {"timestamp": "2024-11-13T14:13:41.205905+00:00", "levelname": "ERROR", "name": "celery.worker.consumer.consumer", "message": "consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.\nTrying again in 4.00 seconds... (2/100)\n", "req_id": "00000000-0000-0000-0000-000000000000"}
celery-1 | {"timestamp": "2024-11-13T14:13:45.207179+00:00", "levelname": "ERROR", "name": "celery.worker.consumer.consumer", "message": "consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.\nTrying again in 6.00 seconds... (3/100)\n", "req_id": "00000000-0000-0000-0000-000000000000"}
celery-1 | {"timestamp": "2024-11-13T14:13:51.208857+00:00", "levelname": "ERROR", "name": "celery.worker.consumer.consumer", "message": "consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.\nTrying again in 8.00 seconds... (4/100)\n", "req_id": "00000000-0000-0000-0000-000000000000"}
celery-1 | {"timestamp": "2024-11-13T14:13:59.210915+00:00", "levelname": "ERROR", "name": "celery.worker.consumer.consumer", "message": "consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.\nTrying again in 10.00 seconds... (5/100)\n", "req_id": "00000000-0000-0000-0000-000000000000"}
celery-1 | {"timestamp": "2024-11-13T14:14:09.212957+00:00", "levelname": "ERROR", "name": "celery.worker.consumer.consumer", "message": "consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.\nTrying again in 12.00 seconds... (6/100)\n", "req_id": "00000000-0000-0000-0000-000000000000"}
./rcstack status -v --configs
root@rhodecode:/opt/docker-rhodecode# ./rcstack status -v --configs
Running hostname: http://rhodecode.neurosoft.ru
SSH Port: 9022
Edition: ce:5.0.0
License: 35c0-7ffe-ecb3-5beb
rcstack:
config (in-host):
env: /opt/docker-rhodecode/.custom/.runtime.env
yaml overrides (in-host):
.custom/docker-compose-router.override.yaml
.custom/docker-compose-services.override.yaml
.custom/docker-compose-apps.override.yaml
.custom/docker-compose-metrics.override.yaml
RhodeCode:
configs (in-host):
ini: config/_shared/rhodecode.ini
gunicorn: config/_shared/gunicorn_conf_rc.py
volumes (in-container):
repo-store: /var/opt/rhodecode_repo_store
data-store: /var/opt/rhodecode_data
VCSServer:
configs (in-host):
ini: config/_shared/vcsserver.ini
gunicorn: config/_shared/gunicorn_conf_vcs.py
volumes (in-container):
repo-store: /var/opt/rhodecode_repo_store
data-store: /var/opt/rhodecode_data
Traefik (router):
ssl-certs definitions: .custom/traefik_custom/dynamic/traefik_dynamic_custom.yaml
ssl-certs dir: .custom/traefik_custom/dynamic/certs/
CONTAINER ID NAMES IMAGE STATUS PORTS
1f1b3d955002 rc_cluster_apps-celery-1 rhodecode/rhodecode-ce:5.0.0 Up 4 minutes
9b0640fde378 rc_cluster_apps-celery-beat-1 rhodecode/rhodecode-ce:5.0.0 Up 4 minutes
394f6d115165 rc_cluster_apps-rhodecode-1 rhodecode/rhodecode-ce:5.0.0 Up About a minute (health: starting)
f099de71f098 rc_cluster_apps-sshd-1 rhodecode/rhodecode-ce:5.0.0 Up 4 minutes (healthy)
d7ed8135b6b1 rc_cluster_apps-svn-1 rhodecode/rhodecode-ce:5.0.0 Up 4 minutes (healthy)
58602df2c588 rc_cluster_apps-vcsserver-1 rhodecode/rhodecode-ce:5.0.0 Up 4 minutes (healthy)
9a7aaa0d5f73 rc_cluster_metrics-grafana-1 grafana/grafana:9.5.18 Up 14 minutes 3000/tcp
d93e14154d31 rc_cluster_metrics-loki-1 grafana/loki:2.9.8 Up 20 minutes 3100/tcp
0644c0789c16 rc_cluster_metrics-node-exporter-1 prom/node-exporter:v1.8.0 Up 20 minutes 9100/tcp
395572b241e9 rc_cluster_metrics-prometheus-1 prom/prometheus:v2.51.2 Up 14 minutes 9090/tcp
5cbeb2871ed7 rc_cluster_metrics-statsd-exporter-1 prom/statsd-exporter:v0.26.1 Up 20 minutes (healthy) 9102/tcp, 9125/tcp, 9125/udp
b7fd71122400 rc_cluster_router-traefik-1 traefik:v2.11.6 Up 20 minutes 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:3100->3100/tcp, 0.0.0.0:9022->9022/tcp
158cc797266a rc_cluster_services-channelstream-1 channelstream/channelstream:0.7.1 Up 20 minutes (healthy) 8000/tcp
eec90a90b2a0 rc_cluster_services-database-mysql-1 mysql:8.0.37 Up 15 minutes 3306/tcp, 33060/tcp
63c5be6d1460 rc_cluster_services-elasticsearch-1 elasticsearch:6.8.23 Up 14 minutes (healthy) 9200/tcp, 9300/tcp
10f797cd9c90 rc_cluster_services-nginx-errors-1 nginx:1.27.0 Up 20 minutes 80/tcp
25033ae51fd7 rc_cluster_services-nginx-statics-1 nginx:1.27.0 Up 20 minutes (unhealthy) 80/tcp
653fa7253dd2 rc_cluster_services-redis-1 redis:7.2.5 Up 20 minutes (healthy) 6379/tcp
Can you remove:
and restart whole rcstack ?
I’m terribly sorry:
celery-beat-1
celery-beat-1 | . scheduler -> rc_ee.lib.celerylib.scheduler.DbScheduler
celery-beat-1 |
celery-beat-1 | . logfile -> [stderr]@%DEBUG
celery-beat-1 | . maxinterval -> 10.00 seconds (10s)
celery-beat-1 | {"timestamp": "2024-11-13T14:37:05.946924+00:00", "levelname": "INFO", "name": "celery.beat", "message": "beat: Starting...", "req_id": "00000000-0000-0000-0000-000000000000"}
celery-beat-1 | {"timestamp": "2024-11-13T14:37:05.958606+00:00", "levelname": "INFO", "name": "rhodecode.rc_ee.lib.celerylib.scheduler", "message": "running setup_schedule", "req_id": "00000000-0000-0000-0000-000000000000"}
celery-beat-1 | {"timestamp": "2024-11-13T14:37:06.138086+00:00", "levelname": "ERROR", "name": "rhodecode.rc_ee.lib.celerylib.scheduler", "message": "Could not locate a bind configured on mapper mapped class ScheduleEntry->schedule_entries, SQL expression or this Session.", "req_id": "00000000-0000-0000-0000-000000000000"}
celery-beat-1 | {"timestamp": "2024-11-13T14:37:06.645236+00:00", "levelname": "INFO", "name": "rhodecode.config.middleware", "message": "Pyramid app config starting", "req_id": "00000000-0000-0000-0000-000000000000"}
celery-beat-1 | {"timestamp": "2024-11-13T14:37:06.645462+00:00", "levelname": "INFO", "name": "rhodecode.config.settings_maker", "message": "logging configuration based on main .ini file", "req_id": "00000000-0000-0000-0000-000000000000"}
celery-beat-1 | {"timestamp": "2024-11-13T14:37:06.658755+00:00", "levelname": "INFO", "name": "rhodecode.model", "message": "RhodeCode 5.0.0 initializing db for mysql://rhodecode:***@database-mysql/rhodecode?charset=utf8", "req_id": "00000000-0000-0000-0000-000000000000"}
celery-beat-1 | {"timestamp": "2024-11-13T14:37:06.659318+00:00", "levelname": "INFO", "name": "rhodecode.config.environment", "message": "Enabled VCS backends: odict_keys(['hg', 'git', 'svn'])", "req_id": "00000000-0000-0000-0000-000000000000"}
celery-beat-1 | BDB0002 __fop_file_setup: Retry limit (100) exceeded
celery-beat-1 | {"timestamp": "2024-11-13T14:38:46.946878+00:00", "levelname": "ERROR", "name": "rhodecode.lib.rc_cache.backends", "message": "Failed to initialize db at: /var/opt/rhodecode_data/rhodecode_cache_general.db", "req_id": "00000000-0000-0000-0000-000000000000", "exc_info": "Traceback (most recent call last):\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/lib/rc_cache/backends.py\", line 159, in __init__\n super().__init__(arguments)\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/dogpile/cache/backends/file.py\", line 158, in __init__\n self._init_dbm_file()\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/dogpile/cache/backends/file.py\", line 181, in _init_dbm_file\n fh = dbm.open(self.filename, \"c\")\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib/python3.11/dbm/__init__.py\", line 95, in open\n return mod.open(file, flag, mode)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^\n_dbm.error: [Errno 17] File exists: '/var/opt/rhodecode_data/rhodecode_cache_general.db'"}
celery-beat-1 | {"timestamp": "2024-11-13T14:38:46.948552+00:00", "levelname": "ERROR", "name": "celery.rhodecode.loader", "message": "Failed to bootstrap RhodeCode APP", "req_id": "00000000-0000-0000-0000-000000000000", "exc_info": "Traceback (most recent call last):\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/lib/celerylib/loader.py\", line 161, in _init_celery\n env = bootstrap(ini_file, options=options)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/lib/pyramid_utils.py\", line 57, in bootstrap\n return pyramid_bootstrap(config_uri, request=request, options=options)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/pyramid/paster.py\", line 117, in bootstrap\n app = get_app(config_uri, options=options)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/pyramid/paster.py\", line 30, in get_app\n return loader.get_wsgi_app(name, options)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/plaster_pastedeploy/__init__.py\", line 111, in get_wsgi_app\n return loadapp(\n ^^^^^^^^\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/paste/deploy/loadwsgi.py\", line 246, in loadapp\n return loadobj(APP, uri, name=name, **kw)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/paste/deploy/loadwsgi.py\", line 271, in loadobj\n return context.create()\n ^^^^^^^^^^^^^^^^\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/paste/deploy/loadwsgi.py\", line 738, in create\n return self.object_type.invoke(self)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/paste/deploy/loadwsgi.py\", line 136, in invoke\n return fix_call(context.object, context.global_conf, **context.local_conf)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/paste/deploy/util.py\", line 61, in fix_call\n val = callable(*args, **kw)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/config/middleware.py\", line 109, in make_pyramid_app\n includeme(config)\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/config/middleware.py\", line 329, in includeme\n config.include('rhodecode.lib.rc_cache')\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/pyramid/config/__init__.py\", line 666, in include\n c(configurator)\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/lib/rc_cache/__init__.py\", line 119, in includeme\n configure_dogpile_cache(config.registry.settings)\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/lib/rc_cache/__init__.py\", line 102, in configure_dogpile_cache\n new_region.configure_from_config(settings, f'rc_cache.{namespace_name}.')\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/dogpile/cache/region.py\", line 675, in configure_from_config\n return self.configure(\n ^^^^^^^^^^^^^^^\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/dogpile/cache/region.py\", line 500, in configure\n self.backend = backend_cls.from_config_dict(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/dogpile/cache/api.py\", line 232, in from_config_dict\n return cls(\n ^^^^\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/lib/rc_cache/backends.py\", line 159, in __init__\n super().__init__(arguments)\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/dogpile/cache/backends/file.py\", line 158, in __init__\n self._init_dbm_file()\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/dogpile/cache/backends/file.py\", line 181, in _init_dbm_file\n fh = dbm.open(self.filename, \"c\")\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib/python3.11/dbm/__init__.py\", line 95, in open\n return mod.open(file, flag, mode)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^\n_dbm.error: [Errno 17] File exists: '/var/opt/rhodecode_data/rhodecode_cache_general.db'"}
celery-beat-1 | {"timestamp": "2024-11-13T14:38:46.951627+00:00", "levelname": "ERROR", "name": "celery.utils.dispatch.signal", "message": "Signal handler <function on_beat_init at 0x7a4ccf937600> raised: OSError('Failed to load pyramid ENV. Probably there is another error present that prevents from running pyramid app')", "req_id": "00000000-0000-0000-0000-000000000000", "exc_info": "Traceback (most recent call last):\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/celery/utils/dispatch/signal.py\", line 276, in send\n response = receiver(signal=self, sender=sender, **named)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/lib/celerylib/loader.py\", line 195, in on_beat_init\n _init_celery('celery beat')\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/lib/celerylib/loader.py\", line 166, in _init_celery\n raise EnvironmentError(\nOSError: Failed to load pyramid ENV. Probably there is another error present that prevents from running pyramid app"}
celery-1
celery-1 | - ** ---------- .> results: disabled://
celery-1 | - *** --- * --- .> concurrency: {min=2, max=10} (prefork)
celery-1 | -- ******* ---- .> task events: ON
celery-1 | --- ***** -----
celery-1 | -------------- [queues]
celery-1 | .> celery exchange=celery(direct) key=celery
celery-1 |
celery-1 |
celery-1 | [tasks]
celery-1 | . celery.accumulate
celery-1 | . celery.backend_cleanup
celery-1 | . celery.chain
celery-1 | . celery.chord
celery-1 | . celery.chord_unlock
celery-1 | . celery.chunks
celery-1 | . celery.group
celery-1 | . celery.map
celery-1 | . celery.starmap
celery-1 |
celery-1 | {"timestamp": "2024-11-13T14:38:48.556201+00:00", "levelname": "WARNING", "name": "py.warnings", "message": "/home/rhodecode/venv/lib/python3.11/site-packages/celery/worker/consumer/consumer.py:507: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine\nwhether broker connection retries are made during startup in Celery 6.0 and above.\nIf you wish to retain the existing behavior for retrying connections on startup,\nyou should set broker_connection_retry_on_startup to True.\n warnings.warn(\n", "req_id": "00000000-0000-0000-0000-000000000000"}
celery-1 | {"timestamp": "2024-11-13T14:38:48.558053+00:00", "levelname": "ERROR", "name": "celery.worker.consumer.consumer", "message": "consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.\nTrying again in 2.00 seconds... (1/100)\n", "req_id": "00000000-0000-0000-0000-000000000000"}
celery-1 | {"timestamp": "2024-11-13T14:38:50.558945+00:00", "levelname": "ERROR", "name": "celery.worker.consumer.consumer", "message": "consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.\nTrying again in 4.00 seconds... (2/100)\n", "req_id": "00000000-0000-0000-0000-000000000000"}
celery-1 | {"timestamp": "2024-11-13T14:38:54.560244+00:00", "levelname": "ERROR", "name": "celery.worker.consumer.consumer", "message": "consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.\nTrying again in 6.00 seconds... (3/100)\n", "req_id": "00000000-0000-0000-0000-000000000000"}
celery-1 | {"timestamp": "2024-11-13T14:39:00.561830+00:00", "levelname": "ERROR", "name": "celery.worker.consumer.consumer", "message": "consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.\nTrying again in 8.00 seconds... (4/100)\n", "req_id": "00000000-0000-0000-0000-000000000000"}
celery-1 | {"timestamp": "2024-11-13T14:39:08.563541+00:00", "levelname": "ERROR", "name": "celery.worker.consumer.consumer", "message": "consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.\nTrying again in 10.00 seconds... (5/100)\n", "req_id": "00000000-0000-0000-0000-000000000000"}
celery-1 | {"timestamp": "2024-11-13T14:39:18.565498+00:00", "levelname": "ERROR", "name": "celery.worker.consumer.consumer", "message": "consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.\nTrying again in 12.00 seconds... (6/100)\n", "req_id": "00000000-0000-0000-0000-000000000000"}
celery-1 | {"timestamp": "2024-11-13T14:39:30.567881+00:00", "levelname": "ERROR", "name": "celery.worker.consumer.consumer", "message": "consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.\nTrying again in 14.00 seconds... (7/100)\n", "req_id": "00000000-0000-0000-0000-000000000000"}
celery-1 | {"timestamp": "2024-11-13T14:39:44.570324+00:00", "levelname": "ERROR", "name": "celery.worker.consumer.consumer", "message": "consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.\nTrying again in 16.00 seconds... (8/100)\n", "req_id": "00000000-0000-0000-0000-000000000000"}
celery-1 | {"timestamp": "2024-11-13T14:40:00.572749+00:00", "levelname": "ERROR", "name": "celery.worker.consumer.consumer", "message": "consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.\nTrying again in 18.00 seconds... (9/100)\n", "req_id": "00000000-0000-0000-0000-000000000000"}
celery-1 | {"timestamp": "2024-11-13T14:40:18.575978+00:00", "levelname": "ERROR", "name": "celery.worker.consumer.consumer", "message": "consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.\nTrying again in 20.00 seconds... (10/100)\n", "req_id": "00000000-0000-0000-0000-000000000000"}
celery-1 | {"timestamp": "2024-11-13T14:40:38.579365+00:00", "levelname": "ERROR", "name": "celery.worker.consumer.consumer", "message": "consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.\nTrying again in 22.00 seconds... (11/100)\n", "req_id": "00000000-0000-0000-0000-000000000000"}
celery-1 | {"timestamp": "2024-11-13T14:41:00.583128+00:00", "levelname": "ERROR", "name": "celery.worker.consumer.consumer", "message": "consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.\nTrying again in 24.00 seconds... (12/100)\n", "req_id": "00000000-0000-0000-0000-000000000000"}
celery-1 | {"timestamp": "2024-11-13T14:41:24.587177+00:00", "levelname": "ERROR", "name": "celery.worker.consumer.consumer", "message": "consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.\nTrying again in 26.00 seconds... (13/100)\n", "req_id": "00000000-0000-0000-0000-000000000000"}
rc_cluster_services-nginx-statics-1
root@rhodecode:/opt/docker-rhodecode# docker logs rc_cluster_services-nginx-statics-1
2024/11/13 13:56:13 [notice] 1#1: using the "epoll" event method
2024/11/13 13:56:13 [notice] 1#1: nginx/1.27.0
2024/11/13 13:56:13 [notice] 1#1: built by gcc 12.2.0 (Debian 12.2.0-14)
2024/11/13 13:56:13 [notice] 1#1: OS: Linux 6.8.0-48-generic
2024/11/13 13:56:13 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2024/11/13 13:56:13 [notice] 1#1: start worker processes
2024/11/13 13:56:13 [notice] 1#1: start worker process 21
2024/11/13 13:56:13 [notice] 1#1: start worker process 22
2024/11/13 13:56:18 [error] 21#21: *1 open() "/var/opt/rhodecode_static_data/robots.txt" failed (2: No such file or directory), client: 127.0.0.1, server: localhost, request: "GET /_health_static HTTP/1.1", host: "127.0.0.1"
2024/11/13 13:56:23 [error] 22#22: *2 open() "/var/opt/rhodecode_static_data/robots.txt" failed (2: No such file or directory), client: 127.0.0.1, server: localhost, request: "GET /_health_static HTTP/1.1", host: "127.0.0.1"
2024/11/13 13:56:28 [error] 21#21: *3 open() "/var/opt/rhodecode_static_data/robots.txt" failed (2: No such file or directory), client: 127.0.0.1, server: localhost, request: "GET /_health_static HTTP/1.1", host: "127.0.0.1"
2024/11/13 13:56:33 [error] 21#21: *4 open() "/var/opt/rhodecode_static_data/robots.txt" failed (2: No such file or directory), client: 127.0.0.1, server: localhost, request: "GET /_health_static HTTP/1.1", host: "127.0.0.1"
2024/11/13 13:56:38 [error] 21#21: *5 open() "/var/opt/rhodecode_static_data/robots.txt" failed (2: No such file or directory), client: 127.0.0.1, server: localhost, request: "GET /_health_static HTTP/1.1", host: "127.0.0.1"
2024/11/13 13:56:43 [error] 21#21: *6 open() "/var/opt/rhodecode_static_data/robots.txt" failed (2: No such file or directory), client: 127.0.0.1, server: localhost, request: "GET /_health_static HTTP/1.1", host: "127.0.0.1"
2024/11/13 13:56:48 [error] 21#21: *7 open() "/var/opt/rhodecode_static_data/robots.txt" failed (2: No such file or directory), client: 127.0.0.1, server: localhost, request: "GET /_health_static HTTP/1.1", host: "127.0.0.1"
2024/11/13 13:56:54 [error] 21#21: *8 open() "/var/opt/rhodecode_static_data/robots.txt" failed (2: No such file or directory), client: 127.0.0.1, server: localhost, request: "GET /_health_static HTTP/1.1", host: "127.0.0.1"
2
Can you comment on celery-1? 
Please make sure you have set:
use_celery = true
; connection url to the message broker (default redis)
celery.broker_url = redis://redis:6379/8
; results backend to get results for (default redis)
celery.result_backend = redis://redis:6379/8
inside rhodecode.ini
these parameters were the same from the very beginning when installing the docker version. They are the same now.
There are still such errors:
rhodecode-1 | {"timestamp": "2024-11-17T08:35:28.826920+00:00", "levelname": "ERROR", "name": "rhodecode.lib.rc_cache.backends", "message": "Failed to initialize db at: /var/opt/rhodecode_data/rhodecode_cache_general.db", "req_id": "00000000-0000-0000-0000-000000000000", "exc_info": "Traceback (most recent call last):\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/lib/rc_cache/backends.py\", line 159, in __init__\n super().__init__(arguments)\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/dogpile/cache/backends/file.py\", line 158, in __init__\n self._init_dbm_file()\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/dogpile/cache/backends/file.py\", line 181, in _init_dbm_file\n fh = dbm.open(self.filename, \"c\")\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib/python3.11/dbm/__init__.py\", line 95, in open\n return mod.open(file, flag, mode)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^\n_dbm.error: [Errno 17] File exists: '/var/opt/rhodecode_data/rhodecode_cache_general.db'"}
sshd-1 | Connection closed by authenticating user rhodecode 127.0.0.1 port 41612 [preauth]
This file did not exist and I created it with a command inside the container:
touch /var/opt/rhodecode_data/rhodecode_cache_general.db
After this container restarted for the first time and started working, the interface loaded and I was able to log in using my Active Directory account. One when going to some repositories on the web or an example in example.com/_admin/audit_logs I get a server error: > RhodeCode
500 Internal Server Error
The server has either error or is incapable of performing the requested operation.
Exceptions Tracker - Exception ID:
example.com/_admin/audit_logs
Traceback (most recent call last):
File "/home/rhodecode/venv/lib/python3.11/site-packages/pyramid_mako/__init__.py", line 148, in __call__
result = template.render_unicode(**system)
File "/home/rhodecode/venv/lib/python3.11/site-packages/mako/template.py", line 444, in render_unicode
return runtime._render(
File "/home/rhodecode/venv/lib/python3.11/site-packages/mako/runtime.py", line 874, in _render
_render_context(
File "/home/rhodecode/venv/lib/python3.11/site-packages/mako/runtime.py", line 916, in _render_context
_exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
File "/home/rhodecode/venv/lib/python3.11/site-packages/mako/runtime.py", line 943, in _exec_template
callable_(context, *args, **kwargs)
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/templates/base/root.mako", line 159, in render_body
${next.body()}
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/templates/base/base.mako", line 38, in render_body
${next.main()}
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/templates/admin/admin_audit_logs.mako", line 34, in render_main
<%include file="/admin/admin_log_base.mako" />
File "/home/rhodecode/venv/lib/python3.11/site-packages/mako/runtime.py", line 793, in _include_file
callable_(ctx, **kwargs)
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/templates/admin/admin_log_base.mako", line 22, in render_body
${base.gravatar_with_user(l.user.email)}
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/templates/base/base.mako", line 245, in render_gravatar_with_user
email = h.email_or_none(contact)
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/lib/helpers.py", line 959, in email_or_none
user = User.get_by_username(
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/model/db.py", line 1097, in get_by_username
hash_key = _hash_key(username)
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/model/db.py", line 128, in _hash_key
return sha1_safe(k)
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/lib/hash_utils.py", line 40, in sha1_safe
val = sha1(safe_bytes(s))
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/lib/str_utils.py", line 114, in safe_bytes
raise ValueError(f'safe_bytes cannot convert other types than str: got: {type(str_)}')
ValueError: safe_bytes cannot convert other types than str: got: <class 'NoneType'>
going to some repositories
++ TRACEBACK ++
Traceback (most recent call last):
File "/home/rhodecode/venv/lib/python3.11/site-packages/pyramid/tweens.py", line 41, in excview_tween
response = handler(request)
^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/pyramid/router.py", line 143, in handle_request
response = _call_view(
^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/pyramid/view.py", line 674, in _call_view
response = view_callable(context, request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/pyramid/config/views.py", line 170, in attr_view
return view(context, request)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/pyramid/config/views.py", line 196, in predicate_wrapper
return view(context, request)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/pyramid/viewderivers.py", line 427, in rendered_view
result = view(context, request)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/pyramid/viewderivers.py", line 129, in _class_view
response = getattr(inst, attr)()
^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/lib/auth.py", line 2534, in local_wrapper
return wrapper(func, *args, **kwds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/lib/auth.py", line 1846, in __wrapper
return func(*fargs, **fkwargs)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/lib/auth.py", line 2534, in local_wrapper
return wrapper(func, *args, **kwds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/lib/auth.py", line 1920, in __wrapper
return func(*fargs, **fkwargs)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/apps/repository/views/repo_summary.py", line 136, in summary
self._get_readme_data(self.db_repo, c.visual.default_renderer)
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/apps/_base/__init__.py", line 452, in _get_readme_data
readme_data, readme_filename = generate_repo_readme(
^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/decorator.py", line 232, in fun
return caller(func, *(extras + args), **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/lib/rc_cache/utils.py", line 85, in get_or_create_for_user_func
return self.get_or_create(key, user_func, timeout, should_cache_fn, (arg, kw))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/dogpile/cache/region.py", line 1094, in get_or_create
with Lock(
File "/home/rhodecode/venv/lib/python3.11/site-packages/dogpile/lock.py", line 185, in __enter__
return self._enter()
^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/dogpile/lock.py", line 94, in _enter
generated = self._enter_create(value, createdtime)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/dogpile/lock.py", line 178, in _enter_create
return self.creator()
^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/dogpile/cache/region.py", line 1048, in gen_value
created_value = creator(
^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/apps/_base/__init__.py", line 423, in generate_repo_readme
readme_node = ReadmeFinder(_renderer_type).search(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/model/repo.py", line 1142, in search
if commit.get_node(path).kind != NodeKind.DIR:
^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/lib/vcs/backends/hg/commit.py", line 329, in get_node
if path in self._file_paths:
^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/zope/cachedescriptors/property.py", line 109, in __get__
value = func(inst)
^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/lib/vcs/backends/hg/commit.py", line 129, in _file_paths
return self._remote.ctx_list(self.raw_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/lib/vcs/client_http.py", line 258, in repo_remote_attr
return self._call(name, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/lib/vcs/exceptions.py", line 189, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/lib/vcs/client_http.py", line 321, in _call
result = remote_call(cache_key)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/decorator.py", line 232, in fun
return caller(func, *(extras + args), **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/lib/rc_cache/utils.py", line 85, in get_or_create_for_user_func
return self.get_or_create(key, user_func, timeout, should_cache_fn, (arg, kw))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/dogpile/cache/region.py", line 1094, in get_or_create
with Lock(
File "/home/rhodecode/venv/lib/python3.11/site-packages/dogpile/lock.py", line 185, in __enter__
return self._enter()
^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/dogpile/lock.py", line 94, in _enter
generated = self._enter_create(value, createdtime)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/dogpile/lock.py", line 178, in _enter_create
return self.creator()
^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/dogpile/cache/region.py", line 1048, in gen_value
created_value = creator(
^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/lib/vcs/client_http.py", line 319, in remote_call
return _remote_call(url, payload, EXCEPTIONS_MAP, self._session)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/lib/vcs/client_http.py", line 86, in _remote_call
response = msgpack.unpackb(response.content)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "msgpack/_unpacker.pyx", line 194, in msgpack._cmsgpack.unpackb
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0 in position 0: invalid continuation byte
This is probably due to the file names that are located in the repository, they have an invalid encoding for the rhodecode docker version, how to get around this problem and the problem described above?
Summary
/home/rhodecode/venv/lib/python3.11/site-packages/pyramid/tweens.py:41 in excview_tween │
│ │
│ 38 │ │
│ 39 │ def excview_tween(request): │
│ 40 │ │ try: │
│ ❱ 41 │ │ │ response = handler(request) │
│ 42 │ │ except Exception as exc: │
│ 43 │ │ │ response = _error_handler(request, exc) │
│ 44 │ │ return response │
│ │
│ ╭───────────────────────────────────────────────────────────────────────── locals ─────────────────────────────────────────────────────────────────────────╮ │
│ │ exc = UnicodeDecodeError('utf-8', b'\xd0\xf3\xea\xee\xe2\xee\xe4\xf1\xf2\xe2\xee \xef\xee \xf1\xee\xe7\xe4\xe0\xed\xe8\xfe │ │
│ │ \xe8\xed\xf1\xf2\xe0\xeb\xeb\xff\xf2\xee\xf0\xe0.docx', 0, 1, 'invalid continuation byte') │ │
│ │ handler = <bound method Router.handle_request of <pyramid.router.Router object at 0x7e59c05bc610>> │ │
│ │ request = <Request at 0x7e59b3774450 GET http://rhodecode.example.com/InstallerTemplate>
Hi,
can you try to clear up REDIS instance and sessions data ?
Have been completed:
Settings - User Sessions - Cleanup Old Sessions - Picking All will log-out you, and all users in the system. Cleanup sessions button
And
Settings - Remap and Rescan - Import New Groups or Repositories - Destroy old data & Invalidate cache for all repositories - button Rescan Filesystem