Restarting of Celery images (rc_cluster_apps-celery-1) with fresh install of CE 5.9.1 on DietPi (Debian)

Support
1

I am creating a brand new RhodeCode CE in a VM using DietPi (based on Debian Trixie).

The VM is brand new. A new DietPI installation with Docker and Docker Compose installed. I start the rhodecode installation following the steps at:

Basically,

./rcstack init

Last time I tested with the absolute default settings, just to avoid problems.

Only thing I have changed is disabling the ElasticSearch and applying the temporary workaround to fix the recent Traefik issue with Docker v29. BTW, I have also posted the fix which is working in my case at Docker 29 incompatability with traefik 2.11 - #2 by ismdiego

Then, I do this.

./rcstack stack router up --detach
./rcstack stack services up --detach
./rcstack stack rhodecode up --detach

And when I test the status:

./rcstack status

The first three containers (celery-1, celery-beat-1 and rhodecode-1) are always restarting.

RCSTACK: v5.29.1, running hostname: http://rhc

CONTAINER ID   NAMES                                 IMAGE                               STATUS                          PORTS
f8515428b5ae   rc_cluster_apps-celery-1              rhodecode/rhodecode-ce:5.9.1        Restarting (1) 45 seconds ago
7ef7a74ec26d   rc_cluster_apps-celery-beat-1         rhodecode/rhodecode-ce:5.9.1        Restarting (1) 45 seconds ago
3fa31cc4c56d   rc_cluster_apps-rhodecode-1           rhodecode/rhodecode-ce:5.9.1        Restarting (1) 45 seconds ago
f8e018c41a6b   rc_cluster_apps-sshd-1                rhodecode/rhodecode-ce:5.9.1        Up 9 minutes (healthy)
08885fc3e6b5   rc_cluster_apps-svn-1                 rhodecode/rhodecode-ce:5.9.1        Up 9 minutes (healthy)
b9c34792c925   rc_cluster_apps-vcsserver-1           rhodecode/rhodecode-ce:5.9.1        Up 9 minutes (healthy)
0c2f5e6f7b40   rc_cluster_router-traefik-1           traefik:v2.11.6                     Up 9 minutes                    0.0.0.0:80->80/tcp, [::]:80->80/tcp, 0.0.0.0:443->443/tcp, [::]:443->443/tcp, 0.0.0.0:3100->3100/tcp, [::]:3100->3100/tcp, 0.0.0.0:9022->9022/tcp, [::]:9022->9022/tcp
65402026c449   rc_cluster_services-channelstream-1   channelstream/channelstream:0.7.1   Up 9 minutes (healthy)          8000/tcp
b7bd3c0d7e7a   rc_cluster_services-database-1        postgres:14.13                      Up 9 minutes (healthy)          5432/tcp
41581b08ee3b   rc_cluster_services-nginx-errors-1    nginx:1.27.0                        Up 9 minutes                    80/tcp
ca8261d490ad   rc_cluster_services-nginx-statics-1   nginx:1.27.0                        Up 9 minutes (healthy)          80/tcp
aa9b2d42d7ce   rc_cluster_services-redis-1           redis:7.2.5                         Up 9 minutes (healthy)          6379/tcp

If I check the web page, it says:

RhodeCode backend server is unreachable

I am stuck with no options… any help would be great.

Many thanks

2

Some more info, if I run the rhodecode stack without the --detach option, I can see this error:

rhodecode-1    | PermissionError: [Errno 13] Permission denied: '/var/opt/rhodecode_repo_store/.cache'

And then celery-beat-1 fails with:

celery-beat-1  | {"timestamp": "2025-11-13T19:20:22.422852+00:00", "levelname": "ERROR", "name": "celery.rhodecode.loader", "message": "Failed to bootstrap RhodeCode APP. Probably there is another error present that prevents from running pyramid app"

and the same for celery-1:

celery-1       | {"timestamp": "2025-11-13T19:20:22.423274+00:00", "levelname": "ERROR", "name": "celery.rhodecode.loader", "message": "Failed to bootstrap RhodeCode APP. Probably there is another error present that prevents from running pyramid app"

As I said, this is a brand new fresh install. So I think the permissions of /var/opt/rhodecode_repo_store should be ok in the Docker image. Maybe this is the problem?

3

Are you sure there are all recursive permissions set, and also ones that allow directory creation under the repo store ?

What is the output if you run this

./rcstack stack rhodecode up rhodecode

4

This is the output:

root@DietPi:~/docker-rhodecode# ./rcstack stack router up -d
RCSTACK: v5.29.1, running router stack command
[+] Running 1/1
 ✔ Container rc_cluster_router-traefik-1  Started                                                                                                         0.9s
root@DietPi:~/docker-rhodecode# ./rcstack stack services up -d
RCSTACK: v5.29.1, running services stack command
[+] Running 5/5
 ✔ Container rc_cluster_services-database-1       Started                                                                                                 0.7s
 ✔ Container rc_cluster_services-channelstream-1  Started                                                                                                 0.6s
 ✔ Container rc_cluster_services-redis-1          Started                                                                                                 0.6s
 ✔ Container rc_cluster_services-nginx-statics-1  Started                                                                                                 0.6s
 ✔ Container rc_cluster_services-nginx-errors-1   Started                                                                                                 0.5s
root@DietPi:~/docker-rhodecode# ./rcstack stack rhodecode up rhodecode
RCSTACK: v5.29.1, running rhodecode stack command
[+] Running 1/1
 ✔ Container rc_cluster_apps-rhodecode-1  Created                                                                                                         0.6s
Attaching to rhodecode-1
rhodecode-1  | ENTRYPOINT: Running rhodecode_http with cmd '/usr/local/bin/rhodecode_bin/bin/gunicorn'
rhodecode-1  | checking if config files needs bootstrapping
rhodecode-1  | ensuring config files have proper permissions for rhodecode user
rhodecode-1  | fixing permissions for /etc/rhodecode/conf/rhodecode.ini
rhodecode-1  | ownership of '/etc/rhodecode/conf/rhodecode.ini' retained as rhodecode:rhodecode
rhodecode-1  | mode of '/etc/rhodecode/conf/rhodecode.ini' retained as 0664 (rw-rw-r--)
rhodecode-1  | fixing permissions for /etc/rhodecode/conf/vcsserver.ini
rhodecode-1  | ownership of '/etc/rhodecode/conf/vcsserver.ini' retained as rhodecode:rhodecode
rhodecode-1  | mode of '/etc/rhodecode/conf/vcsserver.ini' retained as 0664 (rw-rw-r--)
rhodecode-1  | exposing static files...
rhodecode-1  | sending incremental file list
rhodecode-1  |
rhodecode-1  | sent 18,720 bytes  received 265 bytes  5,424.29 bytes/sec
rhodecode-1  | total size is 9,022,702  speedup is 475.25
rhodecode-1  | ENTRYPOINT: Starting rhodecode_http initial bootstrap
rhodecode-1  | Running script *.py on enterprise-1
rhodecode-1  | /home/rhodecode/venv/lib/python3.11/site-packages/pyramid/asset.py:2: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
rhodecode-1  |   import pkg_resources
rhodecode-1  | {"timestamp": "2025-11-14T16:18:39.233078+00:00", "levelname": "INFO", "name": "rhodecode.config.middleware", "message": "Pyramid app config starting", "req_id": "00000000-0000-0000-0000-000000000000"}
rhodecode-1  | {"timestamp": "2025-11-14T16:18:39.233959+00:00", "levelname": "INFO", "name": "rhodecode.config.settings_maker", "message": "logging configuration based on main .ini file", "req_id": "00000000-0000-0000-0000-000000000000"}
rhodecode-1  | Traceback (most recent call last):
rhodecode-1  |   File "/usr/local/bin/rhodecode_bin/bin/rc-ishell", line 7, in <module>
rhodecode-1  |     sys.exit(main())
rhodecode-1  |              ^^^^^^
rhodecode-1  |   File "/home/rhodecode/venv/lib/python3.11/site-packages/click/core.py", line 1161, in __call__
rhodecode-1  |     return self.main(*args, **kwargs)
rhodecode-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
rhodecode-1  |   File "/home/rhodecode/venv/lib/python3.11/site-packages/click/core.py", line 1082, in main
rhodecode-1  |     rv = self.invoke(ctx)
rhodecode-1  |          ^^^^^^^^^^^^^^^^
rhodecode-1  |   File "/home/rhodecode/venv/lib/python3.11/site-packages/click/core.py", line 1443, in invoke
rhodecode-1  |     return ctx.invoke(self.callback, **ctx.params)
rhodecode-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rhodecode-1  |   File "/home/rhodecode/venv/lib/python3.11/site-packages/click/core.py", line 788, in invoke
rhodecode-1  |     return __callback(*args, **kwargs)
rhodecode-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
rhodecode-1  |   File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/lib/rc_commands/ishell.py", line 45, in main
rhodecode-1  |     with bootstrap(ini_path) as env:
rhodecode-1  |          ^^^^^^^^^^^^^^^^^^^
rhodecode-1  |   File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/lib/pyramid_utils.py", line 44, in bootstrap
rhodecode-1  |     return pyramid_bootstrap(config_uri, request=request, options=options)
rhodecode-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rhodecode-1  |   File "/home/rhodecode/venv/lib/python3.11/site-packages/pyramid/paster.py", line 117, in bootstrap
rhodecode-1  |     app = get_app(config_uri, options=options)
rhodecode-1  |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rhodecode-1  |   File "/home/rhodecode/venv/lib/python3.11/site-packages/pyramid/paster.py", line 30, in get_app
rhodecode-1  |     return loader.get_wsgi_app(name, options)
rhodecode-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rhodecode-1  |   File "/home/rhodecode/venv/lib/python3.11/site-packages/plaster_pastedeploy/__init__.py", line 111, in get_wsgi_app
rhodecode-1  |     return loadapp(
rhodecode-1  |            ^^^^^^^^
rhodecode-1  |   File "/home/rhodecode/venv/lib/python3.11/site-packages/paste/deploy/loadwsgi.py", line 246, in loadapp
rhodecode-1  |     return loadobj(APP, uri, name=name, **kw)
rhodecode-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rhodecode-1  |   File "/home/rhodecode/venv/lib/python3.11/site-packages/paste/deploy/loadwsgi.py", line 271, in loadobj
rhodecode-1  |     return context.create()
rhodecode-1  |            ^^^^^^^^^^^^^^^^
rhodecode-1  |   File "/home/rhodecode/venv/lib/python3.11/site-packages/paste/deploy/loadwsgi.py", line 738, in create
rhodecode-1  |     return self.object_type.invoke(self)
rhodecode-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rhodecode-1  |   File "/home/rhodecode/venv/lib/python3.11/site-packages/paste/deploy/loadwsgi.py", line 136, in invoke
rhodecode-1  |     return fix_call(context.object, context.global_conf, **context.local_conf)
rhodecode-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rhodecode-1  |   File "/home/rhodecode/venv/lib/python3.11/site-packages/paste/deploy/util.py", line 61, in fix_call
rhodecode-1  |     val = callable(*args, **kw)
rhodecode-1  |           ^^^^^^^^^^^^^^^^^^^^^
rhodecode-1  |   File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/config/middleware.py", line 97, in make_pyramid_app
rhodecode-1  |     sanitize_settings_and_apply_defaults(global_config, settings)
rhodecode-1  |   File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/config/config_maker.py", line 121, in sanitize_settings_and_apply_defaults
rhodecode-1  |     settings_maker.make_setting(
rhodecode-1  |   File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/config/settings_maker.py", line 227, in make_setting
rhodecode-1  |     self.settings[key] = parser_func(input_val)
rhodecode-1  |                          ^^^^^^^^^^^^^^^^^^^^^^
rhodecode-1  |   File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/config/settings_maker.py", line 117, in _dir_func
rhodecode-1  |     os.makedirs(input_val, mode=mode, exist_ok=True)
rhodecode-1  |   File "<frozen os>", line 215, in makedirs
rhodecode-1  |   File "<frozen os>", line 225, in makedirs
rhodecode-1  | PermissionError: [Errno 13] Permission denied: '/var/opt/rhodecode_repo_store/.cache'
rhodecode-1 exited with code 1 (restarting)

The problem seems the PermissionError, as before.

I don’t know what are you referring to. I think /var/opt/rhodecode_repo_store path is inside a docker volume, as I selected the default options (repo store = internal). Anyway, how can I check if this is the case?

In the docker volumes folder of my DietPI installation, there are these ones:

drwx-----x 3 root root  4096 Nov 13 18:09 rc_cluster_services_es_data
drwx-----x 3 root root  4096 Nov 13 18:09 rc_cluster_services_mysql_data
drwx-----x 3 root root  4096 Nov 13 18:09 rc_cluster_services_pg_data
drwx-----x 3 root root  4096 Nov 13 18:09 rc_cluster_services_redis_data
drwx-----x 3 root root  4096 Nov 13 18:02 rc_datavolume
drwx-----x 3 root root  4096 Nov 13 18:02 rc_reposvolume

Thank you very much for your help. Appreciated

5

Uhm… I don’t know if you made some changes, but now the installation is working FINE :face_with_open_eyes_and_hand_over_mouth: :thinking:

I have reinstalled all from zero (again) and this time is working as expected (!?). I even tried with no default options, no problems.

I have noticed that the DietPI had a minor update today, so maybe it was their problem (?) :man_facepalming:

Sorry for disturbing you. And thanks again for the help.

6

Thanks for confirmation, we’re not sure why this happened tbh. But happy that was resolved.