Installation difficulties: Restarting containers and failed rootless attempts

1

Hello,

I’ve been trying to evaluate RhodeCode off and on as time permits and I can’t even get it running. My hope was to purchase an EE license for a team of 5 full time developers and a handful of part time ones for our office. The ability to host SVN and Git in a unified application while we migrate off our existing SVN repositories is very appealing and the code review features look great.

I first evaluated RhodeCode about a year and a half ago and did manage to get it running with some effort. I unfortunately don’t recall the issues I ran into at the time, but do recall needing to reach out to RhodeCode support. Unfortunately at that time, high priority projects came up and I was forced to abandon the evaluation.

I have since tried installing it half a dozen or so times over the last couple years as timed permitted. I have never been able to get it running since then. These installation attempts were in WSL2 Debian 13 in Windows 11 and on a Debian 13 server on actual hardware.

Issues I have run into:

  1. The rhodecode stack containers get stuck restarting after a fresh install. Attempting to access the URL to bring up the page gives me a single line “404 not found”.
  2. Today, attempting to troubleshoot the above:
    1. docker log for the restarting rhodecode containers gives no output.
    2. Another forum post here gave me the idea to just run rcstack stack rhodecode up without --detach. I was able to watch the output there. Every restart would fail due to a permissions error with writing to my storage path /var/opt/rhodecode. This is the same issue as was reported by this post. I am also running Debian. An apt upgrade did not resolve the issue nor did the subsequent numerous reinstall attempts.
  3. To rule out my environment, my last installation attempt was done running everything as root. I created /opt/rhodecode as root, created/var/opt/rhodecode as root, ensured both folders were empty, and ran the following command from /opt/rhodecode:
    ```
    sudo ./rcstack init --arg-accept-eula --arg-rc-edition ce --arg-performance-profile minimal --arg-hostname myhost --arg-storage-path /var/opt/rhodecode --arg-database-type postgres --arg-use-ssl n --arg-admin-email my@email.com --arg-admin-user admin --install-docker n
    ```
    1. Everything went fine, it seemed. So I sudo ./rcstack stack all up -d.
      Everything appeared to work but rc_cluster_apps-celery, celery-beat, and rhodecode containers were all stuck in restart loops. Watching the output indicated the permissions errors I mentioned above.
  4. Prior to that, today, I had also tried running rhodecode rootless in docker (which is my actual goal). Starting from a clean slate, I tried to use rcstack as the rhodecode user which was configured for linger and had subuid and subgid mappings in the 200000 range. The /opt/rhodecode and /var/opt/rhodecode folders were both owned by rhodecode user at the time.
    1. The rcstack init script failed when trying to chown /var/opt/rhodecode to 999:999. I want to note that my subuid and subgid mapping for rootless containers were rhodecode:200000:65536, so I believe it actually should actually be chown 200999:200999.
    2. I commented out the chown, set the owner manually, and ran init again, with the same parameters, and it seemed to work. Except rcstack forgot my storage path settings and defaulted back to ‘internal’.
    3. So I wiped everything and ran init again, this time with the chown command commented out ahead of time. Everything appearing to work, but the rhodecode stack containers were stuck in a restart loop as described above.
  5. I also wanted to use my own external postgres database server, but:
    1. The rcstack documentation states that to use an external database server, simply inform rcstack that I’m using postgres and I will be prompted for the database credentials and location. That prompt never happened. rcstack still configured itself for the internal postgres container.
    2. Following the instructions on setting up the external postgres sql in the rcstack documentation described how to disable the postgres container and set the connection string in rhodecode.ini. It also noted that I would need to manually manage the database. That’s fine except:
      1. I could find no information in the rcstack documentation, the rhodecode documentation, the community, or google about how to do that. I couldn’t find anything about how to create the initial schema Rhodecode would need or whther rhodecode would take care of that on first run, or even what permissions the rhodecode database user would need for rhodecode to access the exterrnal database.
      2. The rcstack cli db and db-upgrade commands, which I assumed would assist with this, fail stating that the postgres container wasn’t running.
  6. Less of an issue, but just wanted to mention it, a few months ago I also tried using podman. This is when I learned that the community claims of podman being a drop-in replacement for docker is a lie. Not your fault, but podman support would be nice.

There are also some documentation gaps that, IMO, led me to spending so much time trying to get this working:

  • The documentation is split between rcstack and rhodecode and I find myself jumping between the two constantly. For example
    • The rhodecode install documentation says that you should install as a specific user. The rcstack install documentation doesn’t mention this at all. This is essentially what I did when I tried to run rhodecode as a rootless docker container and this is when I ran into the chown permissions problem.
    • The rhodecode documentation has information on configuring the database connection. This is also in the rcstack documentation, but more detailed. The rhodecode documentation doesn’t mention this at all.
  • As mentioned above, the rcstack external database documentation says I need to manage the database myself if I choose to use an external database but provides no guidance that I can find on how to do create the database, upgrade the database, or what user permissions are necessary for the rhodecode database user.
  • The post installation steps mention needing to configure celery and install rabbitmq or redis before you can use it. I’m confused about why this shows up after bringing the stack online and what the celery containers are for if I was supposed to configure this manually. What functions of Rhodecode are impaired if I don’t configure it?
  • There’s no information about how to get rhodecode to start itself up on system startup. I wasn’t super familiar with Docker when I started this process, so it took me some time to find that docker handles this. It would be nice if the rhodecode or rcstack documentation could point users like me in that direction.
  • The arg-performance-profile argument isn’t documented in the rcstack documentation. The help in rcstack help only says the allowed values are minimal, balanced, or optimized. What do those actually mean? What do they change? How many users/repos determine which I need? What are the hardware requirements of each?
  • The dev-tools argument just says it installs the dev-tools and full stack. I had to do more searching to find out it’s not a flag that applies to my situation and is for developing rhodecode itself.
  • The post install steps in the rhodecode documentation states that I will be presented with choices to migrate existing repositories, but this never came up when using rcstack init to install rhodecode.
  • The issue tracker integration says to see the “Jira Integration” section. The “Jira Integration” section just says “Please check examples in the view for configuration the issue trackers.” What view? Where? Is this for Jira Cloud or Data Center?
  • There’s no information about troubleshooting. As I said above, the docker logs command for the containers was always empty. The rcstack documentation about Logging Configuration gives me information about the logging stack used and how to configure it, but what do I look at if I get the 404 error I’m getting? What do I look at when a rhodecode container just keeps restarting? Where do I look if I’m having problems with my SSL certificates? Or database connection/permissions issues? Or storage location permissions issues?

I really like what I see about RhodeCode. Enough that I have put more time into this than I usually would any other application. But I am at the point where I need to decide if it’s worth fighting anymore and look at alternatives like gitlab, azure devops, or stick with gitea despite its limitations.

So if I’m missing something obvious, any guidance would be greatly appreciated.