Hey !
Thanks for confirmations.
We believe the problem might be with routing to RhodeCode instance, can you double check your set domain in
.custom/.runtime.env vs what you enter in the browser ?
as an alternative, in the override file you can open a dedicated port for RhodeCode to reach it directly bypassing traefik router.
Uncomment in .custom/docker-compose-apps.override.yaml
# ports:
# # expose port for optional direct port connection
# - "10020:10020"
Good catch, I was using the IP address. However, I did verify the DNS name used during setup, entered the IP/Name into the local DNS server, and used the DNS name in the browser, but still got the same error message.
After un-commenting the ports: field and rebooting the Docker host, attempts to load the page result in a “This site can’t be reached” message (ERR_CONNECTION_REFUSED).
Something interesting: Executing ./rcstack status AND netstat -t on the Docker host does not show any service listening on port 10020.
the direct port is not on https right ?
Correct, to reduce complexity, https was not setup.
But again, whichever service is supposed to be listening to port 10020 (nginx?), is not listening. netstat -t doesn’t show anything listening to port 10020.
RhodeCode itself, if you uncomment that override file port 10020 is exposed that allows a direct connection to RhodeCode instance.
just make sure you run ./rcstack stack-all down && ./rcstack stack-all up -d
Yes, as mentioned I did uncomment the ports: directive, and for good measure rebooted the Docker host, but still receive the same “Rhodecode backend server unreachable” message even when attempting to connect at port 10020.
Also, when running netstat -t it appears that Traefik is not listening to port 10020.
So let me ask one last question. Is there any difference between running Rhodecode on Alpine vs Debian? Could that be the reason I am, presumably, seeing different behavior than you guys?
no, it’s all Docker based images and setup, it should work independently on Ubuntu/Debian/Redhat or Alpine.
But we must say we did use alpine for base images, and ditched that to using ubuntu because of some alpine weirdness. It’s minimalistic OS and we found some issues, but that was running our stack raw inside Alpine.
Maybe we missed something in the setup, but does 10020 port show up in ./rcstack status page ?
@Seann btw, can you run
./rcstack stack router logs -f while accessing the host ? maybe there’s something important printed there that we’re missing
Unfortunately I cannot capture helpful info, because Traefik is generating an overwhelming amount of data:
rc_cluster_router-traefik-1 | 172.18.0.1 - - [01/Jun/2023:02:47:11 +0000] "POST /loki/api/v1/push HTTP/1.1" - - "-" "-" 15135 "-" "-" 0ms
rc_cluster_router-traefik-1 | 172.18.0.1 - - [01/Jun/2023:02:47:12 +0000] "POST /loki/api/v1/push HTTP/1.1" - - "-" "-" 15136 "-" "-" 0ms
rc_cluster_router-traefik-1 | 172.18.0.1 - - [01/Jun/2023:02:47:13 +0000] "POST /loki/api/v1/push HTTP/1.1" - - "-" "-" 15137 "-" "-" 0ms
rc_cluster_router-traefik-1 | 172.18.0.1 - - [01/Jun/2023:02:47:15 +0000] "POST /loki/api/v1/push HTTP/1.1" - - "-" "-" 15138 "-" "-" 0ms
rc_cluster_router-traefik-1 | 172.18.0.1 - - [01/Jun/2023:02:47:15 +0000] "POST /loki/api/v1/push HTTP/1.1" - - "-" "-" 15139 "-" "-" 0ms
rc_cluster_router-traefik-1 | 172.18.0.1 - - [01/Jun/2023:02:47:19 +0000] "POST /loki/api/v1/push HTTP/1.1" - - "-" "-" 15140 "-" "-" 0ms
rc_cluster_router-traefik-1 | 172.18.0.1 - - [01/Jun/2023:02:47:19 +0000] "POST /loki/api/v1/push HTTP/1.1" - - "-" "-" 15141 "-" "-" 0ms
rc_cluster_router-traefik-1 | 172.18.0.1 - - [01/Jun/2023:02:47:22 +0000] "POST /loki/api/v1/push HTTP/1.1" - - "-" "-" 15142 "-" "-" 0ms
rc_cluster_router-traefik-1 | 172.18.0.1 - - [01/Jun/2023:02:47:25 +0000] "POST /loki/api/v1/push HTTP/1.1" - - "-" "-" 15143 "-" "-" 0ms
rc_cluster_router-traefik-1 | 172.18.0.1 - - [01/Jun/2023:02:47:27 +0000] "POST /loki/api/v1/push HTTP/1.1" - - "-" "-" 15144 "-" "-" 0ms
rc_cluster_router-traefik-1 | 172.18.0.1 - - [01/Jun/2023:02:47:29 +0000] "POST /loki/api/v1/push HTTP/1.1" - - "-" "-" 15145 "-" "-" 0ms
rc_cluster_router-traefik-1 | 172.18.0.1 - - [01/Jun/2023:02:47:31 +0000] "POST /loki/api/v1/push HTTP/1.1" - - "-" "-" 15146 "-" "-" 0ms
rc_cluster_router-traefik-1 | 172.18.0.1 - - [01/Jun/2023:02:47:34 +0000] "POST /loki/api/v1/push HTTP/1.1" - - "-" "-" 15147 "-" "-" 0ms
rc_cluster_router-traefik-1 | 172.18.0.1 - - [01/Jun/2023:02:47:37 +0000] "POST /loki/api/v1/push HTTP/1.1" - - "-" "-" 15148 "-" "-" 0ms
rc_cluster_router-traefik-1 | 172.18.0.1 - - [01/Jun/2023:02:47:40 +0000] "POST /loki/api/v1/push HTTP/1.1" - - "-" "-" 15149 "-" "-" 0ms
rc_cluster_router-traefik-1 | 172.18.0.1 - - [01/Jun/2023:02:47:42 +0000] "POST /loki/api/v1/push HTTP/1.1" - - "-" "-" 15150 "-" "-" 0ms
rc_cluster_router-traefik-1 | 172.18.0.1 - - [01/Jun/2023:02:47:42 +0000] "POST /loki/api/v1/push HTTP/1.1" - - "-" "-" 15151 "-" "-" 0ms
rc_cluster_router-traefik-1 | 172.18.0.1 - - [01/Jun/2023:02:47:43 +0000] "POST /loki/api/v1/push HTTP/1.1" - - "-" "-" 15152 "-" "-" 0ms
rc_cluster_router-traefik-1 | 172.18.0.1 - - [01/Jun/2023:02:47:44 +0000] "POST /loki/api/v1/push HTTP/1.1" - - "-" "-" 15153 "-" "-" 0ms
rc_cluster_router-traefik-1 | 172.18.0.1 - - [01/Jun/2023:02:47:45 +0000] "POST /loki/api/v1/push HTTP/1.1" - - "-" "-" 15154 "-" "-" 0ms
rc_cluster_router-traefik-1 | 172.18.0.1 - - [01/Jun/2023:02:47:46 +0000] "POST /loki/api/v1/push HTTP/1.1" - - "-" "-" 15155 "-" "-" 0ms
rc_cluster_router-traefik-1 | 172.18.0.1 - - [01/Jun/2023:02:47:47 +0000] "POST /loki/api/v1/push HTTP/1.1" - - "-" "-" 15156 "-" "-" 0ms
rc_cluster_router-traefik-1 | 172.18.0.1 - - [01/Jun/2023:02:47:50 +0000] "POST /loki/api/v1/push HTTP/1.1" - - "-" "-" 15157 "-" "-" 0ms
rc_cluster_router-traefik-1 | 172.18.0.1 - - [01/Jun/2023:02:47:53 +0000] "POST /loki/api/v1/push HTTP/1.1" - - "-" "-" 15158 "-" "-" 0ms
rc_cluster_router-traefik-1 | 172.18.0.1 - - [01/Jun/2023:02:47:55 +0000] "POST /loki/api/v1/push HTTP/1.1" - - "-" "-" 15159 "-" "-" 0ms
rc_cluster_router-traefik-1 | 172.18.0.1 - - [01/Jun/2023:02:47:57 +0000] "POST /loki/api/v1/push HTTP/1.1" - - "-" "-" 15160 "-" "-" 0ms
rc_cluster_router-traefik-1 | 172.18.0.1 - - [01/Jun/2023:02:47:57 +0000] "POST /loki/api/v1/push HTTP/1.1" - - "-" "-" 15161 "-" "-" 0ms
error from daemon in stream: Error grabbing logs: invalid character '\x00' looking for beginning of value
As mentioned previously, ./rcstack status and netstat -t do not show any service listening on port 10020, even though the services: directive is un-commented in .custom/docker-compose-apps.override.yaml.
CONTAINER ID NAMES IMAGE STATUS PORTS
5e7d9dedd06a rc_cluster_apps-celery-1 rhodecode/rhodecode-ce:4.28.0 Up 54 minutes
64f8f6901187 rc_cluster_apps-celery-beat-1 rhodecode/rhodecode-ce:4.28.0 Up 54 minutes
2ba9045fe044 rc_cluster_apps-rhodecode-1 rhodecode/rhodecode-ce:4.28.0 Restarting (1) Less than a second ago
d91742edac70 rc_cluster_apps-sshd-1 rhodecode/rhodecode-ce:4.28.0 Up 54 minutes (healthy)
440affeb0b2c rc_cluster_apps-svn-1 rhodecode/rhodecode-ce:4.28.0 Up 54 minutes (healthy)
f76c332ede8a rc_cluster_apps-vcsserver-1 rhodecode/rhodecode-ce:4.28.0 Restarting (1) 11 seconds ago
2f3d3cd9fc83 rc_cluster_router-traefik-1 traefik:v2.10.1 Up 54 minutes 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp, 0.0.0.0:3100->3100/tcp, :::3100->3100/tcp, 0.0.0.0:9022->9022/tcp, :::9022->9022/tcp
76ae98e540dd rc_cluster_services-channelstream-1 channelstream/channelstream:0.7.1 Up 54 minutes (healthy) 8000/tcp
c9aa9e64fdde rc_cluster_services-database-1 postgres:14.8 Up 54 minutes (healthy) 5432/tcp
9dde6a017be5 rc_cluster_services-elasticsearch-1 elasticsearch:6.8.23 Up 54 minutes (healthy) 9200/tcp, 9300/tcp
f6ebc9565421 rc_cluster_services-nginx-1 nginx:1.23.4 Up 54 minutes (healthy) 80/tcp
aa4c0d98724e rc_cluster_services-nginx-errors-1 nginx:1.23.4 Up 54 minutes 80/tcp
984102266e04 rc_cluster_services-redis-1 redis:7.0.11 Up 54 minutes (healthy) 6379/tcp
And it appears the the Rhodecode service is still constantly restarting, which is likely why it is not available.
You might ask whoever is doing the Docker work to setup a fresh VM with nothing else installed, and go thru the setup process to see if they also see what I am seeing.
Obviously something isn’t right, and perhaps something in their environment is enabling things to work properly, which prevents them from seeing what end users would see. I am not sure what else would be the issue.
Perhaps they might also want to document the path where they are putting the “rhocdecode-docker” folder, which user the service is being run as, any packages installed after a fresh OS installation, and so forth. Also posting recommended VM specs might be good too, as many people will likely run this in a VM.
There is a missing piece somewhere, and ensuring everyone is starting off from the same point could help eliminate a lot of variables. Not everyone has the time to troubleshoot installation issues like I have, they will walk away if things don’t go smoothly the first time, especially with a commercial product.
Just FYI: Everything reported these last two days was on Alpine using the latest Docker image (I believe 39).
Bottom line is, it appears there is more work needed before this is ready to roll out to new or existing users, because the point of using something like Docker is that it works the same everywhere for everyone, and that is not currently the case.
And, I understand having this exchange posted in the forums may not be of much benefit to other users, so feel free to remove if you think it is appropriate.
Just FYI: The reason I am making the effort to get this working, is that the docs state this will be the way you will be deploying Rhodecode in the future.
And the reason Rhodecode is of interest, is because you support SVN, which is still needed for a few projects. And, it would be nice to have everything under one roof, as opposed to one solution for git, and another for svn.
Plus, we need LDAP auth, and do not want to open all repositories to the public. In fact, most will remain private.
There are other solutions out there that tick all of those boxes, but for the reasons above and others which I would need more time to articulate, it seems your product might be the best for our particular use case.
So, the point is, I don’t mind going thru the process if the end result is a working Rhodecode cluster that can provide what will be needed going forward.
Thank you for added feedback.
We’ll go ahead and check on Alpine, to see what could go wrong, we don’t have Alpine in our CI workflow to test it, but we’ll consider adding it.
What is really odd that RhodeCode service goes from working to restarting. If service is restarting, Traefik will mark it as unavailable and display the landing page you’ve seen.
I am trying RhodeCode CE for exactly the same reason, and I also have the exact same issue, but the image that is constantly restarting is rc_cluster_apps-svn-1 . Has this issue been resolved? I followed the new install instructions and am on WSL2 Ubuntu 18.04.
Thanks,
Brian
Hi,
can you run:
./rcstack stack rhodecode up svn
That should spawn SVN instance with logs printed to console, it should show why it’s getting restarted.
Is there a flag to get more debug information? This is what I got:
./rcstack stack rhodecode up svn
[+] Building 0.0s (0/0)
[+] Running 1/0
✔ Container rc_cluster_apps-svn-1 Created 0.0s
Attaching to rc_cluster_apps-svn-1
rc_cluster_apps-svn-1 | ENTRYPOINT: Running rhodecode_svn with cmd 'httpd'
rc_cluster_apps-svn-1 | [FATAL tini (8)] exec httpd failed: No such file or directory
rc_cluster_apps-svn-1 exited with code 0
can you try to edit and put this change in ?
change httpd to apachectl in the docker-compose-apps.yaml file
I made that change and the svn service is starting now
It is now showing rc_cluster_services-nginx-statics-1 as “unhealthy” - it did before but I was hoping that was because the svn container was restarting. The console shows it looking for /var/log/rhodecode_static_data/static/robots.txt and failing to find the missing file. I saw robots.txt in /var/log/rhodecode_data/static/robots.txt . I created the missing directory and copied the robots.txt, and the rest of the /var/log/rhodecode_data/static contents over from within the rc_cluster_services-nginx-statics-1 container. This stopped the log entry from complaining about a missing robots.txt file, but the container is showing as unhealthy still and the browser shows an error page at http://localhost. I see a few config files mention this path, but one of them shows a copy of /var/log/rhodecode_data/static to /var/log/rhodecode_static_data, so I don’t want to touch too much. Is there another patch you can do?
Please disregard the statics healthcheck for now, this is something we need to disable for 4.X.X for 5.0.0 this works fine. IT’s a new feature, and we’ll work on making that only work for the 5.X series of RhodeCode.
otherwise does it work ok ?
I can’t get a web page showing at localhost. It is saying:
RhodeCode backend server is unreachable
check service health status:
./rcstack status
note: it might take up to 60s to start and verify services are in healthy state
make sure rhodecode stack is running with:
./rcstack stack rhodecode up --detach
Possible Causes
The RhodeCode server is starting right now.
The RhodeCode server is being restarted.
RhodeCode Stack is not running.
RhodeCode Stack services are unhealthy.
Reload page
ohh, i believe using localhost might be a problem, we need to check it.
Could you edit domain in .runtime.env to somelithing like rhodecode-test and make sure where you run you VM it actually points to this domain ?