Services stack does not start: "plugin loki found but disabled"

1

Unfortunately my new dockerized deployment of 5.0.3 stopped working. After a service downtime of the host, Rhodecode does not start.

The command to start the service stack simply returns:

Error response from daemon: error looking up logging plugin loki: plugin loki found but disabled

How can I fix this?

2

@rhodecode-support Can you help, please? I really need my SVN back …

3

After a lot of googling I found the solution in a different project.

If this error occurs, you need to give the command

docker plugin enable loki

in the server shell. After that the services stack can be started normally.

@dolobanko This should be definitely in the docs! And/or ./rcstack should check/handle this.

1 Like
4

Hi! Glad that you were able to resolve the issue, and thanks for a feedback! We will definitely add this to the docs and consider adding check to rcstack!

5

@rhodecode-support
Coming back to my old issue, I periodically see this issue when I reboot my Rhodecode-VM. Rhodecode does not start up because of a disabled Loki.

Checking the rcstack code, in bootstrap_docker_commons Loki will be installed it it’s enabled and missing:

loki_driver=$(docker plugin ls … --filter enabled=true | grep loki || echo “”)
if [[ $loki_driver == “” ]]; then
docker plugin install … --alias loki …
fi

Wouldn’t it make more sense to also cover the state like this:

  • If Loki exists and is disabled: –> Enable
  • If Loki does not exist: –> Install
if docker plugin inspect loki:latest >/dev/null 2>&1; then
  docker plugin enable loki:latest >/dev/null 2>&1 || true
  echo "bootstrap_docker_commons: loki driver present (enabled or now enabled)"
else
  docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions
fi
6

Hi! Let me check on this

7

What version of rhodecode and rcstack are you using?

8

If we change that logic, you would still need to run init command in the VM, is that expected ? The loki logic is under ./rcstack init command which install all required dependencies

9

Hi @andverb_rhodecode

Rhodecode: 5.9.1

rcstack: 5.32.0

10

The problem mostly occurs when the VM is rebooted. In this case I always need to give the command docker plugin enable loki. It’ll be great if it would be checked and corrected automatically if the plugin is there but disabled before the first one runs into a 404 error.

11

This is a weird behaviour, can you explain why plugin gets disabled on VM restart ?

12

No, unfortunately I can’t. I simply see this behavior after a VM reboot.

13

past the init phase we don’t do these checks, we don’t even check for docker command to be present. If we’d do this would be a significant overhead for each command of rcstack to check if loki plugin is enabled.

There’s a plugin mechanism for rcstack, that allows users to run their own commands before, we might enable this so you could pass this