Rcstack: Can't stop or start the stack completely, docker-compose error


I upgraded my Ubuntu server this morning and now rcstack won’t stop or start the full stack any more.

Here is the upgrade list:

Start-Date: 2024-01-31  09:45:03
Commandline: apt upgrade --autoremove --with-new-pkgs
Requested-By: ubuntu (1000)
Upgrade: containerd.io:amd64 (1.6.26-1, 1.6.27-1), docker-compose-plugin:amd64 (2.21.0-1~ubuntu.22.04~jammy, 2.24.2-1~ubuntu.22.04~jammy), udev:amd64 (249.11-0ubuntu3.11, 249.11-0ubuntu3.12), libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2, 1.20.0-1~ubuntu22.04.3), docker-ce-cli:amd64 (5:24.0.7-1~ubuntu.22.04~jammy, 5:25.0.1-1~ubuntu.22.04~jammy), systemd-timesyncd:amd64 (249.11-0ubuntu3.11, 249.11-0ubuntu3.12), tzdata:amd64 (2023c-0ubuntu0.22.04.2, 2023d-0ubuntu0.22.04), libpam-systemd:amd64 (249.11-0ubuntu3.11, 249.11-0ubuntu3.12), libsystemd0:amd64 (249.11-0ubuntu3.11, 249.11-0ubuntu3.12), libnss-systemd:amd64 (249.11-0ubuntu3.11, 249.11-0ubuntu3.12), systemd:amd64 (249.11-0ubuntu3.11, 249.11-0ubuntu3.12), libudev1:amd64 (249.11-0ubuntu3.11, 249.11-0ubuntu3.12), docker-buildx-plugin:amd64 (0.11.2-1~ubuntu.22.04~jammy, 0.12.1-1~ubuntu.22.04~jammy), docker-ce:amd64 (5:24.0.7-1~ubuntu.22.04~jammy, 5:25.0.1-1~ubuntu.22.04~jammy), motd-news-config:amd64 (12ubuntu4.4, 12ubuntu4.5), libssl3:amd64 (3.0.2-0ubuntu1.12, 3.0.2-0ubuntu1.13), docker-ce-rootless-extras:amd64 (5:24.0.7-1~ubuntu.22.04~jammy, 5:25.0.1-1~ubuntu.22.04~jammy), base-files:amd64 (12ubuntu4.4, 12ubuntu4.5), systemd-hwe-hwdb:amd64 (249.11.4, 249.11.5), systemd-sysv:amd64 (249.11-0ubuntu3.11, 249.11-0ubuntu3.12), openssl:amd64 (3.0.2-0ubuntu1.12, 3.0.2-0ubuntu1.13)
End-Date: 2024-01-31  09:45:46

And the resulting Docker version:

docker --version
Docker version 25.0.1, build 29cf629

Now cd ~/docker-rhodecode && sudo ./rcstack stack all down gives:

validating /home/ubuntu/docker-rhodecode/.custom/docker-compose-metrics.override.yaml: services.grafana.labels array items[0,6] must be unique
# or
validating /home/ubuntu/docker-rhodecode/.custom/docker-compose-apps.override.yaml: services.rhodecode.labels array items[1,7] must be unique
# or
validating /home/ubuntu/docker-rhodecode/.custom/docker-compose-services.override.yaml: services.channelstream.labels array items[0,8] must be unique

Any advice on how to restore the situation to a working stack?
I gave unique labels to all of those services, but it’s a bit dirty and I’m not sure it won’t break something…

Well, actually it broke everything :sweat_smile:

Seems this is a docker problem: [BUG] Regression: array items[0,1] must be unique starting from 2.24.1 · Issue #11371 · docker/compose · GitHub

can you try running:

./rcstack cli docker-upgrade --compose

make sure to self-update before too

OK so this downgrades my Docker to version 2.24.5.

If I understand correctly, I should not manage Docker through the system’s package manager, but rather thanks to rcstack directly. Do you confirm?

You don’t need to manage it via rcstack. Rcstack does help users to install and upgrade docker.

The rcstack managed way also does use apt repositories and a bootstrap using get.docker.com.

It’s odd it downgraded docker to 24.X, but we recommended this command to actually help upgrade docker compose (this is a plugin) that had this regressions bug from GH ticket.

I believe things should work on top of 25.X docker, but compose plugin needs to be update to resolve the issues.

For reference our test machine run on top of this:

./rcstack cli docker-info
Docker version 25.0.1, build 29cf629
Docker Compose version v2.24.5

We think that you apt-get did update compose to a version that had that bug from GH issue

OK sorry, that’s a misunderstanding on my side. I got:

found docker-compose at /root/.docker/cli-plugins, removing it now...
new compose version v2.24.5 to be installed in /root/.docker/cli-plugins
updated compose to v2.24.5

but Docker is still at version 25.0.1.

Then I have

sudo ./rcstack cli docker-info
Docker version 25.0.1, build 29cf629
Docker Compose version v2.24.5

just like you.

Is still broken with that setup ?

No, it works like that, thank you.

Seems like the Docker bug is fixed in the latest release :+1: