Rcstack: Can't stop or start the stack completely, docker-compose error

Support
1

Hello,

I upgraded my Ubuntu server this morning and now rcstack won’t stop or start the full stack any more.

Here is the upgrade list:

Start-Date: 2024-01-31  09:45:03
Commandline: apt upgrade --autoremove --with-new-pkgs
Requested-By: ubuntu (1000)
Upgrade: containerd.io:amd64 (1.6.26-1, 1.6.27-1), docker-compose-plugin:amd64 (2.21.0-1~ubuntu.22.04~jammy, 2.24.2-1~ubuntu.22.04~jammy), udev:amd64 (249.11-0ubuntu3.11, 249.11-0ubuntu3.12), libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2, 1.20.0-1~ubuntu22.04.3), docker-ce-cli:amd64 (5:24.0.7-1~ubuntu.22.04~jammy, 5:25.0.1-1~ubuntu.22.04~jammy), systemd-timesyncd:amd64 (249.11-0ubuntu3.11, 249.11-0ubuntu3.12), tzdata:amd64 (2023c-0ubuntu0.22.04.2, 2023d-0ubuntu0.22.04), libpam-systemd:amd64 (249.11-0ubuntu3.11, 249.11-0ubuntu3.12), libsystemd0:amd64 (249.11-0ubuntu3.11, 249.11-0ubuntu3.12), libnss-systemd:amd64 (249.11-0ubuntu3.11, 249.11-0ubuntu3.12), systemd:amd64 (249.11-0ubuntu3.11, 249.11-0ubuntu3.12), libudev1:amd64 (249.11-0ubuntu3.11, 249.11-0ubuntu3.12), docker-buildx-plugin:amd64 (0.11.2-1~ubuntu.22.04~jammy, 0.12.1-1~ubuntu.22.04~jammy), docker-ce:amd64 (5:24.0.7-1~ubuntu.22.04~jammy, 5:25.0.1-1~ubuntu.22.04~jammy), motd-news-config:amd64 (12ubuntu4.4, 12ubuntu4.5), libssl3:amd64 (3.0.2-0ubuntu1.12, 3.0.2-0ubuntu1.13), docker-ce-rootless-extras:amd64 (5:24.0.7-1~ubuntu.22.04~jammy, 5:25.0.1-1~ubuntu.22.04~jammy), base-files:amd64 (12ubuntu4.4, 12ubuntu4.5), systemd-hwe-hwdb:amd64 (249.11.4, 249.11.5), systemd-sysv:amd64 (249.11-0ubuntu3.11, 249.11-0ubuntu3.12), openssl:amd64 (3.0.2-0ubuntu1.12, 3.0.2-0ubuntu1.13)
End-Date: 2024-01-31  09:45:46

And the resulting Docker version:

docker --version
Docker version 25.0.1, build 29cf629

Now cd ~/docker-rhodecode && sudo ./rcstack stack all down gives:

validating /home/ubuntu/docker-rhodecode/.custom/docker-compose-metrics.override.yaml: services.grafana.labels array items[0,6] must be unique
# or
validating /home/ubuntu/docker-rhodecode/.custom/docker-compose-apps.override.yaml: services.rhodecode.labels array items[1,7] must be unique
# or
validating /home/ubuntu/docker-rhodecode/.custom/docker-compose-services.override.yaml: services.channelstream.labels array items[0,8] must be unique

Any advice on how to restore the situation to a working stack?
I gave unique labels to all of those services, but it’s a bit dirty and I’m not sure it won’t break something…

2

Well, actually it broke everything :sweat_smile:

3

Seems this is a docker problem: [BUG] Regression: array items[0,1] must be unique starting from 2.24.1 · Issue #11371 · docker/compose · GitHub

4

can you try running:

./rcstack cli docker-upgrade --compose

make sure to self-update before too

6

OK so this downgrades my Docker to version 2.24.5.

If I understand correctly, I should not manage Docker through the system’s package manager, but rather thanks to rcstack directly. Do you confirm?

7

You don’t need to manage it via rcstack. Rcstack does help users to install and upgrade docker.

The rcstack managed way also does use apt repositories and a bootstrap using get.docker.com.

It’s odd it downgraded docker to 24.X, but we recommended this command to actually help upgrade docker compose (this is a plugin) that had this regressions bug from GH ticket.

I believe things should work on top of 25.X docker, but compose plugin needs to be update to resolve the issues.

For reference our test machine run on top of this:

./rcstack cli docker-info
Docker version 25.0.1, build 29cf629
Docker Compose version v2.24.5

We think that you apt-get did update compose to a version that had that bug from GH issue

8

OK sorry, that’s a misunderstanding on my side. I got:

found docker-compose at /root/.docker/cli-plugins, removing it now...
new compose version v2.24.5 to be installed in /root/.docker/cli-plugins
updated compose to v2.24.5

but Docker is still at version 25.0.1.

Then I have

sudo ./rcstack cli docker-info
Docker version 25.0.1, build 29cf629
Docker Compose version v2.24.5

just like you.

9

Is still broken with that setup ?

10

No, it works like that, thank you.

11

Seems like the Docker bug is fixed in the latest release :+1: