How Installing socat Nearly Bricked My Orange Pi 5 Pro (and How I Upgraded It to Ubuntu 24.04 Anyway)

I run an Orange Pi 5 Pro as a home server: Docker, Ollama, PHP, a few background services. It sits on a shelf with no monitor attached, and I manage it entirely over SSH. One afternoon I tried to install a small networking utility and ended up spending several hours rescuing the system, diagnosing a subtle packaging trap in the vendor image, and finally upgrading from Ubuntu 22.04 to 24.04.

This post walks through the whole thing, partly as a record for myself and partly because the root cause is the kind of thing that will catch out anyone running a vendor Ubuntu image on an SBC.

The symptom

fiko@orangepi5pro:~$ sudo apt install socat
sudo: account validation failure, is your account locked?
sudo: a password is required

"Account validation failure" comes from the PAM account stage, not from the password check itself. The usual suspects are:

• A lockout from too many failed attempts (faillock)

• An expired password or account

• A wrong system clock, which is common on boards without an RTC battery and can make a password look expired

Ruling out the usual suspects

Fortunately I still had an SSH session open from earlier in the day. That session turned out to be my lifeline, so the first rule was simple: do not close it.

chage -l fiko   # password never expires, account never expires
date            # correct
su -            # su: Authentication failure

Password ageing and the clock were fine. su failing as well suggested something deeper than my own account.

The clue hiding in an error message

I ran a few checks that don't need root, including a look at the journal:

groups
journalctl -n 30 --no-pager

The output was the turning point:

fiko ollama docker
journalctl: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.36' not found (required by /lib/aarch64-linux-gnu/libcrypt.so.1)
journalctl: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found (required by /lib/aarch64-linux-gnu/libcrypt.so.1)

libcrypt had been upgraded to a version built against a newer glibc than the system had. Everything that verifies passwords goes through libcrypt, so sudo, su and SSH password logins were all broken. The su failure had nothing to do with the root password.

The Docker group as a recovery tool

The other important detail in that output was docker. Membership of the docker group is effectively root access, because you can start a container with the host's filesystem mounted. That is a security consideration in normal use, but here it was the way back in:

docker run --rm -it -v /:/host alpine chroot /host /bin/bash

This failed immediately:

/bin/bash: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found (required by /bin/bash)

So it wasn't only libcrypt. Bash itself had been replaced with a version that needed the newer glibc. My existing session only worked because its bash had been loaded into memory before the change.

The fix was to stop trying to run the host's binaries at all. Instead, I worked from inside a container, which brings its own working glibc, and treated the host as a mounted folder:

docker run --rm -it -v /:/host ubuntu bash
cat /host/etc/os-release
dpkg-query --admindir=/host/var/lib/dpkg -W libc6 libcrypt1 bash
tail -60 /host/var/log/apt/history.log
grep -E ' (install|upgrade) ' /host/var/log/dpkg.log | tail -40

The --admindir option lets the container's dpkg-query read the host's package database directly.

What had actually happened

os-release claimed the system was Ubuntu 24.04, but almost every installed package was a 22.04 (jammy) version, including libc6 2.35. The apt sources had ended up pointing at noble (24.04), and the dpkg log showed exactly five packages that had been pulled in from noble before things fell over:

Package	Was (jammy)	Now (noble)
libcrypt1, libcrypt-dev	1:4.4.27-1	1:4.4.36-4build1
base-files	12ubuntu4.7	13ubuntu10.5
debianutils	5.5-1ubuntu2	5.17build1
bash	5.1-6ubuntu1.1	5.2.21-2ubuntu4

base-files is the package that owns /etc/os-release, which explains why the system "thought" it was on 24.04.

Rolling back to jammy

The plan was to put those five packages back to their jammy versions. I downloaded them in a jammy container, then installed them using the host's own dpkg, which was still a jammy binary and therefore still worked:

docker run --rm -it -v /:/host ubuntu:22.04 bash

apt-get update
mkdir -p /host/root/jammy-fix && cd /host/root/jammy-fix
apt-get download bash base-files debianutils libcrypt1 libcrypt-dev

chroot /host /usr/bin/dpkg -iR /root/jammy-fix

Four of the five went in cleanly. Bash failed, for a slightly comic reason: its removal script is a bash script, so dpkg tried to run the broken bash in order to uninstall the broken bash.

The way round it was to put a working bash in place manually, then let dpkg finish properly:

cd /tmp
dpkg-deb -x /host/root/jammy-fix/bash_5.1-6ubuntu1.1_arm64.deb bashx
cp bashx/bin/bash /host/bin/bash.new
chmod 755 /host/bin/bash.new
mv -f /host/bin/bash.new /host/bin/bash

chroot /host /usr/bin/dpkg -i /root/jammy-fix/bash_5.1-6ubuntu1.1_arm64.deb

Using mv rather than copying directly over the binary matters: it swaps the file atomically, so the running bash in my lifeline session kept its old copy in memory and wasn't disturbed.

Finally I pointed apt back at jammy and commented out the noble PPA lines:

sed -i -e '/huaweicloud/s/ noble/ jammy/' \
       -e '/launchpadcontent.*noble/s/^deb /# deb /' \
       /host/etc/apt/sources.list /host/etc/apt/sources.list.d/*.list

After that, a fresh SSH login worked, sudo worked, and sudo apt install socat finally installed what I had wanted in the first place.

Doing the upgrade properly

With the system healthy again, I decided to do the 24.04 upgrade the supported way, using do-release-upgrade. Since the board has a vendor kernel and bootloader that Ubuntu knows nothing about, I took a few precautions first.

Hold the vendor packages, so the upgrader can't replace or remove them:

dpkg -l | grep -E 'orangepi|rockchip|linux-(image|dtb|u-boot)'

sudo apt-mark hold linux-dtb-current-rockchip-rk3588 \
  linux-image-current-rockchip-rk3588 linux-u-boot-orangepi5pro-current \
  orangepi-bsp-cli-orangepi5pro orangepi-config orangepi-firmware orangepi-zsh

Run it inside tmux. A release upgrade takes a long time and stops to ask questions. If the SSH connection drops, the shell running the upgrade is killed, potentially halfway through replacing core libraries. With tmux, the upgrade carries on and you reattach with tmux attach.

tmux
sudo do-release-upgrade

The upgrader complained that it didn't recognise the mirror (repo.huaweicloud.com, which the Orange Pi image uses by default). That's harmless, as the mirror carries noble packages; it simply isn't on the upgrader's list of official mirrors.

The same failure, and the real root cause

Partway through, the upgrade failed in exactly the same way as before: noble's bash was installed, its post-install script couldn't run, and the upgrade aborted.

That was the moment the real cause became clear. The installed glibc was not a stock Ubuntu package:

libc6:arm64  1:2.35-0ubuntu3.4+2widevine

Look at the 1: at the start. That is a version epoch, and in Debian version comparison, any epoch outranks no epoch. So dpkg and apt consider 1:2.35 to be newer than noble's 2.39-0ubuntu8.9.

The consequences:

• apt never upgrades libc6, because it believes the installed version is already newer.

• Packages like bash, which depend on libc6 (>= 2.38), have their dependency satisfied on paper, so apt happily installs them.

• In reality the system is still running glibc 2.35, and anything built for 2.38 breaks.

This was almost certainly what caused the original breakage earlier in the day too. The Widevine-patched glibc exists for DRM video playback in browsers, which is irrelevant on a headless server.

Installing noble's glibc by hand

The fix was to replace every epoch-versioned glibc package with noble's version, then let the rest of the upgrade continue. First, find them all:

dpkg-query --admindir=/host/var/lib/dpkg -W | grep widevine

This listed glibc-source, libc-bin, libc-dev-bin, libc6, libc6-dev and locales. I downloaded the noble versions in a noble container:

docker run --rm -it -v /:/host ubuntu:24.04 bash

apt-get update
mkdir -p /host/root/noble-libc && cd /host/root/noble-libc
apt-get download libc6 libc-bin libc-dev-bin libc6-dev locales glibc-source

Installing them all at once hit two blockers:

1. libc6 breaks chrony (<< 4.2-3~). Noble's glibc declares the jammy chrony incompatible. Since chrony would be upgraded moments later anyway, --auto-deconfigure lets dpkg temporarily mark it unconfigured instead of refusing.

2. A file conflict on sotruss-lib.so. In noble that file moved from libc6 to libc6-dev. Normally the handover is automatic, but the epoch made dpkg think the old libc6 was newer, so it refused. Installing libc6 first resolved it.

So, one at a time:

chroot /host /usr/bin/dpkg -i --auto-deconfigure /root/noble-libc/libc6_2.39-0ubuntu8.9_arm64.deb
chroot /host /bin/bash -c 'echo bash ok'

chroot /host /usr/bin/dpkg -i /root/noble-libc/libc6-dev_2.39-0ubuntu8.9_arm64.deb
chroot /host /usr/bin/dpkg --configure -a

Seeing bash ok was a genuinely satisfying moment.

Finishing the upgrade

With glibc sorted, I didn't rerun do-release-upgrade, because the sources were already on noble. Plain apt finishes the job:

tmux new -s upgrade
sudo apt -f install
sudo apt full-upgrade

This brought up a long series of prompts about modified configuration files. On a vendor image, the right answer is almost always N (keep your current version), because the vendor has customised these files for the hardware:

File	Why keep it
/etc/systemd/journald.conf	Orange Pi keeps logs in RAM to reduce storage wear
/etc/sysctl.conf	Kernel settings from you, Docker or the vendor
/etc/logrotate.conf and logrotate.d/*	More aggressive rotation to match the RAM logging
/etc/initramfs-tools/initramfs.conf	Critical. Sets a compression method the vendor kernel can boot. Noble's default is zstd, which the old 6.1 vendor kernel may not handle
/etc/ssh/sshd_config	Critical. Controls how you log in
Apache configs	Your own web server setup

A couple of other prompts came up along the way. Postfix was pulled in as a new dependency, and I chose Local only, which handles system mail without opening port 25 to the network. iperf3 asked whether to run as a daemon, and I said no.

The upgrade ended with W: Operation was interrupted before it could finish, probably because NetworkManager restarted mid-run. Running sudo dpkg --configure -a followed by another sudo apt full-upgrade tidied that up.

Pre-reboot checklist

Rebooting a headless board into a broken boot configuration would mean pulling out the storage and fixing it on another machine, so I checked everything first:

grep VERSION= /etc/os-release
dpkg -l | grep -E 'orangepi|rockchip|linux-(image|dtb|u-boot)'
grep ^COMPRESS /etc/initramfs-tools/initramfs.conf
ls -lt /boot | head
sudo sshd -t && echo "sshd config ok"
systemctl status ssh.socket --no-pager
sudo netplan generate && echo "netplan ok"

Check	Expected
os-release	24.04
Held packages	All seven still installed, marked hi
COMPRESS	Not zstd
/boot	Freshly regenerated uInitrd and initrd.img
sshd -t	Config valid
ssh.socket	Active and listening (noble starts SSH via socket activation)
netplan generate	Valid (noble migrates NetworkManager connections to netplan)

I also opened a fresh SSH login from my laptop before closing anything. Everything passed, I rebooted, and the board came back up on Ubuntu 24.04.

Lessons learned

Never switch your apt sources to a new release by hand. Pointing sources.list at a newer release and running apt install gives you a mix of packages from two releases, installed in whatever order apt decides. do-release-upgrade exists precisely to manage that ordering.

Check vendor images for epoch-versioned core packages before upgrading. This command would have saved me hours:

dpkg-query -W libc6 libc-bin locales

If any version starts with 1: and Ubuntu's doesn't, apt will never upgrade it, and a release upgrade will break in exactly the way described here. Replace those packages with stock Ubuntu ones first.

Keep an existing session open while fixing access problems. My original SSH session, still running a bash loaded before the breakage, was what made recovery possible without physical access.

Know your back doors, and treat them seriously. The docker group gave me root when nothing else would. That's useful in an emergency, but it also means anyone in that group effectively has root on the machine.

Work from a container rather than chrooting into a broken system. When the host's own shell and libraries are broken, a container with -v /:/host gives you working tools. Commands like dpkg-query --admindir=/host/var/lib/dpkg and chroot /host /usr/bin/dpkg let you inspect and repair the host selectively.

Run long upgrades in tmux. A dropped connection should never be able to kill a release upgrade halfway through.

On vendor images, keep your config files. The vendor customised them for a reason, and some of them (initramfs compression in particular) decide whether the board boots at all.

All of that, just to install socat.