# How Installing socat Nearly Bricked My Orange Pi 5 Pro (and How I Upgraded It to Ubuntu 24.04 Anyway) I run an Orange Pi 5 Pro as a home server: Docker, Ollama, PHP, a few background services. It sits on a shelf with no monitor attached, and I manage it entirely over SSH. One afternoon I tried to install a small networking utility and ended up spending several hours rescuing the system, diagnosing a subtle packaging trap in the vendor image, and finally upgrading from Ubuntu 22.04 to 24.04. This post walks through the whole thing, partly as a record for myself and partly because the root cause is the kind of thing that will catch out anyone running a vendor Ubuntu image on an SBC. ## The symptom ```plaintext fiko@orangepi5pro:~$ sudo apt install socat sudo: account validation failure, is your account locked? sudo: a password is required ``` "Account validation failure" comes from the PAM account stage, not from the password check itself. The usual suspects are: * A lockout from too many failed attempts (`faillock`) * An expired password or account * A wrong system clock, which is common on boards without an RTC battery and can make a password look expired ## Ruling out the usual suspects Fortunately I still had an SSH session open from earlier in the day. That session turned out to be my lifeline, so the first rule was simple: do not close it. ```bash chage -l fiko # password never expires, account never expires date # correct su - # su: Authentication failure ``` Password ageing and the clock were fine. `su` failing as well suggested something deeper than my own account. ## The clue hiding in an error message I ran a few checks that don't need root, including a look at the journal: ```bash groups journalctl -n 30 --no-pager ``` The output was the turning point: ```plaintext fiko ollama docker journalctl: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.36' not found (required by /lib/aarch64-linux-gnu/libcrypt.so.1) journalctl: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found (required by /lib/aarch64-linux-gnu/libcrypt.so.1) ``` `libcrypt` had been upgraded to a version built against a newer glibc than the system had. Everything that verifies passwords goes through libcrypt, so `sudo`, `su` and SSH password logins were all broken. The `su` failure had nothing to do with the root password. ## The Docker group as a recovery tool The other important detail in that output was `docker`. Membership of the `docker` group is effectively root access, because you can start a container with the host's filesystem mounted. That is a security consideration in normal use, but here it was the way back in: ```bash docker run --rm -it -v /:/host alpine chroot /host /bin/bash ``` This failed immediately: ```plaintext /bin/bash: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found (required by /bin/bash) ``` So it wasn't only libcrypt. **Bash itself** had been replaced with a version that needed the newer glibc. My existing session only worked because its bash had been loaded into memory before the change. The fix was to stop trying to run the host's binaries at all. Instead, I worked from inside a container, which brings its own working glibc, and treated the host as a mounted folder: ```bash docker run --rm -it -v /:/host ubuntu bash cat /host/etc/os-release dpkg-query --admindir=/host/var/lib/dpkg -W libc6 libcrypt1 bash tail -60 /host/var/log/apt/history.log grep -E ' (install|upgrade) ' /host/var/log/dpkg.log | tail -40 ``` The `--admindir` option lets the container's `dpkg-query` read the host's package database directly. ## What had actually happened `os-release` claimed the system was Ubuntu 24.04, but almost every installed package was a 22.04 (jammy) version, including `libc6 2.35`. The apt sources had ended up pointing at **noble (24.04)**, and the dpkg log showed exactly five packages that had been pulled in from noble before things fell over: | Package | Was (jammy) | Now (noble) | | --- | --- | --- | | libcrypt1, libcrypt-dev | 1:4.4.27-1 | 1:4.4.36-4build1 | | base-files | 12ubuntu4.7 | 13ubuntu10.5 | | debianutils | 5.5-1ubuntu2 | 5.17build1 | | bash | 5.1-6ubuntu1.1 | 5.2.21-2ubuntu4 | `base-files` is the package that owns `/etc/os-release`, which explains why the system "thought" it was on 24.04. ## Rolling back to jammy The plan was to put those five packages back to their jammy versions. I downloaded them in a jammy container, then installed them using the host's own `dpkg`, which was still a jammy binary and therefore still worked: ```bash docker run --rm -it -v /:/host ubuntu:22.04 bash apt-get update mkdir -p /host/root/jammy-fix && cd /host/root/jammy-fix apt-get download bash base-files debianutils libcrypt1 libcrypt-dev chroot /host /usr/bin/dpkg -iR /root/jammy-fix ``` Four of the five went in cleanly. Bash failed, for a slightly comic reason: its removal script is a bash script, so dpkg tried to run the broken bash in order to uninstall the broken bash. The way round it was to put a working bash in place manually, then let dpkg finish properly: ```bash cd /tmp dpkg-deb -x /host/root/jammy-fix/bash_5.1-6ubuntu1.1_arm64.deb bashx cp bashx/bin/bash /host/bin/bash.new chmod 755 /host/bin/bash.new mv -f /host/bin/bash.new /host/bin/bash chroot /host /usr/bin/dpkg -i /root/jammy-fix/bash_5.1-6ubuntu1.1_arm64.deb ``` Using `mv` rather than copying directly over the binary matters: it swaps the file atomically, so the running bash in my lifeline session kept its old copy in memory and wasn't disturbed. Finally I pointed apt back at jammy and commented out the noble PPA lines: ```bash sed -i -e '/huaweicloud/s/ noble/ jammy/' \ -e '/launchpadcontent.*noble/s/^deb /# deb /' \ /host/etc/apt/sources.list /host/etc/apt/sources.list.d/*.list ``` After that, a fresh SSH login worked, `sudo` worked, and `sudo apt install socat` finally installed what I had wanted in the first place. ## Doing the upgrade properly With the system healthy again, I decided to do the 24.04 upgrade the supported way, using `do-release-upgrade`. Since the board has a vendor kernel and bootloader that Ubuntu knows nothing about, I took a few precautions first. **Hold the vendor packages**, so the upgrader can't replace or remove them: ```bash dpkg -l | grep -E 'orangepi|rockchip|linux-(image|dtb|u-boot)' sudo apt-mark hold linux-dtb-current-rockchip-rk3588 \ linux-image-current-rockchip-rk3588 linux-u-boot-orangepi5pro-current \ orangepi-bsp-cli-orangepi5pro orangepi-config orangepi-firmware orangepi-zsh ``` **Run it inside tmux.** A release upgrade takes a long time and stops to ask questions. If the SSH connection drops, the shell running the upgrade is killed, potentially halfway through replacing core libraries. With tmux, the upgrade carries on and you reattach with `tmux attach`. ```bash tmux sudo do-release-upgrade ``` The upgrader complained that it didn't recognise the mirror (`repo.huaweicloud.com`, which the Orange Pi image uses by default). That's harmless, as the mirror carries noble packages; it simply isn't on the upgrader's list of official mirrors. ## The same failure, and the real root cause Partway through, the upgrade failed in exactly the same way as before: noble's bash was installed, its post-install script couldn't run, and the upgrade aborted. That was the moment the real cause became clear. The installed glibc was not a stock Ubuntu package: ```plaintext libc6:arm64 1:2.35-0ubuntu3.4+2widevine ``` Look at the `1:` at the start. That is a **version epoch**, and in Debian version comparison, any epoch outranks no epoch. So dpkg and apt consider `1:2.35` to be **newer** than noble's `2.39-0ubuntu8.9`. The consequences: * apt never upgrades libc6, because it believes the installed version is already newer. * Packages like bash, which depend on `libc6 (>= 2.38)`, have their dependency satisfied on paper, so apt happily installs them. * In reality the system is still running glibc 2.35, and anything built for 2.38 breaks. This was almost certainly what caused the original breakage earlier in the day too. The Widevine-patched glibc exists for DRM video playback in browsers, which is irrelevant on a headless server. ## Installing noble's glibc by hand The fix was to replace every epoch-versioned glibc package with noble's version, then let the rest of the upgrade continue. First, find them all: ```bash dpkg-query --admindir=/host/var/lib/dpkg -W | grep widevine ``` This listed `glibc-source`, `libc-bin`, `libc-dev-bin`, `libc6`, `libc6-dev` and `locales`. I downloaded the noble versions in a noble container: ```bash docker run --rm -it -v /:/host ubuntu:24.04 bash apt-get update mkdir -p /host/root/noble-libc && cd /host/root/noble-libc apt-get download libc6 libc-bin libc-dev-bin libc6-dev locales glibc-source ``` Installing them all at once hit two blockers: 1. `libc6 breaks chrony (<< 4.2-3~)`. Noble's glibc declares the jammy chrony incompatible. Since chrony would be upgraded moments later anyway, `--auto-deconfigure` lets dpkg temporarily mark it unconfigured instead of refusing. 2. **A file conflict on** `sotruss-lib.so`. In noble that file moved from `libc6` to `libc6-dev`. Normally the handover is automatic, but the epoch made dpkg think the old libc6 was newer, so it refused. Installing libc6 first resolved it. So, one at a time: ```bash chroot /host /usr/bin/dpkg -i --auto-deconfigure /root/noble-libc/libc6_2.39-0ubuntu8.9_arm64.deb chroot /host /bin/bash -c 'echo bash ok' chroot /host /usr/bin/dpkg -i /root/noble-libc/libc6-dev_2.39-0ubuntu8.9_arm64.deb chroot /host /usr/bin/dpkg --configure -a ``` Seeing `bash ok` was a genuinely satisfying moment. ## Finishing the upgrade With glibc sorted, I didn't rerun `do-release-upgrade`, because the sources were already on noble. Plain apt finishes the job: ```bash tmux new -s upgrade sudo apt -f install sudo apt full-upgrade ``` This brought up a long series of prompts about modified configuration files. On a vendor image, the right answer is almost always **N** (keep your current version), because the vendor has customised these files for the hardware: | File | Why keep it | | --- | --- | | `/etc/systemd/journald.conf` | Orange Pi keeps logs in RAM to reduce storage wear | | `/etc/sysctl.conf` | Kernel settings from you, Docker or the vendor | | `/etc/logrotate.conf` and `logrotate.d/*` | More aggressive rotation to match the RAM logging | | `/etc/initramfs-tools/initramfs.conf` | **Critical.** Sets a compression method the vendor kernel can boot. Noble's default is `zstd`, which the old 6.1 vendor kernel may not handle | | `/etc/ssh/sshd_config` | **Critical.** Controls how you log in | | Apache configs | Your own web server setup | A couple of other prompts came up along the way. Postfix was pulled in as a new dependency, and I chose **Local only**, which handles system mail without opening port 25 to the network. iperf3 asked whether to run as a daemon, and I said no. The upgrade ended with `W: Operation was interrupted before it could finish`, probably because NetworkManager restarted mid-run. Running `sudo dpkg --configure -a` followed by another `sudo apt full-upgrade` tidied that up. ## Pre-reboot checklist Rebooting a headless board into a broken boot configuration would mean pulling out the storage and fixing it on another machine, so I checked everything first: ```bash grep VERSION= /etc/os-release dpkg -l | grep -E 'orangepi|rockchip|linux-(image|dtb|u-boot)' grep ^COMPRESS /etc/initramfs-tools/initramfs.conf ls -lt /boot | head sudo sshd -t && echo "sshd config ok" systemctl status ssh.socket --no-pager sudo netplan generate && echo "netplan ok" ``` | Check | Expected | | --- | --- | | `os-release` | 24.04 | | Held packages | All seven still installed, marked `hi` | | `COMPRESS` | Not `zstd` | | `/boot` | Freshly regenerated `uInitrd` and `initrd.img` | | `sshd -t` | Config valid | | `ssh.socket` | Active and listening (noble starts SSH via socket activation) | | `netplan generate` | Valid (noble migrates NetworkManager connections to netplan) | I also opened a fresh SSH login from my laptop before closing anything. Everything passed, I rebooted, and the board came back up on Ubuntu 24.04. ## Lessons learned **Never switch your apt sources to a new release by hand.** Pointing `sources.list` at a newer release and running `apt install` gives you a mix of packages from two releases, installed in whatever order apt decides. `do-release-upgrade` exists precisely to manage that ordering. **Check vendor images for epoch-versioned core packages before upgrading.** This command would have saved me hours: ```bash dpkg-query -W libc6 libc-bin locales ``` If any version starts with `1:` and Ubuntu's doesn't, apt will never upgrade it, and a release upgrade will break in exactly the way described here. Replace those packages with stock Ubuntu ones first. **Keep an existing session open while fixing access problems.** My original SSH session, still running a bash loaded before the breakage, was what made recovery possible without physical access. **Know your back doors, and treat them seriously.** The `docker` group gave me root when nothing else would. That's useful in an emergency, but it also means anyone in that group effectively has root on the machine. **Work from a container rather than chrooting into a broken system.** When the host's own shell and libraries are broken, a container with `-v /:/host` gives you working tools. Commands like `dpkg-query --admindir=/host/var/lib/dpkg` and `chroot /host /usr/bin/dpkg` let you inspect and repair the host selectively. **Run long upgrades in tmux.** A dropped connection should never be able to kill a release upgrade halfway through. **On vendor images, keep your config files.** The vendor customised them for a reason, and some of them (initramfs compression in particular) decide whether the board boots at all. All of that, just to install `socat`.