The Space That Wasn’t There: Upgrading Fedora in the Dark
A Fedora 43 → 44 release upgrade whose only real wound was a single invisible character.
Fedora 43 → 44. A full release upgrade — and true to the personality of this series, the part that nearly sank me was one invisible character.
The last two chapters of this series were about making my Sway desktop work — turning a black screen into a home, then chasing down volume keys that worked all along. This one is about something scarier: deliberately tearing the whole foundation out and sliding a new one underneath, live, on the only laptop I own.
“continue os update.”
Archaeology · dnf historyIt started, embarrassingly, with three words. I opened a fresh session and typed “continue os update” — to a context that had no memory of what I’d been doing. Continue what, exactly?
So before touching anything, archaeology. dnf history is a diary you didn’t know you were keeping:
dnf history list | head
The diary said: a system-upgrade download a few days back, a dnf upgrade --refresh this morning, a kmod-v4l2loopback rebuild right after. Cross-checked against rpm -E %fedora and the running kernel, the verdict was clear: I was already on Fedora 43, freshly booted. The big upgrade had happened. “Continue” didn’t mean do the upgrade — it meant clean up after it, then line up the next one. Lesson zero: when past-you leaves a vague note, make present-you read the logs before believing the note.
The leftovers that weren’t garbage
Audit · who still needs you?Every release upgrade leaves sediment: packages built for an older Fedora that never got rebuilt, sitting there with a stale .fc42 or .fc41 tag. The lazy move is dnf autoremove and walk away. The honest move is to ask, of each one, who still needs you?
rpm -qa | grep -E '\.fc4[12]\.'
A dozen suspects. But a reverse-dependency check turned half of them from “orphan” into “load-bearing”:
- webkit2gtk4.0 (fc42) ← still pulled in by my remote-desktop client
- javascriptcoregtk4.0 (fc42) ← needed by that webkit
- vamp-plugin-sdk (fc42) ← needed by Audacity, which is itself current
“Leftover” is not a synonym for “garbage.” None of those were safe to yank; the apps that depend on them were perfectly current and just hadn’t been rebuilt against newer libraries yet. (Hold onto the webkit chain — it has a quiet payoff at the end.) The only genuinely dead weight was an old kernel set, and even that would auto-prune at the next kernel install. I removed it by hand purely for the tidiness dopamine.
The config markers, and a directive worth adopting
Review · .rpmnew / .rpmsaveUpgrades scatter .rpmnew and .rpmsave files — the package manager’s way of saying “I had an opinion about this config but didn’t want to overwrite yours.” Two were worth reading instead of reflexively deleting:
- cups-browsed.conf: a post-upgrade scriptlet had quietly set BrowseRemoteProtocols none — a hardening default from the cups-browsed CVE fallout. My old saved copy had the chattier dnssd cups. I kept the locked-down none and deleted the marker. Do not helpfully “restore” your old value here.
- chrony.conf: my custom NTP server line was intact, but the new default retired a deprecated leap-second directive in favour of a modern one:
# old, deprecated leapsectz right/UTC # new leapseclist /usr/share/zoneinfo/leap-seconds.list
Adopting it seemed harmless. Reader, it was not harmless.
The space that wasn’t there
Bug · a missing spaceTo swap the directive in, I reached for sed. And sed gave me this:
leapseclist/usr/share/zoneinfo/leap-seconds.list
Look closely. There is no space after leapseclist. A one-character wound — a malformed directive that chrony would reject outright on the next parse. The keyword and its argument had fused into a single meaningless token.
I only caught it because I’d built a reflex earlier in this journey: verify before you declare victory. Not with my eyes — eyes slide right over a missing space — but with a tool that makes whitespace visible:
cat -A /etc/chrony.conf | grep leap
# leapseclist/usr/share/zoneinfo/leap-seconds.list$cat -A doesn’t editorialise. The missing gap was right there, no $-anchored space in sight. A space is invisible until it bites; cat -A makes it bite first.
…and the config I “fixed” had never loaded
Diagnosis · file ≠ processThen the second gotcha, stacked on the first like it had been waiting. Even if my edit had been clean, it wouldn’t have mattered — because the chronyd running on my system was still the one from boot. Its log proudly said it was “Using right/UTC timezone to obtain leap second data” — the old directive. I had “adopted” a change that was (a) malformed and (b) not even loaded.
Editing a config file does not change a running daemon. Obvious when stated; invisible in the moment. The fix needed both halves:
sed -i 's|^leapseclist/usr|leapseclist /usr|' /etc/chrony.conf # the space systemctl restart chronyd # the reload
And only then did the log tell the truth:
chronyd[34630]: Using leap second list /usr/share/zoneinfo/leap-seconds.list
A restart is not a reload, and a file is not a process. Check the thing that’s running, not the thing on disk.
A restart is not a reload, and a file is not a process.
fritz.box is a place, not a server
Fix · a roaming clock sourceWhile I had chrony open, I asked the question I should have asked years ago: is this thing even syncing? It was not.
chronyc sources chronyc tracking # Leap status : Not synchronised
My config pointed at server fritz.box iburst — my home router. And fritz.box only resolves when I’m home. This is a laptop. Pinning its clock to the living-room router means that everywhere else on Earth, it has no time source at all. So I handed it servers that exist regardless of which café I’m in:
pool 2.fedora.pool.ntp.org iburst sourcedir /run/chrony-dhcp # still let the home router chime in as a bonus
A roaming machine should not depend on furniture.
A roaming machine should not depend on furniture.
Choosing the next floor to stand on
Decide · let the mirrors voteNow the actual upgrade. First question: 43 → what? I didn’t trust release dates from memory, so I let the mirror network vote. A Generally Available release has a full fleet of mirrors; a still-Branched one is sparse. Probing the metalinks, F44 had a healthy mirror count and F45 was still thin and pre-release. 43 → 44 it was — decided by infrastructure, not by my recollection of a calendar.
Then the part everyone clicks through too fast — the GPG key import:
Importing key 0x6D9F90A6: Userid : "Fedora (44) <fedora-44-primary@fedoraproject.org>" Fingerprint: 36F6 12DC F27F 7D1A 48A8 35E4 DBFC F71C 6D9F 90A6
This is the moment you’re being asked to trust a stranger’s signature on everything that’s about to overwrite your OS. So I checked it before typing y: the keyfile was owned by the official fedora-gpg-keys package, rpm -V said it was unmodified on disk, and the fingerprint matched the prompt character for character. Thirty seconds to not blindly accept a key. Worth it.
The dnf5 tell
Detail · dnf4 vs dnf5A small thing tipped me off mid-upgrade. The status line came back as “Offline-Transaktion testen” — and that exact German phrasing wasn’t how the dnf I remembered talked. It was the giveaway that dnf on this box is now dnf5, with the old dnf4 lurking as the dnf-3 binary. That one detail explained an earlier mystery (why all the dnf4 state directories I’d gone hunting through were empty — I’d been excavating the wrong era) and reassured me about the upgrade itself: in dnf5’s system-upgrade, package conflicts fail at resolve time, before a single file downloads. The fact that it staged at all was proof my load-bearing orphans hadn’t blocked anything.
Staged, lights off
Execute · upgrade in the darkThe download left an offline transaction parked and ready. The actual swap happens in a minimal environment after a reboot, with the real system halted — no live filesystem being rewritten under running programs. Pre-flight checklist: on AC power, work saved, deep breath.
dnf system-upgrade download --releasever=44 # already done, days ago dnf system-upgrade reboot # commit
Then the screen goes into that stark, progress-bar-on-black offline updater, six thousand-odd packages march past, and the machine reboots into a Fedora it has never been before. Upgrading in the dark, literally.
Did it land clean?
Verify · trust nothingIt did. And because this series has taught me not to trust the calm-looking parts, I didn’t take the login screen’s word for it. The health check, in order:
cat /etc/fedora-release # Fedora release 44 (Forty Four) systemctl --failed # 0 loaded units listed dnf check # (silence — no broken deps, no duplicates)
No failed services. No dependency wreckage. The new kernel (7.0.12-200.fc44) booted on the first try, with two fc43 kernels kept as fallbacks.
There was one heart-stopping moment in the upgrade log:
systemd ... Failed to start jobs: Invalid argument systemd ... Remote peer disconnected
Pure adrenaline — until I remembered where those ran. Package scriptlets firing in the offline upgrade environment can’t talk to a system manager that isn’t fully alive yet. Those errors are expected and harmless; the calm-looking login screen was telling the truth and the scary-looking log was the innocent one. Same lesson as the black workspace and the flawless volume keys, wearing a sysadmin costume this time.
And the two questions the notes file most wanted answered:
- Did chrony survive the jump? Completely. Post-upgrade: Leap status: Normal, four pool servers all at full reach, a sub-millisecond offset, and the log still reading Using leap second list /usr/share/zoneinfo/leap-seconds.list. The directive I’d fought a missing space for came through the release upgrade intact.
- Did the orphans finally break? Better than that — some of them resolved themselves. The whole fc42 webkit chain I’d agonised over keeping — webkit2gtk4.0, javascriptcoregtk4.0, vamp-plugin-sdk — was simply gone, quietly replaced by fc44 builds that no longer needed the old libraries. The careful “do not remove these” detective work from the morning was rendered moot by the upgrade itself. Two stragglers remain at fc42 (xl2tpd, hfsutils), pure leaf apps with no fc44 rebuild yet — exactly the harmless kind of leftover I’d argued for keeping.
What I’d tell past-me (upgrade edition)
- Read the logs before you trust the note. “Continue” meant nothing until dnf history told me what I’d actually already done. Past-you is an unreliable narrator; the transaction log is not.
- Verify a config edit with cat -A. Spaces are invisible until they bite. A directive that looks right and a directive that is right differ by characters your eyes refuse to render.
- A restart is not a reload, and a file is not a process. I’d “fixed” a config that the running daemon had never read. Always check the live process, not just the file on disk.
- Don’t pin a roaming laptop’s clock to your living room. fritz.box is a place. Give a mobile machine sources that exist everywhere it goes.
- Read GPG fingerprints before you trust them. Thirty seconds of checking beats handing root to a signature you never looked at.
- Know which dnf you’re actually running. dnf4 and dnf5 keep state in different places and phrase things differently. I wasted real time spelunking the wrong era’s directories.
The recurring moral of this whole series keeps reasserting itself: on this system, nothing happens for you, so every fix is real and comprehensible — but the trouble is almost never where you first point the finger. This time it was a character I couldn’t see, in a file a daemon hadn’t read, syncing to a router that wasn’t there.
The cleanup that had nothing to do
Postscript · nothing to doThere’s a tidy ritual for mopping up release leftovers — dnf distro-sync, which drags every stray package into line with the new release. I’d been saving it as the satisfying final beat: the grand cleanup that would sweep away all that fc43 sediment in one cathartic transaction.
I ran it. It thought for a moment and said: nothing to do.
Every leftover was already the newest thing that exists. The orphans I’d planned to purge had either dissolved on their own during the upgrade or were genuinely the latest available build. The dramatic cleanup I’d lined up was an anticlimax by design — the system had quietly cleaned up after itself while I wasn’t looking, and left me holding a broom in a room that was already swept.
Fitting, somehow. The scariest operation in this series — gutting the OS and rebooting into the dark — went off without drama, and the part I’d saved for the triumphant finish turned out to be a no-op. The black-screen panic, the phantom volume keys, the self-shooting pkill — and now an upgrade whose only real wound was a missing space I inflicted on myself. I’m starting to suspect the machine was never the problem.