All posts

The Space That Wasn’t There: Upgrading Fedora in the Dark

Sway · Dell XPS 13 · Fedora 43 → 44

The Space That Wasn’t There: Upgrading Fedora in the Dark

A Fedora 43 → 44 release upgrade whose only real wound was a single invisible character.

Fedora 43 → 44 · 6000+ packages · upgrading in the dark

Fedora 43 → 44. A full release upgrade — and true to the personality of this series, the part that nearly sank me was one invisible character.

The last two chapters of this series were about making my Sway desktop work — turning a black screen into a home, then chasing down volume keys that worked all along. This one is about something scarier: deliberately tearing the whole foundation out and sliding a new one underneath, live, on the only laptop I own.


00Log · Where am I?

“continue os update.”

Archaeology · dnf history

It started, embarrassingly, with three words. I opened a fresh session and typed “continue os update” — to a context that had no memory of what I’d been doing. Continue what, exactly?

So before touching anything, archaeology. dnf history is a diary you didn’t know you were keeping:

shell
dnf history list | head

The diary said: a system-upgrade download a few days back, a dnf upgrade --refresh this morning, a kmod-v4l2loopback rebuild right after. Cross-checked against rpm -E %fedora and the running kernel, the verdict was clear: I was already on Fedora 43, freshly booted. The big upgrade had happened. “Continue” didn’t mean do the upgrade — it meant clean up after it, then line up the next one. Lesson zero: when past-you leaves a vague note, make present-you read the logs before believing the note.

01Log · Orphans

The leftovers that weren’t garbage

Audit · who still needs you?

Every release upgrade leaves sediment: packages built for an older Fedora that never got rebuilt, sitting there with a stale .fc42 or .fc41 tag. The lazy move is dnf autoremove and walk away. The honest move is to ask, of each one, who still needs you?

shell
rpm -qa | grep -E '\.fc4[12]\.'

A dozen suspects. But a reverse-dependency check turned half of them from “orphan” into “load-bearing”:

  • webkit2gtk4.0 (fc42) ← still pulled in by my remote-desktop client
  • javascriptcoregtk4.0 (fc42) ← needed by that webkit
  • vamp-plugin-sdk (fc42) ← needed by Audacity, which is itself current

“Leftover” is not a synonym for “garbage.” None of those were safe to yank; the apps that depend on them were perfectly current and just hadn’t been rebuilt against newer libraries yet. (Hold onto the webkit chain — it has a quiet payoff at the end.) The only genuinely dead weight was an old kernel set, and even that would auto-prune at the next kernel install. I removed it by hand purely for the tidiness dopamine.

02Log · .rpmnew

The config markers, and a directive worth adopting

Review · .rpmnew / .rpmsave

Upgrades scatter .rpmnew and .rpmsave files — the package manager’s way of saying “I had an opinion about this config but didn’t want to overwrite yours.” Two were worth reading instead of reflexively deleting:

  • cups-browsed.conf: a post-upgrade scriptlet had quietly set BrowseRemoteProtocols none — a hardening default from the cups-browsed CVE fallout. My old saved copy had the chattier dnssd cups. I kept the locked-down none and deleted the marker. Do not helpfully “restore” your old value here.
  • chrony.conf: my custom NTP server line was intact, but the new default retired a deprecated leap-second directive in favour of a modern one:
/etc/chrony.conf
  # old, deprecated
  leapsectz right/UTC
  # new
  leapseclist /usr/share/zoneinfo/leap-seconds.list
  

Adopting it seemed harmless. Reader, it was not harmless.

03Log · One character

The space that wasn’t there

Bug · a missing space

To swap the directive in, I reached for sed. And sed gave me this:

/etc/chrony.conf
leapseclist/usr/share/zoneinfo/leap-seconds.list

Look closely. There is no space after leapseclist. A one-character wound — a malformed directive that chrony would reject outright on the next parse. The keyword and its argument had fused into a single meaningless token.

I only caught it because I’d built a reflex earlier in this journey: verify before you declare victory. Not with my eyes — eyes slide right over a missing space — but with a tool that makes whitespace visible:

shell
cat -A /etc/chrony.conf | grep leap
# leapseclist/usr/share/zoneinfo/leap-seconds.list$

cat -A doesn’t editorialise. The missing gap was right there, no $-anchored space in sight. A space is invisible until it bites; cat -A makes it bite first.

04Log · Stacked gotcha

…and the config I “fixed” had never loaded

Diagnosis · file ≠ process

Then the second gotcha, stacked on the first like it had been waiting. Even if my edit had been clean, it wouldn’t have mattered — because the chronyd running on my system was still the one from boot. Its log proudly said it was “Using right/UTC timezone to obtain leap second data” — the old directive. I had “adopted” a change that was (a) malformed and (b) not even loaded.

Editing a config file does not change a running daemon. Obvious when stated; invisible in the moment. The fix needed both halves:

shell
sed -i 's|^leapseclist/usr|leapseclist /usr|' /etc/chrony.conf   # the space
systemctl restart chronyd                                        # the reload

And only then did the log tell the truth:

journalctl -u chronyd
chronyd[34630]: Using leap second list /usr/share/zoneinfo/leap-seconds.list

A restart is not a reload, and a file is not a process. Check the thing that’s running, not the thing on disk.

A restart is not a reload, and a file is not a process.

05Log · Clock

fritz.box is a place, not a server

Fix · a roaming clock source

While I had chrony open, I asked the question I should have asked years ago: is this thing even syncing? It was not.

shell
chronyc sources
chronyc tracking   # Leap status : Not synchronised

My config pointed at server fritz.box iburst — my home router. And fritz.box only resolves when I’m home. This is a laptop. Pinning its clock to the living-room router means that everywhere else on Earth, it has no time source at all. So I handed it servers that exist regardless of which café I’m in:

/etc/chrony.conf
pool 2.fedora.pool.ntp.org iburst
sourcedir /run/chrony-dhcp     # still let the home router chime in as a bonus

A roaming machine should not depend on furniture.

A roaming machine should not depend on furniture.

06Log · 43 → ?

Choosing the next floor to stand on

Decide · let the mirrors vote

Now the actual upgrade. First question: 43 → what? I didn’t trust release dates from memory, so I let the mirror network vote. A Generally Available release has a full fleet of mirrors; a still-Branched one is sparse. Probing the metalinks, F44 had a healthy mirror count and F45 was still thin and pre-release. 43 → 44 it was — decided by infrastructure, not by my recollection of a calendar.

Then the part everyone clicks through too fast — the GPG key import:

dnf system-upgrade
Importing key 0x6D9F90A6:
 Userid     : "Fedora (44) <fedora-44-primary@fedoraproject.org>"
 Fingerprint: 36F6 12DC F27F 7D1A 48A8 35E4 DBFC F71C 6D9F 90A6

This is the moment you’re being asked to trust a stranger’s signature on everything that’s about to overwrite your OS. So I checked it before typing y: the keyfile was owned by the official fedora-gpg-keys package, rpm -V said it was unmodified on disk, and the fingerprint matched the prompt character for character. Thirty seconds to not blindly accept a key. Worth it.

07Log · dnf5

The dnf5 tell

Detail · dnf4 vs dnf5

A small thing tipped me off mid-upgrade. The status line came back as “Offline-Transaktion testen” — and that exact German phrasing wasn’t how the dnf I remembered talked. It was the giveaway that dnf on this box is now dnf5, with the old dnf4 lurking as the dnf-3 binary. That one detail explained an earlier mystery (why all the dnf4 state directories I’d gone hunting through were empty — I’d been excavating the wrong era) and reassured me about the upgrade itself: in dnf5’s system-upgrade, package conflicts fail at resolve time, before a single file downloads. The fact that it staged at all was proof my load-bearing orphans hadn’t blocked anything.

08Log · Commit

Staged, lights off

Execute · upgrade in the dark

The download left an offline transaction parked and ready. The actual swap happens in a minimal environment after a reboot, with the real system halted — no live filesystem being rewritten under running programs. Pre-flight checklist: on AC power, work saved, deep breath.

shell
dnf system-upgrade download --releasever=44   # already done, days ago
dnf system-upgrade reboot                      # commit

Then the screen goes into that stark, progress-bar-on-black offline updater, six thousand-odd packages march past, and the machine reboots into a Fedora it has never been before. Upgrading in the dark, literally.

09Log · Health check

Did it land clean?

Verify · trust nothing

It did. And because this series has taught me not to trust the calm-looking parts, I didn’t take the login screen’s word for it. The health check, in order:

shell
cat /etc/fedora-release      # Fedora release 44 (Forty Four)
systemctl --failed           # 0 loaded units listed
dnf check                    # (silence — no broken deps, no duplicates)

No failed services. No dependency wreckage. The new kernel (7.0.12-200.fc44) booted on the first try, with two fc43 kernels kept as fallbacks.

There was one heart-stopping moment in the upgrade log:

upgrade log
systemd ... Failed to start jobs: Invalid argument
systemd ... Remote peer disconnected

Pure adrenaline — until I remembered where those ran. Package scriptlets firing in the offline upgrade environment can’t talk to a system manager that isn’t fully alive yet. Those errors are expected and harmless; the calm-looking login screen was telling the truth and the scary-looking log was the innocent one. Same lesson as the black workspace and the flawless volume keys, wearing a sysadmin costume this time.

And the two questions the notes file most wanted answered:

  • Did chrony survive the jump? Completely. Post-upgrade: Leap status: Normal, four pool servers all at full reach, a sub-millisecond offset, and the log still reading Using leap second list /usr/share/zoneinfo/leap-seconds.list. The directive I’d fought a missing space for came through the release upgrade intact.
  • Did the orphans finally break? Better than that — some of them resolved themselves. The whole fc42 webkit chain I’d agonised over keeping — webkit2gtk4.0, javascriptcoregtk4.0, vamp-plugin-sdk — was simply gone, quietly replaced by fc44 builds that no longer needed the old libraries. The careful “do not remove these” detective work from the morning was rendered moot by the upgrade itself. Two stragglers remain at fc42 (xl2tpd, hfsutils), pure leaf apps with no fc44 rebuild yet — exactly the harmless kind of leftover I’d argued for keeping.
10Field notes

What I’d tell past-me (upgrade edition)

  1. Read the logs before you trust the note. “Continue” meant nothing until dnf history told me what I’d actually already done. Past-you is an unreliable narrator; the transaction log is not.
  2. Verify a config edit with cat -A. Spaces are invisible until they bite. A directive that looks right and a directive that is right differ by characters your eyes refuse to render.
  3. A restart is not a reload, and a file is not a process. I’d “fixed” a config that the running daemon had never read. Always check the live process, not just the file on disk.
  4. Don’t pin a roaming laptop’s clock to your living room. fritz.box is a place. Give a mobile machine sources that exist everywhere it goes.
  5. Read GPG fingerprints before you trust them. Thirty seconds of checking beats handing root to a signature you never looked at.
  6. Know which dnf you’re actually running. dnf4 and dnf5 keep state in different places and phrase things differently. I wasted real time spelunking the wrong era’s directories.

The recurring moral of this whole series keeps reasserting itself: on this system, nothing happens for you, so every fix is real and comprehensible — but the trouble is almost never where you first point the finger. This time it was a character I couldn’t see, in a file a daemon hadn’t read, syncing to a router that wasn’t there.

11Postscript

The cleanup that had nothing to do

Postscript · nothing to do

There’s a tidy ritual for mopping up release leftovers — dnf distro-sync, which drags every stray package into line with the new release. I’d been saving it as the satisfying final beat: the grand cleanup that would sweep away all that fc43 sediment in one cathartic transaction.

I ran it. It thought for a moment and said: nothing to do.

Every leftover was already the newest thing that exists. The orphans I’d planned to purge had either dissolved on their own during the upgrade or were genuinely the latest available build. The dramatic cleanup I’d lined up was an anticlimax by design — the system had quietly cleaned up after itself while I wasn’t looking, and left me holding a broom in a room that was already swept.

Fitting, somehow. The scariest operation in this series — gutting the OS and rebooting into the dark — went off without drama, and the part I’d saved for the triumphant finish turned out to be a no-op. The black-screen panic, the phantom volume keys, the self-shooting pkill — and now an upgrade whose only real wound was a missing space I inflicted on myself. I’m starting to suspect the machine was never the problem.

Comments (0)

No comments yet. Be the first!

Write a Comment