The call was simple enough: an HPE ProLiant DL380 Gen9 running SUSE Linux Enterprise Server 15 — the machine hosting a customer's SAP HANA database — wouldn't come back up after a shutdown. Instead of a login screen, the console showed an emergency shell. By the time the box was truly healthy again, I had found and fixed five separate problems, none of which had anything to do with each other. This is the whole story, wrong turns included, because the wrong turns are where the lessons live.

Scary messages that meant nothing, and a quiet one that meant everything

The console greeted me with genuinely alarming text: [Firmware Bug]: the BIOS has corrupted hw-PMU resources (MSR 38d is 330), plus a wall of CPU security advisories about MDS and MMIO Stale Data. When a server won't boot and the screen says "Firmware Bug", it takes discipline not to chase that first. I made a note of it (it matters later, in Act 3) and moved on — none of those messages were the reason the boot stopped. They appear on every boot of this machine, healthy or not.

My first real hypothesis was a classic for this hardware: the initrd — the small starter filesystem the kernel uses before the real disks come online — missing the driver for the HPE Smart Array P440ar RAID controller. On ProLiant boxes that's a familiar failure. But it didn't hold: the disks were visible from the emergency shell, and the root filesystem's UUID matched what the kernel command line expected. Wrong turn number one, ruled out in a few minutes precisely because I checked instead of assumed.

The actual clue was almost invisible: a gap in the timestamps. Scrolling through journalctl -xb, the log just… paused for about 89 seconds. Nothing crashed, nothing errored — the system stood still. And 90 seconds happens to be systemd's default timeout for waiting on a device.

# What failed, and why?
journalctl -xb          # full log of this boot — look for time gaps
systemctl --failed      # which units gave up
ls /dev/disk/by-uuid    # which disks actually exist right now
cat /proc/cmdline       # what root device the kernel was told to use

systemctl --failed then said it in plain language: Timed out waiting for device — followed by a UUID in the short FAT format that USB sticks use — and Dependency failed for /mnt/usb. Someone had once plugged in a USB drive, added it to /etc/fstab (the file that lists what gets mounted at boot), and later walked away with the drive. Without the nofail option, systemd treats every fstab line as a promise: if the device doesn't appear, the boot is considered broken, and you get the emergency shell. CAUSE 1

# Before: boot hangs 90s, then fails, if the stick is absent
UUID=XXXX-XXXX  /mnt/usb  vfat  defaults                            0  0
# After: boot continues without it, and only waits 5 seconds
UUID=XXXX-XXXX  /mnt/usb  vfat  nofail,x-systemd.device-timeout=5s  0  0

The humbling part: the very first messages on the screen had named the culprit exactly. "Timed out waiting for device… Dependency failed for /mnt/usb" is not a riddle. The lesson I keep relearning is to read systemd's words literally before forming theories. While in there, the journal also showed FAT-fs (sda1): Volume was not properly unmounted on the EFI boot partition — scar tissue from earlier unclean shutdowns — so that got a fsck.vfat cleanup too. One reboot later, the server was up. The story could have ended here. It did not.

"Certificate verify failed" — everywhere

With the box booting, the customer's application needed a Python package update. pip refused: CERTIFICATE_VERIFY_FAILED. Annoying but common — usually an application-level issue. Except curl failed against the same host too (error 60, the certificate one). Then zypper ref failed on every single repository. That escalation changed the diagnosis completely: this wasn't an app problem, the whole operating system had stopped trusting the internet.

There was even a chicken-and-egg twist: the obvious "reinstall the ca-certificates package" repair doesn't work when downloading the package itself requires the TLS that's broken. (If you're ever fully cornered: fetching one RPM with verification disabled is an acceptable last resort, because RPM packages carry their own GPG signatures that get checked on install. But it's better to fix the actual cause.)

For system-wide TLS failures I walk a fixed ladder, cheapest check first:

• The clock. Certificates are valid from-until; a wrong system date breaks all of them. (Fine here.)
• The trust store. Does /etc/ssl/ca-bundle.pem exist and contain a healthy ~140+ certificates? (Fine here.)
• What the server actually sends. openssl s_client shows the certificate chain exactly as presented — and its verify error number is the fastest route to the root cause.

echo | openssl s_client -connect pypi.org:443 -servername pypi.org 2>&1 | grep -E "s:|i:|verify"
 0 s:CN = pypi.org          ← leaf, signed by…
   i:CN = corp-SubCA        ← …an intermediate that is nowhere in the chain
verify error:num=2:unable to get issuer certificate

Two things jumped out. First, pypi.org was presenting a certificate issued by corp-SubCA — the customer's own internal CA, not anything PyPI would ever use. Second, verify error num=2 means "I can see who signed this, but I can't find that signer's certificate". Together they told the whole story.

What a TLS-inspecting proxy actually does

This network, like many corporate networks, runs a security appliance that inspects encrypted traffic. People imagine their connection reaching the website with a checkpoint in the middle. In reality there is no single connection at all — there are two:

Your machine never sees the real website's certificate. It sees an imitation minted by the proxy, signed by the company's private certificate authority. That's not malware — it's the appliance doing its declared job — but it only works if every machine trusts that private CA, and if the proxy hands over the complete paperwork. Which brings us to what "complete" means. A certificate chain has three links: a root CA (installed on your machine, the anchor of trust), an intermediate (signed by the root, does the day-to-day signing), and the leaf (the certificate for the actual website). The server is supposed to send the leaf plus the intermediate; your machine supplies the root.

The proxy here served the forged leaf and, oddly, the root — but not the intermediate corp-SubCA. Broken chain, no verification, no TLS, anywhere. CAUSE 2

And why did every Windows laptop in the building work fine against the same proxy? Two reasons. Windows machines get the corporate CAs pushed automatically via Group Policy, and Windows performs "AIA chasing" — when an intermediate is missing, it follows a URL embedded in the certificate and downloads the missing link on its own. OpenSSL, which nearly everything on Linux uses, deliberately refuses to do that. So "works on Windows, fails on Linux" is practically a signature for exactly this condition: incomplete chain.

cp corp-RootCA.crt corp-SubCA.crt /etc/pki/trust/anchors/
update-ca-certificates
trust list | grep -c corp   # confirm both landed in the store

With both the root and the missing intermediate in the system trust store, OpenSSL can complete the chain locally. curl worked. zypper refreshed all repositories. (The genuinely correct fix is configuring the proxy to serve its full chain — that's a ticket for the network team; the server-side fix unblocks you today.)

The problem that came back twenty minutes later

pip worked — so the first thing that happened was, naturally, pip install --upgrade pip. And TLS errors immediately returned, in the venv only. Nothing on the system had changed; pip itself had. SUSE ships a patched pip that reads the system trust store — the one I had just fixed. Upgrading replaced it with the upstream version from PyPI, which ignores the system store entirely and uses its own bundled certificate collection (a package called certifi) that has never heard of corp-RootCA. CAUSE 3 One line makes pip typo-proof and permanent:

pip config set global.cert /etc/ssl/ca-bundle.pem

The application itself — a Python-based security-feed aggregator, essentially an RSS reader for vulnerability announcements — failed the same way at runtime, for the same certifi reason. But its failure pattern was diagnostic gold: only the Microsoft and Google/Feedburner feeds worked, everything else failed. Those were precisely the domains on the proxy's SSL-inspection bypass list — traffic to them passed through untouched with genuine, publicly-trusted certificates, while every inspected domain got the corporate forgery that certifi rejects. When some TLS destinations work and others don't, ask what the working ones have in common on the network, not in the code. CAUSE 4

The five-minute fix that took an hour

The standard fix for Python's requests library is an environment variable pointing at the system bundle. I set it. It changed nothing. I checked the service, the shell profile, the restart order — nothing. The actual problem, discovered embarrassingly later:

REQUEST_CA_BUNDLE=/etc/ssl/ca-bundle.pem    # what I typed — ignored silently
REQUESTS_CA_BUNDLE=/etc/ssl/ca-bundle.pem   # what the library reads — note the S

One missing letter. CAUSE 5 And this is the quietly vicious property of environment variables: a misspelled one isn't an error, it's just a variable nobody reads. No warning, no log line, nothing. Since then I prefer mechanisms that can't be typo'd into silence — pip config as above, or setting the default in the application's own code with os.environ.setdefault("REQUESTS_CA_BUNDLE", "/etc/ssl/ca-bundle.pem"), where at least a typo lives in one reviewable place.

Closing the loop — and dodging a downgrade

Remember the [Firmware Bug] message from the very first screen? With everything else healthy, I circled back to it and applied HPE's final Service Pack for ProLiant Gen9 — the 2022.08 SPP, the last firmware bundle this generation will ever receive — using SUM in online mode, straight from the running OS.

And there sat the trap. SUM showed several components with a "Forced" toggle — which sounds like "extra thorough" and actually means "the version I'm offering is not newer than what's installed". The server's BIOS (P89 v3.30, from 2023) and iLO (2.82) were already newer than what the 2022 SPP carried (P89 v2.92, iLO 2.81), because those components kept receiving individual security updates after the final bundle was cut. Clicking deploy-everything would have downgraded the system board firmware on a production HANA host. On end-of-life hardware, the newest official bundle can be older than your machine — always read the Installed vs. Available columns before deploying.

So the deployment was selective: NIC firmware, drive firmware (HPD4 → HPD6), and the management tooling (ssacli and friends) — the pieces that were genuinely newer. Then the verification lap:

sut -status                     # HPE agent happy, nothing pending
ssacli ctrl all show status     # RAID controller / cache / battery: OK
journalctl -b -p err            # current boot, errors only: quiet

One note for anyone else keeping Gen9 machines alive: there will be no further SPPs, ever. Future fixes arrive as individual component updates — uploaded through the iLO web interface or installed as Linux .scexe/.rpm packages. The era of the one big bundle is over for this hardware.

What I'm taking with me

• One symptom, many causes. A single visible failure can sit on top of several independent problems. Fixing one layer and seeing a new error isn't failure — it's progress.
• Read systemd literally. "Timed out waiting for device X" names the culprit. Believe it before building theories.
• ~90 seconds is a signature. A minute-and-a-half gap or hang almost always means systemd waiting for a device or mount. Check fstab.
• Always add nofail to removable and network mounts. A missing USB stick should never cost you a boot.
• For TLS failures, get the verify error number. openssl s_client's num=2 pointed straight at the missing intermediate. Fastest tool in the box.
• "Works on Windows, fails on Linux" for TLS almost always means an incomplete chain plus Windows' AIA chasing papering over it.
• Distro-patched tools are not upstream tools. SUSE's pip trusted the system store; the upgrade quietly didn't.
• Environment variables fail silently. Prefer config files or in-code defaults for anything that matters.
• Never deploy "Forced" firmware blindly. On EOL hardware the last official bundle may be older than your installed versions.