From a PEM to a Running Stack — Part 3: Three Passwords, Two Directories, One Mistaken Certificate
Part 3 of the series. The certificates were technically fine. Still nothing started. This part is about the errors that don't live in the crypto but in the configuration — the ones hardest to find because everything "looks right."
When the error stays exactly the same
After the keystore file was finally in the right format and held a real entry, I restarted the service — and got exactly the same error as before:
ERROR: keystore password was incorrect javax.crypto.BadPaddingException: Given final block not properly padded
Here lies one of the most important lessons of the whole project: if nothing changes after a change, you're probably changing the wrong place. Instead of fiddling further with the file, you have to step back and check whether the service even reads the file you're editing — and with the password you assume.
The detective work: which password does the service actually expect?
The stack drew its configuration from several env files that override one another (defaults in *-default.env, custom tweaks in *-custom.env). Such layered systems are handy, but they obscure which value ends up actually applying.
The decisive step was not to guess, but to ask the service directly what arrives at its end:
docker compose run --rm --entrypoint sh keycloak -c \ 'printf "PW=[%s]\n" "$KC_SPI_TRUSTSTORE_FILE_PASSWORD"'
The square brackets are deliberate: they make it visible whether an invisible space or a Windows line break (\r) is lurking at the end of the value — something like [changeit ] or [wert\r] would otherwise be impossible to spot.
The result was an eye-opener: the service expected a different password than the one I had built the file with. There were in fact several password variables in play, for different purposes — and I had been using the wrong one the whole time.
This stack actually had three different password settings that all sounded similar but controlled different things:
- one for the system truststore of the auth server,
- one for the HTTPS server keystore (the certificate the service itself offers TLS with),
- one for a third, service-specific variant.
On top of that came another service (a REST API middleware) that had its truststore password hard-wired into the start script and wasn't configurable via env files at all.
EVIDENCE: when services share a file, they must agree on one password — a file can only have one.
The solution wasn't a technical one but an organizational decision: unify them all to the same value. Since the one service had its password hardcoded and couldn't be changed, I set all the remaining variables to exactly that value:
KC_HTTPS_KEY_STORE_PASSWORD=<TRUSTSTORE_PW> KC_HTTPS_TRUST_STORE_PASSWORD=<TRUSTSTORE_PW> KC_SPI_TRUSTSTORE_FILE_PASSWORD=<TRUSTSTORE_PW>
— and of course rebuilt all the keystore files with this very password.
VERDICT: with password mismatches, the question is never just "is the file's password correct?" but "does it match what this particular service expects?" And: if several services share a file, they have to agree on one password.
A tip on the side: env files and line endings
Because the value came from an env file that may have been created on Windows, it's worth running after every edit:
sed -i 's/\r$//' service-custom.env
A \r at the end of a password value is invisible, but it turns changeit into changeit\r — and thus into a different password.
After the passwords were sorted, the auth server ran. But the middleware component still threw a trust error on the TLS connection to the auth server — even though I had imported the matching CA certificate.
Again it held true: if nothing changes despite a repair, the service is probably reading a different file. So look at what the container actually mounts:
docker inspect <container> \
--format '{{range .Mounts}}{{.Source}} -> {{.Destination}}{{"\n"}}{{end}}'
EVIDENCE: the auth server drew its certificates from one directory (/host/ssl/), the middleware from a completely different one (/host/installation/certs/). I had been fixing the right file in the wrong place the whole time.
VERDICT: "where is the file the container really reads?" is a question you should answer early and explicitly — don't assume all services share the same path. docker inspect (or docker compose config) shows the truth.
The most insidious mix-up of the entire project: my truststore held an entry with the alias corp-ca and a valid trustedCertEntry. Looked perfect. The trust check still failed.
The reason: the entry was named corp-ca, but its content was the server end certificate (the one of the ERP host), not the actual CA root certificate. I had mis-assigned a file early on and had been consistently dragging the wrong certificate along under the right name ever since.
What uncovered it was a fingerprint comparison. You can't rely on the alias — only the fingerprint tells the truth:
EVIDENCE: an entry in the keystore proves nothing about its content. Only the fingerprint is a reliable identity.
# What's in the truststore?
keytool -list -keystore root.p12 -storetype PKCS12 -storepass <TRUSTSTORE_PW>
# -> corp-ca ... Fingerprint (SHA-256): 79:75:09:...
# What does the server actually serve? Look at the chain:
echo | openssl s_client -connect keycloak.example.local:5443 -showcerts 2>/dev/null \
| awk '/BEGIN CERT/{c++} {print > "cert"c".pem"}'
for f in cert*.pem; do
openssl x509 -in "$f" -noout -subject -issuer -fingerprint -sha256
done
In a true CA root certificate, subject and issuer are identical (it's self-signed). Exactly that one belongs in the truststore — and its fingerprint was a completely different one from the one I had in there. Only importing the right certificate (verified via the matching fingerprint) solved the trust problem.
VERDICT: verify certificates by their fingerprint, not by their name. The name lies, the fingerprint doesn't. And for a trust chain you need the CA root certificate (subject = issuer), not the end certificate.
What Part 3 taught me
- If the error doesn't change after a change, the change was in the wrong place. Step back.
- Ask the service which value arrives at its end (
printf "[%s]"with brackets), instead of resolving env layers in your head. - Several similar password variables are a classic trap — clarify which one the service really reads.
- If services share a file, they need the same password. Unifying is often more pragmatic than juggling.
docker inspectreveals which file a container actually mounts — don't rely on assumptions about paths.- Verify certificates by fingerprint, never by alias. The name lies, the fingerprint doesn't.
- Truststore = CA root certificate (subject = issuer), not the end certificate.
In the finale, it's about the last hurdle, which has nothing more to do with broken files but with the trust architecture itself: what do you do when a service has to trust an internal and a public certificate authority at the same time?