PKI Maintenance Runbook#

This is the operational runbook for the FalseBlue PKI. It documents the CLI procedures for the operations that can be performed without XCA. XCA is still required for any operation that needs the root key or a YubiKey-CA-resident key — the root key lives in the XCA database (pki/db/FalseBlue-CA.xdb.sops) and on the YubiKeys, accessed via opensc-pkcs11. See FalseBlue PKI for the XCA workflow.

PKI Hierarchy#

falseblue.com (Root CA)
├── expires 2032-04-10  — offline, ~6 years as of 2026
├── FalseBlue Intermediate CA  (step-ca on ereshkigal, expires 2035-08-10)
│   └── issues all TLS/ACME/SSH certs for homelab infrastructure
├── YubiKey - Nano 5c  (slot 9C on Nano, ECCP384, HIGH TRUST)
│   └── issues personal signing certs; keep this key at home/secured
└── YubiKey 5c NFC - Keychain - CA  (slot 9C on NFC key, LOWER TRUST)
    └── issues personal signing certs; kept on keychain, higher exposure risk

Hardware Reference#

Root CA Facts#

• Certificate: pki/authorities/falseblue.crt
• Private key: held in the XCA database (pki/db/FalseBlue-CA.xdb.sops) and on the YubiKeys; sops-decrypt the DB, open it in XCA, and access the key via opensc-pkcs11. The XCA DB passphrase is in Bitwarden: FalseBlue Root CA Key
• Expires: 2032-04-10 — plan renewal by 2031 at the latest
• Fingerprint: 66b8732f7a4630403455da2ef05fef269ae63e390c6b5f50f82ecdf872585fe6

Prerequisites#

All tools are available in nixpkgs. On ereshkigal they are already in PATH via environment.systemPackages. Elsewhere, use:

nix shell nixpkgs#yubikey-manager nixpkgs#openssl nixpkgs#sops

step-cli is on ereshkigal’s PATH. For other hosts: nix run nixpkgs#step-cli -- <args>.

Renewal Schedule#

YubiKey CA certs should be issued for 5–10 years per rotation. The root CA lifetime sets the ceiling — no cert it signs can outlive it, so target 2031 or earlier.

Renewing a YubiKey CA Cert#

Repeat this procedure for each physical YubiKey. The private key never leaves the hardware; you export the public key, generate a signed CSR, sign it with the root, and import the new cert back.

Step 1 — Verify the slot#

ykman piv info

Confirm slot 9C has a key (Private key type listed) and note the algorithm. The Nano is ECCP384.

Step 2 — Export the public key from the YubiKey#

ykman piv keys export 9c /tmp/yk-pub.pem

If there is no cert in the slot yet, add --verify --pin <PIN> to confirm the key is present.

Step 3 — Generate a CSR (signed by the on-device private key)#

# For the Nano
ykman piv certificates request \
  --subject "CN=YubiKey - Nano 5c,O=FalseBlue" \
  9c /tmp/yk-pub.pem /tmp/yk.csr

# For the NFC key
ykman piv certificates request \
  --subject "CN=YubiKey - 5cNFC,O=FalseBlue" \
  9c /tmp/yk-pub.pem /tmp/yk.csr

The YubiKey will prompt for the PIN. The CSR is self-signed by the slot key to prove possession.

Step 4 — Open the XCA database (holds the root key)#

The root key is not available as a standalone PEM — it lives in the XCA database and on the YubiKeys, reached via opensc-pkcs11. Sops-decrypt the DB and open it in XCA:

sops -d pki/db/FalseBlue-CA.xdb.sops > /tmp/FalseBlue-CA.xdb
nix run nixpkgs#xca -- /tmp/FalseBlue-CA.xdb
# shred -u /tmp/FalseBlue-CA.xdb when finished

Plug in the root YubiKey and confirm the opensc library path in XCA settings (/run/current-system/sw/lib/opensc-pkcs11.so on NixOS).

Step 5 — Sign the CSR with the root CA in XCA#

Import /tmp/yk.csr into the CSRs tab, right-click it and choose Sign, set Sign with to the root CA (falseblue.com), and apply a CA extension with basicConstraints = critical, CA:TRUE, pathlen:0 and keyUsage = critical, keyCertSign, cRLSign, 3650-day validity. Then export the signed cert from the Certificates tab to pki/authorities/nanoc5c-CA.crt (or nfc5c-CA.crt for the NFC key). See Renewing a Certificate Manually using XCA.

Step 6 — Verify the new cert#

openssl verify -CAfile pki/authorities/falseblue.crt pki/authorities/nanoc5c-CA.crt
openssl x509 -in pki/authorities/nanoc5c-CA.crt -noout -subject -issuer -dates

Step 7 — Import the new cert back onto the YubiKey#

ykman piv certificates import 9c pki/authorities/nanoc5c-CA.crt

Step 8 — Update CRLs and publish#

Update the CRL for each renewed CA (see Updating CRLs). Then commit the new certs:

git add pki/authorities/nanoc5c-CA.crt pki/authorities/nfc5c-CA.crt
git commit -m "pki: renew YubiKey CA certs"

Rebuild to push the updated certs into system trust stores:

nixos-rebuild switch  # on each NixOS host
darwin-rebuild --impure --flake .#<host> switch  # on each macOS host (see scripts/darwin-flake-rebuild.sh)

just Recipe#

A renew-piv-ca recipe is planned for the justfile but is not yet present. Once added, usage will be:

just renew-piv-ca 9c "CN=YubiKey - Nano 5c,O=FalseBlue" pki/authorities/nanoc5c-CA.crt
just renew-piv-ca 9c "CN=YubiKey - 5cNFC,O=FalseBlue" pki/authorities/nfc5c-CA.crt

Until then, follow the manual steps above. The NFC YubiKey must be plugged in when renewing its cert.

Renewing the step-ca Intermediate#

Only needed ~2034. The intermediate lives in sops secrets on ereshkigal.

Because the root key is held in the XCA database / on the YubiKey (no standalone PEM exists), generate the new intermediate key + CSR locally with step, then sign the CSR with the root in XCA (see Renewing a Certificate Manually using XCA):

# Generate a new intermediate key and CSR
step certificate create \
  "FalseBlue Intermediate CA" \
  /tmp/intermediate.csr /tmp/intermediate.key \
  --profile intermediate-ca \
  --csr \
  --not-after 87600h \
  --kty EC --curve P-256

# Sign /tmp/intermediate.csr with the root CA in XCA, exporting /tmp/intermediate.crt.

# Encrypt and store the new key/cert
sops -e /tmp/intermediate.key > <new-sops-secret>
# Update sops.secrets in step-ca.nix to point to the new secret paths
# Rebuild ereshkigal

Updating CRLs After Renewal#

CRLs are published at https://falseblue.com/*.crl. After any renewal or revocation:

Root CA CRL#

The root key is held in the XCA database, so the root CRL is regenerated in XCA (the same flow as the YubiKey CA CRLs below):

sops -d pki/db/FalseBlue-CA.xdb.sops > /tmp/FalseBlue-CA.xdb
nix run nixpkgs#xca -- /tmp/FalseBlue-CA.xdb
# Certificates tab → right-click the root CA (falseblue.com) → Update CRL
# CRLs tab → Export → pki/authorities/falseblue.crl
shred -u /tmp/FalseBlue-CA.xdb

Verify the exported CRL:

openssl crl -in pki/authorities/falseblue.crl -noout -lastupdate -nextupdate

YubiKey CA CRLs#

Signing these CRLs requires the on-device private key via PKCS#11. Do this in XCA, which reaches the YubiKey CA keys through opensc-pkcs11:

sops -d pki/db/FalseBlue-CA.xdb.sops > /tmp/FalseBlue-CA.xdb
nix run nixpkgs#xca -- /tmp/FalseBlue-CA.xdb
# Certificates tab → right-click CA → Update CRL
# CRLs tab → Export → pki/authorities/nanoc5c-CA.crl and nfc5c-CA.crl
shred -u /tmp/FalseBlue-CA.xdb

Publish#

Copy updated .crl and .crt files to the static/ directory in the personal-web repo and deploy.

Slot Reference (Verified 2026-06-06)#

YubiKey 5C Nano — Serial 23183421 (HIGH TRUST, keep at home)#

YubiKey 5C NFC — Serial 24918246 (LOWER TRUST, keychain)#

Slot 9E (Nano) can be renewed with a self-signed cert — no root key needed:

ykman piv keys export 9e /tmp/9e-pub.pem
ykman piv certificates generate --subject "CN=benjamin.craton@gmail.com" --valid-days 3650 9e /tmp/9e-pub.pem

CA escrow and break-glass#

Read this so you never need the next section. The root key exists in exactly two forms, and both sit behind chains a successor must be able to walk:

What this means in practice:

• Borg restore does not help. pki/db has no host recipient, so the host-key route in Restore cannot open it, and the borg credentials are themselves sops-encrypted. Recovering ~/.config/sops/age/keys.txt from a mokou/shinobu home backup is the only backup path, and it requires a working secrets.yaml key first.
• Without one of those two identity files, pki/db/FalseBlue-CA.xdb.sops is unrecoverable and the CA must be rebuilt per the next section; every device that trusts falseblue.crt then needs the new root installed.
• Why it is this narrow: hosts never need the root key (step-ca holds its own intermediate in sops), so the blast radius of a stolen host key stays away from the root. That is the right design; it just needs a third recipient that is not bound to a live machine.

Adding an escrow recipient (the fix): generate an age key offline (age-keygen), store the private half where a successor can reach it, add the public key under &users in .sops.yaml and to the ^pki/db key group, then just sops-updatekeys (it re-keys pki/db/*.sops and secrets.yaml; no new tooling). Then sops --decrypt pki/db/FalseBlue-CA.xdb.sops > /dev/null with the new identity proves it.

TODO(owner): where is the offline escrow copy of a pki/db recipient key (or the mokou identity file), and who may retrieve it?

recovery_files: what is actually there#

age1vtddaavgwu0lyr7qgtwhkz45n9vg9wr2vn8d2km8wzqs6ps0mvxqxtrk3s is the previous tsunami-mokou key, pruned from .sops.yaml in the “Update home keys and prune old keys” commit. Consequences: nobody listed in .sops.yaml can decrypt those three files, and just sops-updatekeys (which selects files by content) fails on each of them; its set -e loop aborts at the first, leaving every later file un-rotated. Resolution, in order:

1. sops -d pki/db/recovery_files/private_keys.yaml from any machine that might still hold the old identity. TODO(owner): does the pruned mokou age key survive in a home backup?
2. If it decrypts: sops updatekeys --yes all three so they match the ^pki/db rule, and rename them to private_keys.yaml.sops, f9e6b4b8-….pfx.sops, pve-….enc.sops (the .pfx rename also takes it out of .gitignore’s *.pfx block, which it currently violates as an already-tracked file).
3. If it does not: git rm them — undecryptable ciphertext is ballast, not a recovery file.

Until then, expect just sops-updatekeys to stop partway; run sops updatekeys --yes on secrets.yaml and the four XCA-*.sops files by hand if you need a rotation to complete.

If the Root Key is Lost#

The root key lives in the XCA database (pki/db/FalseBlue-CA.xdb.sops); the Bitwarden entry FalseBlue Root CA Key is the passphrase that unlocks that database. If the Bitwarden passphrase and the XCA database are both gone:

1. Generate a new root CA with step certificate create --profile root-ca
2. Re-sign the step-ca intermediate and both YubiKey CAs from the new root
3. Distribute the new root cert across all hosts via pki/authorities/falseblue.crt + rebuild
4. Update the fingerprint in pki.md and the step-ca bootstrap command
5. Re-issue all leaf certs via ACME (they will auto-renew from step-ca once the new intermediate is in place)

Equipment using the old root (iDRAC, Synology, etc.) will need the new root cert installed manually.