Guide 46 of 61 · Running a fleet with Central Lite
Keep Central Lite's keys in a hardware security module
Create Central Lite's org with every key wrapped under a key that never leaves your PKCS#11 token, and keep what a restore needs backed up.
Built, not yet claimed preview: this guide describes a package that has not passed its release gate.
Create Central Lite's org with its keystore on a PKCS#11 token: an HSM, or SoftHSM2 for a trial. Every key Central Lite holds is wrapped under one key that the token generates and never lets out, so the key files on disk open only with that token and its PIN. At the end every key reports pkcs11-wrapped custody, and you know what to keep backed up and what a wrong PIN or another token does.
1. Prepare a token for Central Lite
Central Lite reaches the token through its vendor's PKCS#11 module, a shared library on the Central Lite host, and finds it by its label, agentgarm unless you name another. Initialise the token with that label and a user PIN using your HSM's own tools, and make sure the account that runs Central Lite can use the module. To try it without hardware, SoftHSM2 gives you a software token; softhsm2-util asks for the PINs when you leave them off its command line.
Choose before you create the org: the keystore is recorded in the org when init runs, and an existing Central Lite cannot move its keys onto a token later.
softhsm2-util --init-token --free --label agentgarm2. Put the PIN where only Central Lite reads it
Central Lite reads the token's user PIN from keys/pkcs11.pin in its state directory, never from its configuration or the environment. Create the state directory and its keys directory, both 0700 and owned by the account that runs Central Lite, then create the PIN file as 0600 and write the PIN into it on one line with an editor, so it never lands in your shell history.
Central Lite refuses a PIN file that group or others can read, one that is not a plain file, and one that is empty. Anyone who can read this file and reach the token can unwrap Central Lite's keys, so guard the host account as you guard the token.
sudo install -d -o "$(id -un)" -m 0700 /var/lib/agentgarm-central
install -d -m 0700 /var/lib/agentgarm-central/keys
(umask 077 && touch /var/lib/agentgarm-central/keys/pkcs11.pin)3. Create the org with its keys wrapped on the token
Give garm-central init the keystore pkcs11: followed by the module's path, and ?token= with the label when it is not agentgarm; quote it, since the shell would read ? as a pattern. On first use the token generates an AES-256 key labelled agentgarm-central-kek: sensitive, never extractable, and good only for encrypting and decrypting.
Every key Central Lite holds is then 32 bytes of the token's randomness, encrypted on the token under that key, with the key's own name bound in, and stored as a 0600 .p11 file in the keys directory: the signing keys for policy, revocation, approvals and the audit head, the org root and evidence keys that root lite adds, the node certificate authority, the TLS key nodes pin, the single sign-on client key, and the sealing key that protects the tenant keys, each backup's key and the credentials Central Lite keeps. A file opens only under its own name and only with that token. Run this init instead of the one in Stand up Central Lite, since the keystore cannot be moved later; the rest of that guide stays the same.
garm-central init --url https://central.example:8443 --keystore 'pkcs11:/usr/lib/softhsm/libsofthsm2.so?token=agentgarm'4. Check that every key reads pkcs11-wrapped
garm-central status shows the keystore and every key with its key_custody, which reads pkcs11-wrapped for each one. That means the files at rest open only with the token: while Central Lite runs, it unwraps its keys into its own memory and signs there, so the token protects the key files, not a running Central Lite. Keeping each key inside the HSM, and a cloud KMS (kms:, which is refused rather than replaced by anything else), are not built.
Nodes still report the org root of root lite as software custody: it is a single key on the same host as the online keys, whatever holds it. A production org needs an offline root of two or more keys, signed in a ceremony. This keystore has been tested against SoftHSM2; no run against a hardware HSM is recorded yet.
garm-central status5. Keep the key directory and the token backed up
A restore needs three things: a backup, the keys directory with its .p11 files and its secrets, and the token holding agentgarm-central-kek, with its PIN. Each backup's key is sealed in that keystore, never in the backup, and the keystore is in no backup, and every backup, scheduled ones included, and every credential Central Lite stores adds a sealed key under keys/secrets. So copy the keys directory somewhere apart from the backups after each backup, or on the same schedule: a copy taken only after init restores no later backup.
AgentGarm never exports the token's key, so the only copies of it are the ones your HSM makes under its own backup or cloning scheme. If the token and every copy of its key are lost, the .p11 files open for no one: Central Lite's keys are gone, with every backup and the TLS key the machines pin.
garm-central backup create
garm-central backup listSee also
Back up and restore Central LiteWrite Central Lite's encrypted backups by hand and on a schedule, and restore one so that no enrolled node refuses what Central Lite signs next.
6. Know what a wrong PIN or another token does
Central Lite fails closed. With a wrong PIN the token refuses the login, which also counts against the token's own retry limit if it has one, and Central Lite opens no keys, so serve does not start. With no token of the label in the module it says so and stops.
With another token, Central Lite never makes a new key over files the old one wrapped. If the token has no agentgarm-central-kek but the keys directory holds wrapped keys, it refuses and asks you to restore the token's key rather than create another. If the token has a key of that label of its own, each wrapped file fails to unwrap under it. A copied or edited .p11 file, or one renamed to another key's name, fails the same way, and a missing sealing key beside sealed secrets is refused rather than replaced.