OpenBao and Shamir shares: how I built a secrets store

This is an article about a secrets store I have built and not yet opened. The deployment pipeline is finished, all five share holders have passed the rehearsal, and the initialisation ceremony, after which the store starts holding real secrets, has not happened yet. I am writing about it because of that, not in spite of it: everything interesting in this subject happens before the ceremony, and almost nobody writes about it.

Guides to secrets stores usually end at the initialisation command. From there on everything is assumed to work. Below is what that “from there on” actually consists of: how unsealing is arranged, why I turned down a hardware security module as the seal, which checks came back green exactly where they should have gone red, and what is still unverified. Versions and dates were checked against primary sources as of 24 September 2026.

Status: built, rehearsed, not yet running

The store is OpenBao in a container, with Raft integrated storage and Shamir unsealing. The deployment is described in Ansible code: eleven playbooks for initialisation, unsealing, rekeying, rotation, verification and backup, plus about twenty-five helper scripts. The store is not exposed to the outside: it is reachable only from the local network and through a tunnel.

The end-to-end tests run the real playbooks, not simplified copies of them: a container comes up, the ceremony runs, shares are decrypted, the store is unsealed, engines and the break-glass login are configured, and the list of guarantees is checked. Only the unavoidable is substituted: a throwaway certificate instead of a public one, and throwaway encryption keys instead of the real ones that live on hardware keys. The tests also check that a second run changes nothing and that a second ceremony is refused.

What does not exist yet: the production ceremony, a version upgrade run, key and audit log rotation on the production node, and a restore from snapshot into a new cluster. I am not passing a test bench off as operation, and everywhere below that a claim has only been checked on the bench, it says so.

Why OpenBao rather than Vault

OpenBao is a fork of HashiCorp Vault under the MPL-2.0 licence, developed within the OpenSSF at the Linux Foundation. Since 2023 Vault has been distributed under the Business Source License, and for me that settles it: a store that holds access to everything else should not depend on how somebody else’s licence terms change. Add signed package repositories and namespaces without a commercial edition.

The price of that choice is concrete: OpenBao has no official Ansible collection, so I had to write the deployment role myself. It is not a catastrophe, but it is a week of work that choosing Vault would not have cost. I paid it deliberately: a role written for your own problem is easier to understand than someone else’s, and every decision in it is recorded next to the code.

Shamir’s scheme cannot make a share mandatory

The master key is split into five shares, and any three of them unseal the store. Three shares are encrypted to hardware keys, one is written on paper, and one is protected by a password. The requirement everything started from was simple: it must not be possible to open the store without a hardware key.

And here is the non-obvious part. Shamir’s scheme cannot make a share “mandatory”. Any three of five are mathematically equal, and you cannot require, at the level of the scheme, that every set of three include a share from a hardware key. The guarantee can only come from arithmetic: the number of shares that open without a hardware key must be strictly less than the threshold. Here there are two of them against a threshold of three, so no set of three comes together without a token.

The first version of the layout broke this invariant: two passwords, the paper, and a copy in the password manager, which is four shares without a token against a threshold of three. The store would have opened with no hardware key at all, and it looked entirely reasonable. The mistake was caught before the ceremony rather than after, and that is the only reason it could simply be corrected. After the ceremony the layout can only be changed by a full rekey.

Which failures the layout survives

A threshold of three out of five was chosen so that losing one factor does not kill the store. The normal unseal uses the paper, the password and any one hardware key. I tested the layout not against the happy case but against failures:

• the password manager holding the password copy is unavailable: the password is remembered;
• the password is forgotten: there is the copy in the password manager, and if that is unavailable too, the paper and two hardware keys;
• the paper burns: a hardware key, the password and a second hardware key;
• the everyday hardware key is lost: the other two keys plus the paper or the password;
• both the paper and the password manager are stolen: the attacker holds two of the three shares needed, and the threshold is not reached.

One deliberately accepted inconvenience: after every restart somebody has to physically go and fetch the paper. And one dependency I wrote down in advance so as not to forget it: the password copy lives in a password manager that is independent of the store today. If the store ever starts holding the password manager’s own secrets, the layout has to be revisited, or unsealing will require the very thing being unsealed.

I bought a hardware module and did not make it the seal

Stores in this class come with an obvious temptation: a hardware security module as the seal, so the store unseals itself with no people involved. I bought a YubiHSM 2 and, after working it through, turned that role down. The device sits in its factory state, and there are four reasons.

• Logical. “Cannot be opened without a hardware key” and “the machine opens itself” are mutually exclusive. With the module as the seal, the only factor left is a PIN in the container’s environment plus a device in the same host.
• In the code. In the OpenBao 2.6.1 sources, auto-unseal disables encrypting shares to holders’ keys, and the share count and threshold are hard-set to one. A five-of-three scheme has nowhere to live.
• Risk. One USB device whose loss means irreversible loss of the entire store, snapshots included.
• Timing. Version 2.7.0, released on 23 September 2026, moved the built-in PKCS#11 seal out into an external plugin and discontinued the separate HSM build. A replacement exists, but it is an early-version plugin, and I do not put a root of trust on it.

Declining closes nothing for good. Migrating from Shamir to a module-backed seal later keeps the same five shares as recovery keys with the same threshold, and the reverse migration is supported as well. So the order “Shamir now, the module as a future option” loses nothing. The module itself found a job where it is genuinely strong: the offline root of an internal certificate authority, with no connection to the store at all. Why such a root must not live on a host that faces the internet is something I covered in the reverse proxy piece.

An official-looking guide ties the store to one device forever

This is the finding that paid for the whole investigation. The OpenBao repository has an open pull request with documentation for auto-unseal through a YubiHSM. It looks entirely official, but the key creation command in it is missing one flag: exportable-under-wrap.

Yubico’s documentation is unambiguous on this: an object’s capabilities are set when it is created or imported, and they cannot be added to an existing object. A key created without the right to be exported under wrap can never be copied or restored onto another device. Anyone who copies the guide as written ties the store to a single device forever and finds out on the day that device dies.

As of 24 September 2026 the pull request is not merged and the flag is still absent. If I ever come back to a module-backed seal, the key will be created only with the export-under-wrap right and backed up before the store is initialised. The order matters more than anything else here: after initialisation there is nothing left to fix.

Auto-unseal removes the people, and the protection goes with them

The second way to get rid of the people is to have one store unseal another through the transit engine. I turned it down for two reasons. First, the second store also has to be unsealed somehow, so the chicken-and-egg problem simply moves up a floor.

The second reason matters more, and it is almost never said aloud: recovery keys are not unseal keys. With auto-unseal, the shares you hold let you perform certain recovery operations but do not open the store themselves. Lose the unsealing store and the main server becomes a brick for good, including from backups. The convenience is paid for with a point of failure that cannot be recovered.

The holder rehearsal is mandatory because the ceremony is irreversible

This is the most underrated part of the whole design. Being able to encrypt a share to a holder’s key does not mean the holder can open it. The token may not be where it should be, the PIN may not come back, the paper may not be legible. Finding that out on the day of an outage is too late, and the initialisation ceremony cannot be undone.

So before the ceremony comes a rehearsal: every holder decrypts a control share through the same code path the store will use. The initialisation playbook refuses to run until the rehearsal has passed. All five shares were confirmed: the password really is remembered, the paper really is legible and can be typed in, and every token really does open its own share.

The rehearsal also turned up a genuine hardware problem. One of the keys carries both a store share and the certificate used for signing commits. Selecting the encryption applet on that card breaks the signing session: the agent’s log filled up with 254 PIN mismatch lines, and the card kept disappearing from the system. Now the signing agent stops itself while shares are being handled and comes back afterwards, even if the process is interrupted. The normal set of three is arranged so that this key usually is not needed at all.

And a finding from the same work that genuinely unsettled me: a configuration line meant to keep the encryption tool away from the production card had no effect at all. Filtering by reader name is ignored, and only the numeric index works. The fact that the tool had been landing on the right card up to then was luck, not configuration.

Shares are handed out by name, not by number

Unsealing happens on a bad day: the store has sealed itself, something has fallen over, people are on edge. Anything in the tooling designed for a calm person breaks on that day. So several decisions were made not on cryptographic grounds but for the sake of a tired human.

• Shares are handed out by holder name, not by index. In the artifact they sit in an array, and mixing up someone else’s share under stress is one digit away.
• After a share is shown, the screen is cleared together with the scrollback.
• Irreversible actions require typing “YES” in capitals. Not pressing Enter, typing the word.
• One console instead of ten commands with flags. It shows the state, names the next step and prints the exact command before running it. It hides nothing: everything it does also works by hand.
• An unseal assistant sees which tokens are inserted, asks for the PIN for each share and makes up the rest with the password and the paper.

And a measurement that forced the unseal sequence to be reworked. A wrong share returns a success response and advances the progress counter. The store does not say “that share is not ours”, it simply counts it. An interrupted series leaves progress on the server below the threshold, for instance two shares out of three. So every series starts with a mandatory reset, or the next person begins from somewhere other than zero and cannot work out why nothing opens.

Checks that stay green when they should go red

This is the main class of finding across the whole project, and it recurs in very different places. A check reports “all good” in exactly the situation it exists to catch.

Audit. The standard way to confirm the audit log works is to ask the store which audit devices are enabled. Measured: the answer is green even for a device that writes not a single byte. The only honest check is to send an identifiable request and find it in the log file. “Audit is enabled” in a security report while nothing is being written costs more than an honest “there is no audit”.

Capability queries. The store lets you ask “what rights does this token have on this path”. Measured: for some internal paths the query answers “denied” while the request goes through, including with no token and with a deliberately garbage token. Some paths simply never reach the permission subsystem. For checking denials, that is the worst possible kind of error: green exactly where access exists. Every denial now has two independent checks: the capability query and a live request that is required to be refused.

And the rule that came out of it: a guard that cannot go red is indistinguishable from a missing one. Every denial check is run in both directions on a throwaway bench: a login without rights must be refused, and a login with rights must make the check fail.

One word in a policy handed back root-level access

The most expensive mistake, and it was sitting in my own defence from the day before. A review through eight lenses, with three refuters per finding, produced 32 confirmed findings, and the main one was this.

The check “this role cannot write” was built as a list of what is forbidden: create, update, delete and elevated rights. The patch capability was not on that list. Measured by hand: a role allowed only to read and patch policies was refused an ordinary write, but rewrote the policy successfully through PATCH and granted itself rights to everything. Both lines of defence stayed green, because they only tried the kind of write that was on the list.

It was fixed by inverting the logic: instead of a list of what is forbidden, there is now a list of what is allowed. For a role without write access, exactly read, list and deny are permitted, and everything else counts as a write. Listing what is forbidden means losing to every new capability the server gains.

The same review found a second problem of this class: separation of duties that does not separate. A role that writes policies can rewrite its own policy with the same token. The separation creates the appearance of escalation in the audit log, not its impossibility. The statement “this role does not read secrets” in a security report would have been a lie. There is only one honest fix: take policy writing away from that role and make any change of rights a ceremony that requires a threshold of shares.

The watcher was dead while every run came back green

The store contains a honeypot: a secret nobody needs, so any read of it means someone is walking where they should not. Next to it runs a watcher that checks every five minutes whether the honeypot has been read. The scheduler job launched faithfully, exited with a success code and printed “honeypot metrics updated”.

Inside it, the watcher was crashing. The macOS scheduler gives jobs a stripped-down search path that contains neither Ansible nor Docker, and the child process could not find the tool it needed. Success, meanwhile, was defined as the metrics file merely existing, not as it being fresh. The result: the honeypot read counter was frozen. Had anyone read the honeypot during that time, the counter would have stayed at zero. Both channels were silent: the run reported success, and the error file was empty because the crashed process’s output went nowhere.

It is worth saying something about my own mistake in the analysis. At first I explained the infrequent runs by claiming the scheduler holds back jobs that exit non-zero. That was wrong, and the scheduler’s own documentation disproved it: the laptop was going to sleep, and missed runs were coalesced into one on wake. I removed the wrong explanation from the records rather than leaving it next to the right one: keeping a false fact in a project is worse than not knowing.

The lesson is written down separately: “scheduled” and “working” are different claims, and only the first was being checked. The guarantee now checks not that the job is on the schedule but that the measurement is fresh, with a threshold three times the interval so that it does not go red every time the lid closes. A false alarm devalues a guarantee just as reliably as silence does.

An alarm nobody reads any more is worse than no alarm

One morning the test store sealed itself when Docker restarted. The monitoring run sent the same notification, with a sound, every five minutes, and by lunchtime there were about forty of them. Formally the watcher had fired. In practice it had devalued itself: a stream of identical shouts reads as background noise, and the next, real alarm drowns along with them.

There are now three rules. The first notification arrives immediately, and repeats come no more than once an hour. A new alarm on top of an existing one sounds at once, without waiting for the hour of silence to end. When the condition clears, the person hears about it once and without a sound.

And the notification names the cause, not the symptom. Not “the metrics server did not respond” but “the store is sealed, open it from the console, menu item such-and-such”, with the item’s name exactly as it appears in the menu. A symptom makes you investigate from scratch every time; a cause lets you press a button. I deliberately did not make it restart what had fallen over: self-healing hides a recurring fault, and the decision whether to restart something in production stays with a person.

A version upgrade is the only operation that seals the store

Of all the maintenance operations, only a version upgrade seals the store: the image changes, the container restarts, and it has to be opened again with three factors. Encryption key rotation, audit log rotation and backups all run without that. So an upgrade is not a technical operation but an event, for which the share holders have to be available before it begins.

And it is unavoidable. OpenBao has no long-term support line: fixes go only to recent versions, and the abandoned 2.5 line was left with published vulnerabilities that are no longer carried over to it. So a forced upgrade is built into the design itself. You can see it right now: the deployment is pinned to 2.6.1, and on 23 September 2026 both 2.7.0 and 2.6.3 were released, with 2.6.3 fixing eight vulnerabilities, including request data leaking into the audit log in plain text. The store has not even been opened and it is already two security releases behind.

Before an upgrade a set of readiness checks runs, and those checks have been tested against eight scenarios. The upgrade itself has never been carried out. It is the most dangerous open item in the whole design, and I am naming it plainly: the worst outcome here looks like three holders standing at a server that will not come up, with the store sealed.

What is not verified yet

Every claim that something depends on but that has no proof lives in a separate register. A line is closed only by a measurement or a reference to source, and “looks plausible” does not close it. As of today, these are open:

• Restoring a snapshot into a new cluster. The procedure is written down, but the tests only check a restore into the same cluster. Until this has been run, the backup remains a belief rather than a fact.
• A complete version upgrade. The readiness checks exist; the upgrade has not happened.
• Encryption key rotation on the production node. Measured on the bench: data under the previous key stays readable and the share scheme is untouched. Not run on production.
• Audit log rotation on the production node. Nine checks, including rollback, but never rotated on production. A rotation after which audit silently stops writing is worse than no rotation.
• Delivering an alarm to a human. The notification channel has never fired for real: nobody read the honeypot, and the tests check parsing of the event, not that a notification actually appeared on screen.

Twelve items in this register have already been closed by measurement, each with a date and a method. I show this list not as self-criticism: a register of what is unverified is the way to tell a secrets store apart from faith in one.

What I deliberately do not do

• I do not keep shares or the root token in git, neither in plain text nor encrypted. A private repository is not the same thing as an unpublished one.
• I do not use auto-unseal. The convenience is paid for with a point of failure that cannot be recovered.
• I do not put the seal on a hardware module through an early-version plugin. A root of trust should not rest on version 0.1.
• I do not revoke the root token until the break-glass login has been tested with a real login. Otherwise you can end up with no way in at all.
• I do not allow the production container to be recreated without an explicit flag. Recreating it means sealing it, and it cannot happen by accident.
• I do not write denial checks as a list of what is forbidden. Only as a list of what is allowed.
• I do not issue the store’s own certificate from its own PKI engine. A sealed server cannot issue itself a certificate, which is the classic chicken and egg.
• I do not pass a test bench off as operation. Anything checked only on the bench is called exactly that.

The stack

• OpenBao: the secrets store, pinned to 2.6.1 by image digest; Raft integrated storage.
• Ansible: deployment, ceremony, unsealing, checks; a custom role, because there is no official collection.
• Docker: the store’s container.
• YubiKey: three of the five shares, encrypted to the tokens’ OpenPGP keys.
• YubiHSM 2: bought, turned down as the seal, assigned to the offline root of an internal certificate authority.
• Let’s Encrypt: the store’s own certificate through the DNS challenge, not from its own engine.
• Prometheus: metrics, liveness, and the honeypot watcher.

Why this matters if you are paying

People buy a secrets store to remove risk: passwords in files, keys in chat threads, access that only one person remembers. It does remove that risk, but not for free. Centralising secrets turns the risk into a quorum of living people. A store that cannot be unsealed without three of five is exactly as reliable as your list of holders, their PINs and their availability on a bad day.

Two practical conclusions follow. First: “the pipeline is green” and “the ceremony has been held” are different claims, and only the second one works in an outage. Do not ask whether the store is deployed; ask when it was last restored from a snapshot and how long that took. Second: a forced upgrade every few months is built into the design, and each one needs the holders present. That is not a project but a recurring operation, and it needs an owner and a calendar. It is the same principle I apply to access in general: what gets tested is behaviour, not configuration.

Frequently asked questions

Why OpenBao rather than HashiCorp Vault?

OpenBao is a fork of Vault under the MPL-2.0 licence within the OpenSSF at the Linux Foundation, while Vault has been distributed under the Business Source License since 2023. For a store that holds access to everything else, I do not want to depend on somebody else’s licence terms. The price: OpenBao has no official Ansible collection, so the deployment role had to be written from scratch.

Can a share be made mandatory in Shamir’s scheme?

No, all shares within the threshold are mathematically equal. The guarantee that nothing opens without a hardware key can only come from arithmetic: the number of shares that open without a token must be strictly less than the threshold. With five shares and a threshold of three, there can be at most two such shares.

Why not auto-unseal the store through a hardware module?

Because “cannot be opened without a hardware key” and “opens itself” are mutually exclusive, and losing the only module means irreversible loss of the store, snapshots included. On top of that, OpenBao 2.7.0 moved the built-in PKCS#11 seal into an early-version external plugin. Migrating to it later is possible without losing the current shares.

What is dangerous about YubiHSM guides for OpenBao?

If the key is created without the right to be exported under wrap, it can never be copied or restored onto another device, and that right cannot be added later. Such a key ties the store to one device forever. The key has to be created with the exportable-under-wrap flag and backed up before the store is initialised.

How do you check that the audit log is really being written?

Not by asking the store which audit devices are enabled: that answer was measured to be green even for a device that writes not a single byte. The only honest check is to send an identifiable request and find it in the log file.

What does upgrading a store like this cost?

A version upgrade is the only maintenance operation that seals the store, so the share holders have to be available before it begins. OpenBao has no long-term support line, so upgrades come regularly, every few months. It is an operation with an owner and a calendar, not a one-off project.

Why rehearse the holders if share encryption is already tested?

Being able to encrypt a share to a holder’s key does not mean the holder can open it: a token can go missing, a PIN can be forgotten, paper can be illegible. The initialisation ceremony is irreversible, so every holder decrypts a control share in advance through the same path the store will use.

Sources

• OpenBao releases, including 2.7.0 and 2.6.3 of 23 September 2026.
• Seal and unseal in OpenBao: Shamir, auto-unseal, recovery keys.
• The pull request with the YubiHSM guide, open as of 24 September 2026.
• The YubiHSM 2 user guide: object capabilities and export under wrap.
• HashiCorp: Vault’s move to the Business Source License and the company’s acquisition by IBM.
• Shamir’s secret sharing.

Need a consultation?

If you are choosing a secrets store, or already have one and want to know whether it would survive a bad day, book a conversation. We will go through the share layout, the list of holders, the audit checks, and when the store was last restored from a snapshot. Related reading: a Kubernetes cluster on your own hardware, a Proxmox cluster as code.