Secure Document Workflow Audit Trails with NATS KV and Object Store

The scenario

A government permit application must move through draft → under-review → approved/rejected, with two hard requirements: the actual PDF must be stored reliably, and every state transition must be auditable later — who changed it, when, and what it changed from/to — for a compliance review that might happen months afterward.

Step 1 — server + buckets

terminal 1

docker run -d --name nats-server -p 4222:4222 -p 8222:8222 -v nats-js-store:/data nats:latest -js -sd /data
nats context add local --server="nats://localhost:4222" --select

terminal 2 — document bytes

nats object add PERMIT_DOCS --storage file

terminal 2 — workflow state, with a generous history for audits

nats kv add PERMIT_STATUS --history=20

Step 2 — the applicant submits

terminal 3 — upload the document

echo "pretend this is a permit application PDF" > permit-8842.pdf
nats object put PERMIT_DOCS permit-8842.pdf

terminal 3 — record the initial workflow state

nats kv put PERMIT_STATUS permit-8842 '{"status":"draft","by":"applicant-jane","ts":"2026-07-29T09:00:00Z"}'

Step 3 — the workflow advances, each transition tracked

terminal 4 — a reviewer picks it up

nats kv put PERMIT_STATUS permit-8842 '{"status":"under-review","by":"reviewer-tom","ts":"2026-07-29T10:15:00Z"}'

terminal 4 — approved

nats kv put PERMIT_STATUS permit-8842 '{"status":"approved","by":"reviewer-tom","ts":"2026-07-29T14:30:00Z"}'

Step 4 — months later: a full, ordered audit trail

terminal 5 — the auditor pulls the complete history of every transition

nats kv history PERMIT_STATUS permit-8842

Because the bucket was created with --history=20, every one of these transitions — draft, under-review, approved — remains individually retrievable, in order, with no separate audit-logging system required; the KV bucket is the audit trail.

Step 5 — preventing two reviewers from silently overwriting each other

Just like the gaming leaderboard's compare-and-set pattern, two reviewers acting on the same application concurrently should not silently clobber one another — use kv update with the expected revision so a stale write is rejected rather than applied blindly:

terminal 4 — reviewer reads current revision, then updates conditionally

nats kv get PERMIT_STATUS permit-8842 --raw
nats kv update PERMIT_STATUS permit-8842 '{"status":"rejected","by":"reviewer-lee","ts":"2026-07-29T15:00:00Z"}' --revision=3

If another reviewer already advanced the key past revision 3, this update is rejected with wrong last sequence, forcing a re-read of the actual current state before deciding.

Troubleshooting

Symptom	Likely cause	Fix
Auditor can't find an old transition	Bucket's `--history` was too low and the transition was pruned	Set `--history` to comfortably exceed the number of transitions any single document realistically goes through, and never rely on it as the sole system of record for indefinite retention without a documented retention policy
Two reviewers' decisions conflict (one overwrote the other)	`kv put` (unconditional) used instead of `kv update --revision=N`	Always use compare-and-set for workflow transitions touched by more than one actor
Document bytes present but workflow state missing (or vice versa)	Object store and KV bucket writes aren't atomic with each other — a crash between the two steps can leave one without the other	Treat "document uploaded" as incomplete until both the object and its initial KV status exist; reconcile orphaned objects/status entries with a periodic consistency check
`nats kv history` shows transitions out of chronological order	Client-side clock skew in the `ts` field, not the KV revision order itself	Trust the KV revision/sequence number for ordering, treat the `ts` field as informational only

Recap

Concept	Takeaway
Object Store	Right fit for the actual document bytes
KV bucket with `--history`	The workflow state and its audit trail, in one place — no separate logging system
`kv update --revision=N`	Prevents two reviewers from silently overwriting each other's decision
Object + KV together	Large content and small, frequently-changing state each stored in the tool built for it

The scenario#

Step 1 — server + buckets#

Step 2 — the applicant submits#

Step 3 — the workflow advances, each transition tracked#

Step 4 — months later: a full, ordered audit trail#

Step 5 — preventing two reviewers from silently overwriting each other#

Troubleshooting#

Recap#