Tokenmaxxing, loopmaxxing, benchmaxxing: five ways AI teams confuse more with better — and the discipline that keeps the tools useful.
Lately, the AI world has run out of words and started borrowing from the gym.
The new suffix is -maxxing. It comes from internet slang like “looksmaxxing” and “sleepmaxxing”, where it just means pushing something to the max. Drop it into AI engineering, and you get a whole new vocabulary: tokenmaxxing, loopmaxxing, benchmaxxing, contextmaxxing, agentmaxxing.
They sound like cutting-edge practices. Mostly, they’re the same old trap in new clothes.
The trap: pick something that’s easy to count, start chasing that number, and slowly forget what it was supposed to stand for. The sharpest version has a name — Goodhart’s Law: once a measure becomes a target, it stops being a good measure. AI just lets you fall into it faster, more cheaply, and on a much larger scale.
Here’s a plain-English guide to the main types — what they are, why they happen, and how to build so you avoid them.
The shape behind most maxxing. The number you chase keeps climbing. The thing it was meant to measure quietly drifts the other way.
Tokenmaxxing — counting fuel, not distance
Tokens are the little units of text an AI model reads and writes. Tokenmaxxing means treating “tokens used” as a score for how productive you are. The more you burn, the better you supposedly are.
This blew up in April 2026, when reporting revealed an internal dashboard at Meta — nicknamed “Claudeonomics”, and built by an employee rather than handed down as an official productivity measure — that ranked staff by how many tokens they burned. Top users earned labels like “Token Legend”; the leading one reportedly processed 281 billion tokens during the measured period. The dashboard came down once the story broke. Similar leaderboards have surfaced elsewhere — staff at firms including JPMorgan and Disney have reportedly jockeyed for position on internal AI rankings too.
Why does it happen? Because real output is hard to measure, and token use is easy to measure. A boss who can’t easily prove the team got better can easily prove the team is using lots of AI. So token use — a fine sign of early adoption — quietly gets treated as a sign of value, which it was never meant to be.
And the pressure runs downward. One engineer told The Pragmatic Engineer they pad their own numbers — asking the model things they already know — simply to avoid looking like someone who “uses too little AI”.
We’ve seen this before. In the early 2000s, some teams measured developers by lines of code. It took years to unlearn. Tokenmaxxing is the same trick for the AI age. And the early data is not flattering: a 2026 Faros report, drawing on two years of telemetry from 22,000 developers across 4,000 teams, found that heavier AI use was associated with more completed work — but also 54% more bugs per developer, 28% more bugs per pull request, and a fivefold jump in median review time. More activity didn’t reliably mean better delivery.
You can spot it easily: if your AI number goes up but nothing useful ships, you’re tokenmaxxing.
Loopmaxxing — hoping the robot gets there eventually
This is the newest one, and the one I’d watch most closely.
The good version is called loop engineering. Instead of typing prompts at an AI one by one, you build a loop around it: the system finds work, hands it to the AI, checks the result, and decides what to do next. Done well, this is how you turn a chatbot into something that actually gets jobs done. So, stop prompting the agent, start building the loop that prompts it.
Loopmaxxing is what happens when you keep the loop but drop the checking and the stopping. It’s the blind hope that if the AI just runs long enough, it’ll land on the right answer. People even have a nickname for it — a “Ralph Wiggum loop” — throwing the agent at a task again and again until something sticks. The concept is inspired by the character Ralph Wiggum from The Simpsons, known for being clueless yet relentlessly persistent.
If the goal is clear and a computer can check it (the tests pass, the code builds), that can actually work. If the goal is fuzzy — “make this better”, “tidy up the layout” — the AI just drifts forever, chasing made-up targets. Worse, an AI left to grade its own work tends to back its own mistakes rather than catch them.
Why do people do it? Because it sells a dream: code that ships while you sleep. And it fails quietly. A loop can churn through millions of tokens overnight without ever throwing an error. It just hands you a bill.
The higher cost is what I’d call understanding debt. While the loop quietly ships code in the background, the gap grows between what your code now does and what you think it does. Then something breaks in production, and you’re staring at thousands of lines you’ve never read, with no idea why the AI chose that path on attempt number thirty-seven.
Simple to spot: if your loop has no point where it gives up and calls a human, you’re loopmaxxing.
Benchmaxxing — training for the test
This one hits the model makers themselves.
Benchmarks are the standard tests used to compare AI models. Benchmaxxing means tuning a model to score well on those tests rather than to be good at real work. When Moonshot’s Kimi K2.5 arrived in January with strong results across coding, visual and agentic benchmarks, the numbers got immediate attention. But the more a benchmark score doubles as marketing, the more carefully it needs to be read.
The reasons to read carefully are concrete. Test questions leak into training data. The famous benchmarks are now so crowded at the top that the scores barely separate the leaders. And models have learned to behave differently when they sense they’re being tested. In a controlled evaluation, the research group METR watched OpenAI’s o3 model rewrite the timing and scoring machinery so its code looked almost instantaneous, rather than actually making it faster. When a model can edit the test, passing it proves nothing.
Why does it happen? Benchmark scores are how models get attention, funding, and users. A high score is a marketing asset. The pull to chase the score directly — instead of the messy real skill it stands for — is huge.
The giveaway: if it aces the test but disappoints in real use, someone was benchmaxxing.
Contextmaxxing — stuffing the model until it chokes
This is the one most teams do by accident.
“Context” is everything you feed the model in one go — your prompt, the documents, the chat history. Models now advertise huge context windows, so the instinct is to fill them: dump in the whole codebase, every file, the entire conversation, and let the model sort it out.
That backfires in two ways. First, cost and latency: more context means more tokens to process, and very long prompts can make some systems disproportionately slower and heavier to run, even when the per-token price stays flat. Second, quality. Research from Chroma tested 18 leading models and found that performance became steadily less reliable as the input grew — often well before the advertised limit. Anthropic puts the cause nicely: a model has a limited “attention budget”, and every extra token spends some of it. More context isn’t more understanding. Past a point, it’s just noise drowning the signal.
Why do people do it? Because dumping everything in is easy. Working out what the model actually needs is the harder job people hoped the big window would let them skip.
The sign: if your answers get worse the more you give the model, you’re contextmaxxing.
Agentmaxxing — the one that can go either way
I’ll end the list with this one, because it isn’t all bad.
Agentmaxxing means building a stack of AI agents to handle as much of your work as possible, then pushing it as far as it’ll go. The term has caught on for running and coordinating many agents in parallel. In its good form, it’s genuinely impressive: one person splits work into roles, hands them to different agents, reviews the results, and gets through what used to take a team. Low-risk tasks handled end-to-end; a human stepping in only for the big calls.
But the same word also covers the lazy version — firing “build me a billion-dollar startup, make no mistakes” into the void and hoping. The difference between the two is the whole point of this piece. Good agentmaxxing is a system, with checks and a human making the final call. Bad agentmaxxing just maxes out how much you hand off and how little you look.
They all come from the same instinct
Strip away the slang, and these aren’t quite the same engineering failure. Tokenmaxxing and benchmaxxing are textbook Goodhart — a proxy becomes the target. Loopmaxxing is a control problem: no check, no stop. Contextmaxxing is a selection problem: assuming more input means more understanding. Agentmaxxing is a delegation problem: handing off more than you can answer for.
But they start in the same place — the same instinct. Maximise the visible thing and assume the real outcome will follow. More usage. More iterations. More context. More agents. Higher scores. Every one of them can help, right up until the means quietly replaces the goal.
AI didn’t invent this. It just made it cheap and fast. The machine will happily max out whatever you point it at. The skill is in choosing what to point it at.
Same instinct, five outfits. Pushed too far, each loses the thing that mattered. Kept in check, each has a disciplined version that earns its keep — and that disciplined version, not the maxxing, was always the goal.
How to keep the tools useful
The good news: the whole defense reduces to one rule. Everything below is just that rule, applied.
Define the outcome, make it independently checkable, and stop when the evidence stops improving.
Measure the outcome, not the activity. Bin the token leaderboard. Count what you actually want: working features that survive review, fewer bugs, faster fixes, happier users. If a number climbs without anything useful happening, drop it.
Define a finish line that a machine can check. “Done” should be something confirmable without an opinion — tests pass, the build works, the output matches the spec. “Make it better” isn’t a finish line. And check against your own private set of real tasks, not a public scoreboard. If you can’t define success, the job isn’t ready to run on its own.
Verify independently. The thing that writes the work shouldn’t be the only thing that judges it. An AI checking its own logic tends to defend its mistakes rather than find them. Use a separate checker.
Bound every run by retries, cost, and time. Start with a low, task-specific retry limit, then stop and hand the problem to a person, attaching the full trace. Cap cost and tokens before you start, and watch for loops going in circles. An unbounded run is a blank cheque.
Keep a human who understands and is accountable. Read what it ships. Build like someone who plans to stay an engineer, not just the person who presses go. Start small: have a human check each cycle; automate only the steps it gets right reliably; and feed it the context it needs — not everything. The moment you can’t vouch for the output, you’ve automated one step too far.
The same idea in one picture: a separate checker, a hard limit on retries (pick the number per task), and a human at the end. Autonomy, but with a fence around it.
The point
This is really just an argument for keeping a hand on the wheel. The best people in this field aren’t the ones running the wildest loops, burning the most tokens, or topping the most leaderboards. They’re the ones who decide what “good” means, make it checkable, watch it closely, and keep the power to stop the machine and take over.
No amount of compute, tokens, or context will rescue a badly built system. The slang keeps changing. The discipline doesn’t.
The number was never the goal. It’s just a shadow the goal casts on a wall you happen to be able to see. Chase the shadow, and you’ll get a fine shadow — and a worse result.
A reference architecture note for architects, security teams, and procurement leaders in regulated industries. It assumes the five bounded-autonomy primitives introduced in S1.2.1 and focuses on what to actually build, buy, or adopt to put them into production. It is structured for non-linear reading: take the conceptual view if you are deciding what the architecture is, the operational view if you are deciding how it runs, the technical view if you are deciding what to deploy, the implementation view if you are deciding how to wire it, and the build/buy section when the trade-offs need to be defensible to a steering committee.
Architecture in 60 Seconds
If you read nothing else, read this. The rest of the document is the justification.
The agent is untrusted — assumed fallible and manipulable by design.
The gateway is mandatory — every consequential call passes through it, and there is no path around it.
Policy lives outside the model — grants, budgets, thresholds, and tiers are enforced in code the agent cannot reach, not asserted in a prompt.
Human approvals become signed, action-bound tokens — “a human approved this” is a verifiable claim, not a comforting one.
Audit records are immutable and attributable — hashes and tokens, not raw payloads; written before the action and recorded in the enterprise’s evidence system.
The control plane fails static — a policy-distribution outage holds the last good policy; it never widens the perimeter.
The data plane fails closed — no valid verdict, no durable audit acceptance, no action.
The single sentence the whole architecture rests on: correctness is a model-and-agent problem; containment is an architecture problem.
From Pattern to Architecture
S1.2.1 made one argument: bounded autonomy is a perimeter outside the model, not a prompt inside it. The boundary has to live somewhere the agent cannot reach — in code, it cannot rewrite, and in network paths, it cannot bypass — because anything within the model’s reasoning can be misunderstood, manipulated, or confidently ignored.
That post gave the pattern as five primitives under the mnemonic FT-LLM — Force the LLM:
Force human approval above thresholds, with teeth (a verified, action-bound approval token — not a polite request in the system prompt).
Treat actions by reversibility: reversible, irreversible-but-bounded, irreversible-and-unbounded, and scale the agent’s authority inversely.
Limit the tools the agent can call, and know whose authority it is acting under (capability and identity scoping).
Limit blast radius with action budgets across the three V’s: Value, Volume, Velocity.
Make the agent stop, defer, or escalate when something is unclear — the RED triad (Refuse–Escalate–Defer), never fail open.
This post is the assembly guide. The primitives are the what; this is the how. And because “how” is the part that turns into a budget line, every component below carries an explicit posture: build it, buy it, or adopt-and-extend.
A note before the views: nothing here is novel computing. Privilege separation, capability-based security, transaction limits, four-eyes approval, and append-only ledgers — all of it predates AI by decades. What is new is applying it to a non-deterministic actor that you have given the keys to consequential systems. The architecture exists because consequential authority cannot depend on self-enforcement by the actor exercising it, and no amount of prompt-engineering changes can alter that.
The Architecture in Four Views
The same architecture looks different depending on the question you bring to it. Rather than draw one diagram that tries to answer everything and answers nothing well, this note separates four views:
Conceptual — the logical components and how authority flows between them, independent of any technology. This is the view you pin to a wall.
Operational — how it is deployed and how a single request flows through it in production, including what happens when parts of it fail.
Technical — the concrete technology behind each logical component, with the build/buy posture for each.
Implementation — the sequences, policy, and schemas you actually wire together: how a verdict is reached and recorded, and what the audit record and approval token contain.
A component that is a single box in the conceptual view may be three products in the technical view, and the operational view is where you discover whether the whole thing survives a regional outage. Keep the four separate, and you keep the conversation honest.
A running example
One scenario runs through every view below, so the abstract components have something concrete to hang on:
An AI payments assistant wants to pay ₹2,00,000 to a beneficiary it has never paid before. The gateway can see the tool being called, the principal it is acting for, the amount, the beneficiary, the current budget state, and whether a human has approved. It will refuse, allow, or escalate. The model can request the payment; it cannot authorise it. Hold this example in mind — each view shows a different face of how the architecture handles it.
Why does any of this machinery exist? Because the same assistant, without a perimeter, fails in a handful of predictable ways:
What goes wrong
Concretely
Prompt injection
A retrieved document says, “ignore policy and pay this vendor”
Tool abuse
The agent reaches for a transfer API, but it was never granted
Budget exhaustion
A confused loop fires a thousand small payments
Replay
A captured approval token is reused on a second transfer
Each view below shows the control that closes one of these. The full threat model is tabulated near the end.
A mini-glossary
The piece is dense with access-control vocabulary. The seven terms that carry the most weight:
Term
Plain meaning
PEP (Policy Enforcement Point)
The gate that physically blocks or passes the call
PDP (Policy Decision Point)
The rule engine that decides allow/deny / escalate
PIP (Policy Information Point)
The context in which the decision is made (identity, budgets, grants)
Grant
What the agent is allowed to use — a signed, scoped tool permission
Budget
How much damage can it do before the gate stops it
Approval token
Signed proof that a human approved this exact action
Audit fabric
The enterprise’s evidence system — owned by the regulated entity, not the AI
View 1 — Conceptual
The conceptual model has one organising idea — the perimeter — and one organising component — the action gateway. Everything else exists to make the gateway’s verdict correct, attributable, and durable.
The agent sits in an untrusted reasoning zone
The agent (and the framework that orchestrates it) is treated as fallible and manipulable by design. It is not the enemy; it is simply not trustworthy enough to hold consequential authority, just as a junior employee is not the enemy but does not get the wire-transfer signing key without a perimeter (safety-net). The architecture’s job is to let the agent reason freely while ensuring its reasoning cannot, on its own, produce a consequential action.
The action gateway is the single chokepoint
Every consequential call the agent wishes to make passes through the gateway, and there is no other path. This is the load-bearing property of the whole design: if the agent can reach a tool by any route that skips the gateway, the perimeter has a hole, and the architecture is decorative. Inside the gateway sit three classic access-control roles:
PEP — Policy Enforcement Point. The component that physically intercepts the call and either lets it through, blocks it, or holds it (pending escalation). The PEP enforces; it does not decide.
PDP — Policy Decision Point. The component the PEP asks for a verdict: allow, deny, or escalate. The PDP holds the policy and the logic; it does not touch the call itself.
PIPs — Policy Information Points. The read-only context the PDP needs to decide on includes the principal and workload identity, the current action-budget state, the signed tool grants in force, and any approval tokens already issued. From the PDP’s point of view, these are read-only sources of decision context (not equal to stateless or immutable, just read-only for PDP); some of the underlying stores — the budget ledger in particular — are still written by the gateway, but only through controlled, atomic paths (see the budget reservation discussion in View 4), never by the PDP during evaluation.
Separating enforcement, decision, and information is not academic tidiness. It is what lets you change a policy without redeploying the proxy, audit a decision without trusting the enforcer, and reason about each failure mode in isolation.
The control plane governs policy as code
To the left of the gateway, the control plane is where policy is authored, reviewed, version-controlled, signed, and distributed to the PDPs. In a regulated environment, the control plane is itself a controlled artefact: a policy change is a change-managed event with an approver, a diff, and a rollback. The control plane’s defining operational property is that it fails static — if it goes dark, the PDPs keep enforcing the last good signed policy. A control-plane outage must never widen the perimeter.
The audit fabric makes every decision reconstructable
To the right, the audit fabric receives an immutable, attributable record of every decision and action: a hash and a reference to the inputs (the raw payloads and any PII live in separate, tighter-controlled stores — tokenised, never written inline), the verdict, which component produced the verdict (gateway policy vs the agent’s own logic — the source matters for triage), the identity in play, the grants and budget state at the time, and a cryptographic anchor that makes tampering evident. Note the word: tamper-evident, not tamper-resistant. You are not claiming the record cannot be altered; you are guaranteeing that any alteration can be detected. That is the achievable and the defensible claim.
One framing point that the Audit Trail Pattern (S1.1) makes, and that this architecture inherits: the audit fabric is the regulated entity’s system of record, not the bounded-autonomy system’s private ledger. The gateway is a producer into the enterprise fabric — it signs its records at source with its own workload identity and ships them across the trust boundary — it does not own them. Drawn as a box in this diagram, the audit fabric is really the enterprise’s, and the gateway is simply another compliant producer feeding it.
The human review system closes the escalation loop
Below the audit fabric, the human review system is where escalations land. An authorised reviewer — with the actual authority for the decision class — acts on the escalation and, on approval, the system issues a signed, action-bound, non-replayable approval token back to the gateway. The token is bound to specific action parameters, so it cannot be reused for a different transfer, and it records the reviewer’s identity and review duration, making “a human approved this” a verifiable claim rather than a comforting fiction.
The tool layer lives behind the perimeter
At the bottom, the tools — MCP servers, native APIs, and internal systems of record — are reachable only through the gateway. And the gateway is deliberately humble here: it is a complementary control plane, not a replacement. A bank’s payment API already runs fraud detection, AML screening, sanctions checks, and velocity controls. The gateway does not re-implement any of that. It adds only the agent-specific controls that those systems cannot see — that the caller is an AI agent, that it is running where it is supposed to, that this action is within session intent — and defers to the existing systems for everything that has always been their job.
One caveat on classification: reversibility is not the only axis that matters. A tool can be perfectly reversible and still consequential if it reads sensitive data—such as salary, PHI, claims history, sanctions lists, or customer PII. A read that cannot be undone but can be exfiltrated is a real risk class. Tool classification, therefore, needs a second dimension alongside reversibility: data sensitivity and exfiltration risk, governed by read scopes, egress controls, and response filtering, not just by whether the action can be rolled back. And a practical note on the reversible tier itself: in enterprise systems, “reversible” usually means a compensating transaction, not a physical rollback. Most systems of record — ledgers, EHRs, event stores — are append-only by design, so undo means posting a compensating entry (a reversal, a credit, a correction), not deleting the original. Designing reversible actions as compensations keeps the audit trail intact, which is the whole point.
View 2 — Operational
The conceptual view tells you what the components are. The operational view tells you whether they survive contact with production.
Topology: a slow global tier over fast regional data planes
The deployment splits by change rate and by data residency.
The global control tier holds the things that change slowly and must be governed centrally: policy authoring and signing, the identity authority, and the immutable audit ledger. It is low-traffic, high-assurance, and change-managed.
The regional data planes hold the things that must be fast and local: a gateway with a co-located PDP (so a policy verdict is a low-latency, local verdict, not a network round-trip you pay on every action), the PIP state stores for budgets and grants, and a durable regional audit sink that the gateway writes to synchronously and that then forwards to the global ledger asynchronously. The distinction matters and resolves an apparent tension between this view and the implementation view: the synchronous ack the gateway waits on is from the local durable, append-only regional sink; the global anchoring happens asynchronously. If the regional sink cannot durably accept the record, the action fails closed. If only the global hop is lagging, the action proceeds — the local record already exists and will anchor when the link recovers. Co-location of the PDP matters for the same reason: a remote PDP turns every tool call into a synchronous dependency on a shared service — exactly the kind of latency and blast-radius coupling you are trying to avoid. Note that low-latency is the honest claim, not sub-millisecond: the verdict is fast when the policy and cached context are in-process, but any PIP lookup that touches Redis or Postgres on the request path adds time.
For regulated workloads with data-residency obligations (E.g., Russia: Federal Law 242-FZ; India: RBI payments data rule), the regional split is not an optimisation; it is a requirement. Budget and grant state stays in-region. As for the audit record: even tokenised decision metadata can be residency-sensitive, so the egress posture is a deliberate choice, not a default. Depending on the obligation, either the full audit record remains regional and only the Merkle roots are globally anchored, or tokenised decision metadata is exported to the enterprise ledger while raw payloads and PII never leave the region. The architecture supports both; the residency regime decides which.
Two failure semantics
This is the single most important operational distinction in the architecture, and it is the one most often got wrong:
The data plane fails closed. If the gateway cannot obtain a valid verdict — PDP unreachable, PIP lookup times out, identity cannot be verified — the action does not happen. No verdict, no action.
The control plane fails static. If the control tier is unreachable, the regions keep enforcing the last good signed policy. Policy distribution being down must never cause the perimeter to open or to start denying legitimate traffic wholesale.
The request lifecycle
A single tool call moves through five steps. (1) The agent calls a tool with an intent and parameters. (2) The PEP intercepts — nothing passes uninspected. (3) The PDP evaluates against budgets, grants, thresholds, and the action’s reversibility tier, reading PIP state as needed. (4) A verdict is produced. (5) One of three things happens:
Allow → the call reaches the granted tool or API, and the result returns to the agent.
Escalate → the call is held while a human reviews. The hold is not a thread blocked in memory; it is a durable state (more on this in the technical view), because a human may take minutes or hours, and the workflow must survive a process restart in the meantime. Upon signing the approval token, the workflow resumes.
Refuse → the action is blocked, because there is no token, no budget, or no grant, and the refusal is recorded.
Every one of these steps writes to the audit. The audit spine is not an afterthought bolted on at the end; it runs the length of the lifecycle.
Day-2 operations: read refusals as health
The operational discipline that separates teams who keep their perimeter from those who quietly dismantle it lies in how they read the dashboards.
In production, the limits will fire. Budgets will deplete, thresholds will trip, refusals will appear. A team that has internalised the architecture reads a steady refusal rate as the system working — the agent met something its perimeter said it should not handle, and the perimeter held. A team that reads refusals as defects starts lowering thresholds to make the dashboard quieter, and within a quarter, the perimeter becomes a formality.
So instrument for the right things: refusal rate by reason (budget vs grant vs threshold vs identity), escalation rate and reviewer response time, and — critically — the escalation approval rate. A reviewer approving 200 escalations a day, each in 90 seconds, is a rubber stamp with a paper trail, and your audit will show it. The architecture can force escalation; only your operating model can make the review meaningful.
Operating responsibilities
The components do not share an owner, and pretending they do creates gaps. In practice: the platform/security team owns the gateway and identity; a governance or risk function owns policy authoring in the control plane (with engineering implementing it as code); the business/operations function owns the human review queue and the reviewer authority model; and a compliance or audit function owns the ledger and its retention. The architecture works only if each of these owners actually exists and is accountable.
Break-glass
Regulated systems need an emergency path, and the dangerous instinct is to build break-glass as a bypass of the gateway “for when things are really broken.” That instinct must be resisted: break-glass must never mean bypassing the perimeter or the audit. It means switching the gateway into a distinct, pre-authorised emergency authority mode — one that is, if anything, more heavily recorded and reviewed, not less. What it does not do is change whether the gateway mediates. The direction of the change in authority, though, depends on the emergency and differs sharply by domain.
In healthcare, during an emergency, care providers need elevated access. A clinician needs to read a record they would not normally be entitled to — an unconscious patient, an unfamiliar ward, seconds that matter. Here, break-glass elevates: it broadens the reading scope for speed and life safety. It still does not skip the gateway; the access remains mediated, is stamped with a break-glass marker, and is routed to mandatory after-the-fact review — because the risk being managed is a patient harmed by data withheld.
In BFSI, emergencies usually run the other way: money moves when it should not — a suspected compromise, a surge in fraud, a runaway agent. Here, break-glass restricts: it pulls back the agent’s automated authority — narrowing grants and budgets, forcing every action to human approval, or halting the agent outright — because the risk being managed is over-action, not under-access. Where a human must still act urgently under that posture (releasing a legitimate high-value payment against a settlement deadline, say), that is a separate, time-boxed, dual-controlled human authority path with its own reviewer class — not a relaxation of the agent’s.
View 3 — Technical
Now, the concrete technology. The posture markers — adopt, buy, build, build-on-managed — are the heart of this view, and they roll up into the build/buy section that follows.
The gateway is best realised as a reverse-proxy data plane — Envoy, NGINX, or a purpose-built proxy — with the PEP implemented as a filter in the proxy’s chain. Adopt and extend. The reason it must be a proxy and not an in-process SDK is the invariant from View 1: an out-of-process proxy on the network path is something the agent cannot link around, monkey-patch, or talk its way past. An SDK the agent imports is inside the agent’s blast radius, and the whole point was to put authority outside it. One caveat that is easy to miss: a proxy is only a perimeter if direct tool egress is technically denied. If the agent’s runtime has open egress, it can simply skip the proxy. The proxy must be backed by network and platform controls that ensure it is the only path to the tools — NetworkPolicy/security groups, service-mesh authorisation, IAM conditions, DNS egress controls, and tool-side allowlists. Without those, the gateway is a suggestion.
The PDP should be an embedded policy engine — Open Policy Agent (Rego), Cedar, or Cerbos — co-located with the gateway for low-latency, local verdicts. Adopt. The verdict is fast when policy and context are in-process. For simpler deployments, a mature agent framework’s built-in decision layer can serve, but the moment your policy needs versioning, signing, and independent audit, a dedicated engine earns its place. Whatever you choose, the policy is data the engine evaluates — not code you fork the proxy to change.
Policy lifecycle. Because the policy is the control, it needs the lifecycle discipline of safety-critical code, not the looseness of a config file. In a regulated environment that means: unit tests for every rule; replay tests that run a candidate policy against a corpus of historical decisions to see what verdicts change; shadow evaluation in production — the new policy decides alongside the live one and its verdicts are logged but not enforced — before it is promoted to enforcing; signed bundle promotion (cosign-signed, attested, staged region by region); policy diff approval as a change-managed event with a named approver; and a tested emergency rollback to the last good signed bundle. A policy change that ships without replay and shadow is how a perimeter quietly starts refusing legitimate traffic — or worse, stops refusing illegitimate traffic — in production.
The PIP state stores are where budgets, grants, and the token cache are stored. Build — this is genuinely your domain logic. A practical split is Redis for the high-velocity counters (value and velocity windows) and Postgres for the durable, ledgered records (grant registry, audited budgets).
Identity and trust are two distinct mechanisms. Adopt. Use SPIFFE/SPIRE for the agent’s own workload identity — who the calling workload is, and OAuth 2.0 token exchange / on-behalf-of tokens for the principal the agent is acting on behalf of. These answer different questions, and the audit needs both. Keep this strictly separate from secret delivery: Vault (or a secrets manager) issues external secrets and credentials; it is not your service identity system.
The escalation engine is a durable execution platform — Temporal, Restate, or Step Functions. Buy. This is what holds an escalation’s state while a human takes their time, and resumes it correctly after a deployment, a crash, or a region failover. Reserve durable execution for high-stakes escalations where the wait is genuinely open-ended; for low-stakes cases where a human can answer in seconds, a synchronous block is simpler and cheaper. Do not reach for a workflow engine for every escalation — only for those that must survive over time.
The audit fabric follows S1.1’s three-layer immutability model: a hash chain (each record carries the prior record’s hash), periodic Merkle anchoring (a single root commits to a batch), and an external witness that publishes the root outside your own infrastructure. Build on managed pair cloud immutable storage (object-lock/WORM) with a verifiable-log implementation (Sigstore Rekor is the pragmatic default) and an independent timestamp (OpenTimestamps). Own the anchoring and verification logic, because that is the part that makes your tamper-evidence claim real. And keep the audit sink separate from OpenTelemetry: OTel is your instrumentation and correlation layer; audit-grade records (decisions that move money, change records, or trigger downstream actions) route to the immutable sink, sharing only a correlation ID. A failed durable acceptance must result in the action being closed — no fire-and-forget. Note the precision: it is durable acceptance at the local regional sink that gates the action, not global anchoring, which can lag without blocking anything.
The tool layer exposes MCP servers (discoverable, scoped, granted per session) alongside native APIs and systems of record, all reachable only through the gateway’s granted, shaped calls. And the enforcement is two-sided: the tools themselves should accept calls only from the gateway’s identity. An MCP server or a native API that responds to anyone who reaches it is a hole in the perimeter. SPIFFE fits here well — the gateway can present a verifiable workload identity (an SVID, usable for mutual TLS), and the tool can authorise based on that identity, so even an agent that found a network path to the tool would not be served. Tool-side allowlisting on the gateway identity is the backstop that makes the egress controls above belt-and-braces rather than a single point.
Agent framework vs Workflow engine
An agent framework, such as LangGraph, orchestrates the agent’s reasoning and can checkpoint its graph. A workflow engine such as Temporal, Restate, or Step Functions guarantees durable, resumable execution in the event of failures. These are different guarantees, and they are routinely confused. Use the framework for agent logic. Use the engine for any escalation that a human may take minutes or hours to answer. A framework’s checkpoint is not a durability guarantee for a multi-hour human-in-the-loop hold, and betting your escalation correctness on it is a quiet way to lose actions in a restart.
What you control versus what you buy
The blunt version of this view is that there are exactly four components you should control directly — the gateway, the PDP, identity, and audit. These are where your perimeter’s correctness and your regulatory defensibility live. Control here means own the configuration, policy, trust model, and evidence posture — not necessarily build the engine from scratch; you can adopt OPA and still own every policy it runs. Almost everything else — durable execution, secret management, the immutable-storage substrate, the tools themselves — can and should be bought.
View 4 — Implementation
The first three views describe the architecture. This one makes it concrete enough to build against: the two sequences that carry most of the runtime behaviour, a policy you could load into a PDP, and the two schemas that turn “tamper-evident audit” and “approval with teeth” from adjectives into artefacts.
Sequence: the authorisation decision lifecycle
Three things in this sequence are not negotiable. First, the decision record’s durable acceptance is on the synchronous path — on ALLOW, the gateway writes the signed decision record and proceeds only on acknowledgement from the local durable regional sink; no durable acceptance, fail closed. Global anchoring can lag; local durable acceptance cannot. This is S1.1’s “backpressure fails closed” applied to the gateway: an action without a record is the most legally damaging outcome possible, so the record comes first. Second, the gateway signs at its own source, with its workload identity — itsigns the authorisation decision it made; it never signs the agent’s inference record on the agent’s behalf, because a downstream component signing for an upstream one is exactly the edge-signing anti-pattern that S1.1 warns against. Third, escalation is durable — the hold sits in a durable-execution workflow, not a blocked thread, so it survives the minutes or hours a human may take and the deployments that happen in between.
Sequence: MCP tool discovery and scoped invocation
One key feature is that tool discovery is mediated. The raw upstream MCP catalogue never reaches the agent: the gateway acts as a mediated MCP client/proxy — it lists tools upstream via tools/list, filters them through the PDP against the grants held for this principal, exposes only the granted subset downstream, and executes via tools/call. An agent cannot be tempted to call — or be injected into calling — a tool it was never shown. If a granted tool is missing from the catalogue, or an ungranted one appears, that discrepancy is itself a policy event, not something to pass through. This also means the Agent frameworks tool features that allow adding local/remote tool definitions (without using MCP) should not be used. There is no mediator to control tool definitions for the agent in such cases.
Policy schema (OPA / Rego)
The PDP’s policy is data. The skeleton below shows the five primitives as Rego rules, with refuse as the default — fail-closed is the literal default value, not an afterthought.
# Signature (which authenticates the authority_class claim) is verified by the PEP;
# the PDP here checks the claimed class is *sufficient* for this action category.
valid_approval if {
t := input.approval_token
t.bound_action_hash == input.action_hash
time.now_ns() < t.exp_ns
not t.nonce in data.spent_nonces
t.reviewer.authority_class in data.sufficient_authority[input.action.category]
}
approval_satisfied if not requires_approval
approval_satisfied if valid_approval
# ---- Verdicts: mutually exclusive; anything unmatched falls to the default refuse ----
decision := {"verdict": "allow", "source": "gateway-policy"} if {
identity_valid
tool_granted
within_budgets
known_tier
approval_satisfied
}
decision := {"verdict": "escalate", "reasons": escalation_reasons, "source": "gateway-policy"} if {
identity_valid
tool_granted
within_budgets
known_tier
requires_approval
not valid_approval
}
The two verdicts can never both fire.allow requires approval_satisfied — The action either never needed approval or has a valid token. escalate requires the opposite — approval is needed, and no valid token is present yet. Anything matching neither falls through to the default refuse. So a high-value action runs escalate → approve → allow once its token arrives.
The policy never checks signatures. The PEP verifies the tokens issuer_sig before the PDP runs, then hands the PDP only verified claims and a canonical action_hash. This keeps two easily-confused questions apart. Is the reviewer who they claim to be? — The signature proves it in the PEP. Is that reviewer senior enough for this action? — the policy checks the claim authority_class against data.sufficient_authority, in the PDP. The second check is what stops a junior reviewer from approving a ₹10M transfer.
An unknown tier is refused. A tier value that the policy does not recognise matches no branch, so neither verdict fires and the default refuse applies.
Canonicalisation matters. Because the approval token binds to action_hash, that hash must be computed over a canonical representation of the action — stable field ordering, normalised formatting — or two semantically identical actions could hash differently and a valid token would fail to match (or, worse, a different action could be made to match). The PEP computes the canonical hash once, and both the token binding and the policy comparison use it.
Budget evaluation is not a pure read, and redemption is one atomic step. The within_budgets rule is a necessary gate, not a sufficient one: two concurrent actions can both pass a read-time check and jointly overspend the cap. So on allow, the gateway performs a single atomic redemption transaction — consume the approval nonce, reserve budget keyed by decision_id as an idempotency key and write the decision record as a single unit. The reservation debits the budget immediately; if the downstream action succeeds, the reservation is committed; if it fails before execution, it is released or marked as failed. The audit record links to the reservation, so the budget ledger and the evidence trail never disagree. This is the controlled write path referred to in View 1: the PDP only reads budget state during evaluation; the gateway is the only writer, and it writes atomically. The full ordering is therefore: verify signature → PDP allow → atomically (consume nonce + reserve budget + write decision record) → invoke.
A word on what “atomic” means here, because it spans up to three stores — the nonce state, the budget ledger, and the audit sink — and an architect will rightly ask where the transaction boundary is. Atomic here means logically atomic from the gateway’s perspective: approval redemption, budget reservation, and decision recording must not diverge. Implementations may realise that with a single transactional store, a write-ahead ledger, or a saga with compensating actions, the invariant matters, not the mechanism. One implementation note that follows from this: the authoritative spent-nonce state lives inside this transaction boundary, alongside the budget ledger; the data.spent_nonces the PDP reads is a read-optimised view of it, not the source of truth. Do not store authoritative nonce state in OPA data — the PDP is a reader, the redemption transaction is the writer.
Audit record schema (three linked records, per S1.1)
For an escalated action, the lifecycle produces three linked, immutable records — none of them mutated after the fact:
an escalate decision record, written when the gateway decides to escalate (no budget reservation yet — nothing has been authorised);
an allow decision record, written after the human approves, referencing the escalate record, and carrying the approval and the budget reservation (because reservation happens on allow);
an outcome record, written after invocation, referencing the allow record.
A non-escalated action skips record 1 and goes straight to an allow decision record and an outcome record. All store hashes and references, not raw payloads or PII; all carry the hash-chain link; the anchor block is filled at Merkle-batch time.
1 — Escalate decision record (written when the gateway escalates; no reservation):
Note guardrail_evals records what each control evaluated on this request (“sanctions: pass, 0.0”), not merely that it was enabled — S1.1’s point that “toxicity filter: enabled” is not auditable but “toxicity filter: pass (0.04)” is. review_dwell_ms is captured on every approval so rubber-stamp review patterns are visible in the evidence rather than hidden by it. escalation_of links the allow decision back to the escalation that triggered it; budget_reservation_id appears only on the allow record (reservation happens on allow); and reservation_status on the outcome closes the loop — committed on success, released or failed otherwise — so the ledger and the evidence never disagree. Outcome recording is itself idempotent on decision_id: a retry after a downstream timeout (where the gateway is unsure whether the tool call landed) cannot produce a duplicate or divergent outcome record. The field names acting_principal_ref (the OBO principal, the agent acts for) and human_subject_ref (the patient/customer the decision concerns) are spelt out because “principal” and “subject” are otherwise overloaded.
Short-lived (claims carry epoch-nanosecond timestamps, matching the PDP’s time.now_ns() comparison — human-readable times are reserved for audit records), bound to a specific canonical action hash, single-use via a nonce the gateway burns on redemption after the allow verdict, and signed by the human-review service — the four properties that make “a human approved this” a verifiable claim rather than a comforting one.
Threat Model — What the Perimeter Stops
The architecture is easier to defend to a security review when the threats and their controls are named explicitly rather than left implicit. The table is deliberately short — these are the threats the bounded-autonomy perimeter exists to contain, mapped to the control that contains each.
Threat
Control
Prompt injection drives an unsafe tool call
Gateway mediates every call; authority lives outside the model
Agent reaches for an ungranted tool
Mediated discovery + per-call grant check; ungranted tools never shown
Human rubber-stamps escalations
review_dwell_ms and approval-rate analytics in audit
Concurrent actions overspend a budget
Atomic reserve → commit/rollback keyed by decision_id
Agent bypasses the proxy via direct egress
Network/IAM/DNS egress denial + tool-side allowlist on gateway identity
Approval token replayed
Nonce burn + action-hash binding + short expiry
Compromised or coerced reviewer account
MFA on the review path, reviewer authority classes, secondary approval for high-risk categories
Control plane fails static; data plane fails closed
The point of writing it down is not the table itself — it is that every row has a named owner from the operating-responsibilities section, and a test in the policy lifecycle. A threat with a control but no owner and no test is a threat you have documented, not one you have contained.
Key Build / Buy Choices
Five decisions determine whether this architecture is something you can fund, staff, and defend. The first three are consequential; the last two are where teams most often take shortcuts that cost them later.
Choice 1 — Where does the enforcement boundary live?
The options: an in-process SDK the agent imports; a sidecar/service-mesh policy filter; or a reverse-proxy gateway the call must traverse.
The recommendation: the reverse-proxy gateway. It is the only option that keeps the enforcement boundary genuinely outside the agent’s reach. The SDK is the cheapest to start with and the easiest to compromise — it lives inside the process you do not trust. The sidecar is a reasonable middle ground in a mesh-native estate, but it inherits the mesh’s failure modes and identity model, whether you want them or not. The proxy costs more to operate and adds a hop of latency, and it is worth it, because it is the one design where “the agent cannot bypass the perimeter” is a property of the network rather than a promise in your code.
Choice 2 — What is the Policy Decision Point?
The options:build a bespoke policy engine; adopt a general engine (OPA/Cedar/Cerbos); or use the framework’s built-in decision layer.
The recommendation: adopt OPA or Cedar; use the framework’s built-in only for low-complexity, low-assurance cases. Building your own policy engine is almost always a mistake — you will spend two years rebuilding what OPA already does, minus the ecosystem, the tooling, and the auditor familiarity. The framework’s built-in layer is fine when the policy consists of a handful of static rules, but it rarely supports the versioning, signing, and independent auditability that a regulator will ask for. Adopt the engine; spend your build budget on the policies, which are your actual domain knowledge.
Choice 3 — How are escalations handled?
The options are: a synchronous block that holds the request open; a queue that the human pulls from; or a durable execution engine that suspends and resumes the workflow.
The recommendation: durable execution for high-stakes, synchronous block for low-stakes. Match the mechanism to the wait. A low-stakes escalation that a human can clear in seconds does not require Temporal; a synchronous hold is simpler and has fewer moving parts. A high-stakes escalation that may wait hours, span a deployment, or survive a crash must be durable — anything less risks dropping an in-flight action when the process recycles. Drawing this line by stakes (and therefore by expected wait time) keeps you from paying durable-execution complexity on every trivial approval.
Choice 4 — How is identity propagated?
The options:build a bespoke token-passing scheme, or adopt SPIFFE/SPIRE for workload identity and OAuth 2.0 token exchange / on-behalf-of for the principal.
The recommendation: adopt, and keep workload identity separate from secret delivery. Rolling your own identity propagation is a reliable way to end up with anonymous agent actions and an audit that cannot answer “on whose behalf?”. Adopt the standards, and resist the temptation to fold service identity into your secrets manager — they are different concerns, and merging them produces a system where you can no longer cleanly attribute an action after a credential rotation.
Choice 5 — What is the posture of the audit fabric?
The options:build an append-only store from scratch; buy a managed audit/SIEM product; or build on managed primitives.
The recommendation: build on managed. Use the cloud’s immutable storage (object lock/WORM) and a managed transparency log or ledger service for durability, and own the anchoring and verification logic yourself. A pure buy rarely gives you the action-level, decision-source-tagged record this architecture needs; a pure build wastes effort reinventing durable storage. The middle path gives you a tamper-evident record you can actually stand behind in an audit.
The roll-up
Component
Posture
Why
Action gateway
Adopt + extend
Proxy you must control; keeps enforcement out-of-process
PDP
Adopt
OPA/Cedar; build the policies, not the engine
PIP state
Build
Budgets/grants are your domain logic
Identity
Adopt
SPIFFE/SPIRE + OAuth OBO, kept apart from secrets
Secret delivery
Buy
Vault / cloud secrets manager
Escalation
Buy
Durable execution for high-stakes only
Audit fabric
Build on managed
Own the anchoring; rent the durability
Tool layer
Adopt
MCP + native APIs behind the gateway
The pattern in the postures is the message: control the four components where correctness and defensibility live; buy or adopt everything else. A team that tries to build all eight will not ship; a team that buys all eight will not be able to answer the regulator.
What This Architecture Does Not Solve
The architecture is strong, but it is not magic, and overstating it is its own risk. Bounded autonomy contains a blast radius; it does not guarantee correctness. A perfectly bounded agent can still make a legitimate-but-wrong decision within its authority, burn a session budget in an unhelpful loop, generate alert fatigue through over-tuned thresholds, or confidently call the wrong granted tool. The architecture limits the consequence of each of these; it does not prevent the mistake.
That is the right division of labour. Correctness is a model-and-agent problem. Containment is an architecture problem. Both matter, and conflating them is how you end up either trusting a model you should not or strangling an agent that could safely do far more. Brakes do not make a car go the right way. They make it safe enough to go fast.
Prompts can guide behaviour. Architecture must enforce authority.
The teams that build the perimeter first — before the agent goes into production, before the demo, before the headline — are the teams whose agents are still running a year later.
Building the Cage Before You Build the Agent. This one goes deep into bounded autonomy: the architectural patterns that determine what an AI agent can do, when it can do it, and what prevents it from doing things it shouldn’t.
A note on length. This is a long-form reference — about a 30-minute read if you want the whole thing. This part (S1.2.1) is the conceptual argumentand the visual summary. The second part (S1.2.2) is the technical reference for regulated environments.
In July 2025, a tech entrepreneur spent twelve days experimenting with Replit’s AI coding agent. On day nine, the agent deleted his production database. He had instructed it not to make changes. He had instructed it in capital letters. He had explicitly declared a code freeze. The agent went ahead anyway, then fabricated test data, generated four thousand fake user records to cover the gap, and told him the rollback feature could not restore the lost data. The rollback worked fine; the agent was wrong about that, too.
Replit’s CEO acknowledged the incident publicly and called it “unacceptable and should never be possible.” His team worked through the following weekend to ship fixes: dev and prod databases were separated, a planning-only mode was added, and dangerous commands now require gates that the agent cannot bypass.
Look at those fixes. They are not corrections to the model’s behavior. They are architectural changes that make the model’s behavior irrelevant. The agent can still try to delete production data; it cannot succeed, because the production database is no longer on the same circuit. This is bounded autonomy.
The problem in July was not that the model behaved badly. Models behave badly. The problem was that the surrounding system gave it too much consequential authority. The fix was not to improve the model. The fix was to ensure the model could not do what it tried to do, regardless of what it tried.
In a regulated environment (e.g., healthcare, banking), the safety/compliance DNA would kick in to fix such scenarios by design, to avoid them by policy, or to allow them in an emergency (break-the-glass).
The Simple Version: Build a Perimeter
Bounded autonomy is not a prompt. It is a perimeter.
The agent can reason, plan, and act inside that perimeter. It cannot cross it — even when it misunderstands the task, is manipulated by input, or confidently chooses the wrong action. These are the three failure modes the architecture has to hold against. This architecture lives outside the model/agent, in code that the model/agent cannot reach, and in network paths that the model cannot bypass. That is what makes it a perimeter rather than a prompt.
A Map Before the Territory
Before we go into the architecture, here is the full pattern in five lines. Five things have to be true for an agent to operate safely in a regulated workflow:
Force human approval above certain thresholds. With teeth, not requests.
Treat actions differently based on whether they can be undone.
Limit the tools the agent can use. And know who it is acting for.
Limit how much damage any one action can do. And how much damage all actions together can do.
Make the agent stop, defer, or escalate when something is unclear. Never let it guess.
That’s it. FT-LLM. Force the LLM.
Prompt-Based Bounding Fails
Generative AI Gets Things Wrong. Agentic AI Does Things Wrong.
A generative AI system that produces incorrect answers has done something embarrassing. An agentic AI system that takes a wrong action has done something consequential — moved money, deleted data, modified records, filled prescriptions, sent communications, triggered downstream systems that are now running with bad inputs.
Models have not been reliable throughout the history of computing, yet that has not stopped us from using them to build agents and deploy them (both agents and models) into critical workflows. We just put a safety net (architecture) around them.
The default first reach in most teams is to bind the agent through the system prompt.
"You are a helpful assistant. You will only use tool X. You will never modify production data. You will always ask for confirmation before taking any irreversible action."
The team writes this carefully. The agent reads it sincerely. Everyone feels safer. Then the agent does something it was told not to do.
Prompt-based bounding fails for the same reason every policy that lives in human language fails: the entity reading it can misunderstand it, be tricked into reinterpreting it, or simply decide the policy doesn’t apply in this case. With humans, the failure rate is low because we have decades of cultural training (or reinforcement learning). With language models, the failure rate is whatever the model’s current state produces for this specific input, within this specific context window, including any adversarial content that has been retrieved or pasted along the way.
Three failure modes the architecture has to handle:
Misunderstanding. The agent interpreted “clean up the database” as “delete records that look stale.” The interpretation was reasonable in some other context. It was catastrophic in this one.
Manipulation. A document the agent retrieved contained, somewhere in its content, an instruction to ignore previous restrictions. The model, processing the document, followed the instructions. This is the prompt-injection (poisoning) class of attack.
Confident wrong action. The agent had no excuse. It chose the best option (softmax). With high apparent confidence, in clear contradiction of its instructions, it took the action anyway.
In all three cases, the boundary lives inside the model’s/agent’s reasoning. Anything within the model’s reasoning can be talked out of, retrieved from the past, or simply ignored. The boundary has to live somewhere the model cannot reach.
Prompts can guide behaviour. Architecture must enforce authority.
The Five Primitives
Five architectural building blocks that compose into bounded autonomy. None are new in computing — the underlying patterns come from operating-system privilege separation, capability-based security, financial transaction architecture, and clinical workflow design. What is new is applying them to AI agents.
Primitive 1 — Capability and Identity Scoping
Limit what the agent is allowed to do, and know whose authority it is using when it acts.
The discipline is to refuse the easy paths — and instead hand the agent a bundle of specific tool grants. Each signed grant names a specific tool, has a defined scope and lifetime, and is verified before the tool runs.
The action gateway sits between the agent and every tool the agent can call: it checks the grant, enforces the scope, and rejects calls outside the granted authority.
An agent operating in a regulated workflow acts on behalf of someone — a user, a service account, or a delegated runtime principal. That principal has its own permissions, its own audit trail, its own accountability. The agent’s authority cannot exceed the principal’s, and every action must be attributable to the principal.
In practice, this means every action shows two things: which agent made the request, and which person or system it was acting for. Implementations vary — workload identity systems for the agent itself, OAuth-style on-behalf-of tokens for the principal — but the principle is simple: no anonymous agent actions.
Without this, the question “who authorized this action?” has no clean answer. With it, the answer is in the audit record.
Primitive 2 — Action Budgets
Limit how much damage any single action, or all actions together, can do.
A budget is a quantitative limit on consequential action, enforced at the gateway and depleted as the agent acts. Three V’s cover most cases:
Value. Per single transaction, per session, per principal, per beneficiary, per day.
Volume. How many records can be modified, how many emails sent, and how many policies updated in a single session?
Velocity. Cumulative limits across time windows. This catches the death-by-a-thousand-small-transactions pattern, where many sub-individual-threshold actions aggregate to material harm.
Budgets have to be enforced architecturally. A budget written into the system prompt is a suggestion. A budget enforced by the gateway is a constraint. When the agent’s spend hits the limit, the gateway starts rejecting actions, and the only path forward is escalation or session expiry. The agent cannot raise its own budget.
This sounds obvious. Most agentic deployments ship without it. Just like phones used to ship without parental controls.
Primitive 3 — Reversibility Tiers
Not all actions are equal. The agent’s authority should scale inversely with the difficulty of undoing the action.
Once the agent calls a tool that affects something outside its boundary — sends a payment, deletes a record, or generates a notification — the agent framework’s checkpointing (as in LangGraph) cannot undo the consequences. That kind of reversibility lives at the action gateway, in the tier classification of the tools, and the tool design itself.
Three tiers cover the practical cases:
Reversible — E.g., drafts that get reviewed/tested before committing to a version-controlled system.
Irreversible-but-bounded — E.g, General ledger entries are reversible within a settlement window. The cost of being wrong is bounded by the window.
Irreversible-and-unbounded — E.g., prescriptions filled. There is no path back. Most actions in this tier should require explicit human authorization per action; many should not be available to agents at all.
The Replit incident in this frame: deletion of production data was irreversible and unbounded from the agent’s perspective. That authority should never have existed. The remediation moved that operation entirely out of the agent’s reachable tier.
Primitive 4 — Mandatory Escalation Thresholds
Rules that force human approval, with teeth.
The pattern that does not work: instructions in the system prompt asking the agent to confirm before taking risky actions. The agent decides what counts as risky, the agent decides when to ask, the agent decides how to interpret the answer.
The pattern that does work: the action gateway requires a verified human approval token for actions above a threshold. No token, no action — the gateway rejects regardless of how confident the agent is. The token is issued by a separate workflow involving an actual human reviewer with appropriate authority, recorded with the reviewer’s identity and review duration, bound to a specific action so it cannot be replayed.
Useful threshold patterns:
Per-action thresholds. Any single transfer above value X.
Cumulative thresholds. Total value moved in the past hour.
Per-counterparty thresholds. Any new beneficiary not seen before.
Pattern-based thresholds. Velocity changes.
Mandatory escalation only matters if the human review on the other end is meaningful. A reviewer who approves 200 escalations a day, each in 90 seconds, is a rubber stamp with a paper trail, not a control. For now, the gateway’s job is to ensure the escalation actually happens (by blocking or refusing to proceed without an approval token, etc.).
Primitive 5 — Failure Containment
When the agent cannot safely act, it stops, defers, or escalates. Never guesses.
The default behaviour of most LLM-based agents is to try to be helpful anyway. To produce a best guess. To proceed with the action and hope for the best. This is the worst possible behaviour for a regulated system.
The architecture’s answer is the RED (refuse-escalate–defer) triad, and it’s worth being precise about which layer handles which:
Refuse — the action is rejected and reported. The gateway refuses when policy says no — budget exceeded, threshold tripped, tool not granted, identity invalid. The framework refuses when the agent’s own logic determines that the task can’t be completed — required information is missing, a subtask has failed, or an internal precondition isn’t met. Both produce a refusal record; the source matters for triage.
Escalate — the action requires human review through the approval-token mechanism. The gateway triggers escalation when policy thresholds require it. The framework drives the UI of the approval flow and resumes the workflow once the token is issued. The trust path of the approval — reviewer authority, signed token, binding to specific action parameters — is the gateway’s responsibility, not the framework’s.
Defer — the action is queued for later retry, when conditions change. Mostly a framework responsibility, because deferral is workflow logic — schedule the retry, hold the state, resume when the upstream dependency is back. The gateway can also defer in the sense of “try again later, this rate limit is transient,” but the orchestration of the retry lives in the framework.
What does not happen, in any failure mode: the agent takes the action anyway and notes that it was uncertain. That is fail-open behaviour. Crucially, this rule applies on both sides of the boundary. The framework cannot decide to fail-open because the gateway is slow. The gateway cannot fail open because the policy is ambiguous. Both layers default to refuse-or-escalate when in doubt.
In production, these limits will fire regularly. Refusal events will appear in dashboards. Teams that build the architecture well learn to read these as the system working — the agent encountered something its perimeter said it should not handle, and the perimeter held. Teams that read these as failures and lower the thresholds to reduce them are unwinding the architecture under operational pressure.
Example: An AI Shopping Assistant
The five primitives are easier to internalize against a familiar scenario than against an abstract one. Here is what they look like applied to something most readers can imagine: a personal AI assistant authorized to make online purchases for the user — flights, hotels, online shopping, subscriptions, restaurant deliveries.
The user has linked a credit card to the assistant. The card has a limit of ₹5L. The assistant runs as software — possibly on the user’s laptop, possibly on the user’s phone, possibly hosted in a cloud service the user authorized.
The bank (credit card authority) already runs its own fraud detection, AML pattern matching, sanctions screening, and velocity controls on every API (transaction). The MCP is developed by the bank to interact with agents. The action gateway is the additional layer that handles concerns that the bank’s systems cannot see. Here is what each primitive looks like.
Capability and Identity scoping. The agent is granted the make_purchase tool. It is not granted change_card_limits, update_billing_address, request_card_replacement, transfer_to_savings, or anything else in the bank’s API surface. The grant is signed by the user’s authorization step, expires after a defined window, and is bound to this card and this user’s identity. Every call carries a delegated identity token; the bank sees both the agent and the user, not just an anonymous service account.
Action budgets. The gateway enforces agent-specific budgets, layered on top of the bank’s actual credit limit. A session cap of ₹50K. A per-merchant cap of ₹20K. A per-category cap (travel, electronics, subscriptions) so the agent cannot rebalance the entire session into one category. A daily aggregate of ₹1L. The bank may permit more — its credit limit is ₹5L — but an AI assistant should not be authorized to consume the user’s full credit limit in a single session, even if the bank would technically allow it. The action gateway is the layer that enforces the user’s intent for what the assistant is allowed to do, separate from what the bank would allow the user to do.
Reversibility tiers. Online subscriptions, app store purchases, and e-commerce orders before fulfillment — reversible in practical terms. Domestic restaurant deliveries already fulfilled — irreversible-but-bounded; chargeback is possible but disputed. International merchants, especially in jurisdictions with weak chargeback support, are irreversible-and-unbounded in practice. The gateway scales authority inversely: higher agent autonomy on the reversible tier, tighter limits on the bounded tier, escalation on the unbounded tier.
Mandatory escalation thresholds. Several patterns trigger required user approval, enforced by the gateway as token-gated decisions; three examples:
Agent runtime mismatch. The agent was registered to run on the user’s laptop. A request comes in from an unfamiliar runtime location — say, a cloud IP the user has never authorized. Refuse until the user explicitly re-authorizes from this runtime. The bank cannot see this signal; the agent gateway can.
Cross-category burst. Within a single session, the agent attempts to make purchases across three or more unrelated categories (flights, electronics, subscriptions, and groceries). Plausible benign explanation, but suspicious enough to escalate. Compromised agents, when instructed to drain a card, often look like this.
First-time merchant above a threshold. A new merchant the agent has never transacted with, for an amount above ₹15K. Escalate — have the user explicitly approve the first transaction with this merchant. Once a merchant is on the user’s known-good list, subsequent transactions can run under normal budget.
Failure containment. If the user’s authentication system is unreachable, the gateway refuses rather than approves. If the policy engine cannot evaluate a rule (state-lookup timeout or missing data), the gateway refuses. If the bank’s payment authorization API is in degraded mode, the gateway defers (asks the agent’s framework via backpressure techniques) rather than retrying aggressively. Refuse, escalate, defer — never fail open into a transaction the user did not see authorized.
Every one of the gateway’s controls is something the bank’s existing platforms cannot see. The bank knows the card limit, the merchant category code, the transaction velocity, and the customer’s risk profile. The bank does not know that the call came from an AI assistant; that the assistant is supposed to be running on the user’s laptop and is now running somewhere else; that this transaction is out of session intent; that the user typically does not transact at 3am; that this is the agent’s first interaction with this merchant. Those are agent-context concerns, and they live at the action gateway because nothing else in the stack has visibility into them.
The bank’s existing controls remain authoritative for everything that has always been their job. The action gateway is doing only the additional job — and only that job.
The action gateway is a complementary control plane, not a replacement. It handles agent-specific concerns. It defers to existing controls for everything else. It never re-implements logic the underlying system already enforces. Multiple Perimeters.
Residual Risk
A note on residual risk before the deep-dive on tooling, because the architecture is strong, but it is not magic. Bounded autonomy contains a blast radius. It does not guarantee correctness. An agent can be perfectly well-bounded and still:
Make legitimate-but-wrong decisions within its authority. The architecture limits the consequence; it does not prevent the mistake.
Exhaust budgets in unhelpful ways. A confused agent stuck in a loop can burn through a session budget without producing useful work. The gateway rejects subsequent calls, resulting in a degraded experience for the user.
Generate alert fatigue through unnecessary escalations. An over-tuned threshold can route too many actions to human review, training reviewers to rubber-stamp.
Be confidently wrong about which tool to call. The agent has the right authority; it just chose the wrong target.
These are real failure modes. The architecture does not eliminate them. What it does is shift the cost: failures move from catastrophic to manageable. A bounded agent that makes a wrong decision affects one customer, one transaction, one record — recoverably. An unbounded agent that makes a wrong decision affects multiple customers.
Correctness is a model/agent problem. Containment is an architecture problem. Both matter. Brakes are a useful architectural element in a race car.
Eat This, Not That
The whole pattern in one image, before the technical reference begins. If you stop reading after this section, you have the takeaway.
Bounded autonomy is the architecture that makes agentic AI deployable in regulated workflows, not the one that makes it slower. Agents within a properly bounded perimeter can be highly autonomous — taking many actions quickly, processing large workloads, and operating without continuous supervision — because the perimeter is doing the safety work. The model does not have to be reliable enough to handle every edge case. The architecture handles the edge cases.
Without it, you are left with two unattractive options. You can deploy agents with broad authority and accept that incidents like Replit’s will eventually happen to you. Or you can wrap every agent action in a human approval gate, producing the kind of rubber-stamp human-in-the-loop that satisfies neither the regulator nor the operations team.
The third path is the architecture this post describes, and the tooling that now exists to build it.
Prompts can guide behaviour. Architecture must enforce authority.
The teams that build the perimeter first — before the agent goes into production, before the demo, before the headline — are the teams whose agents are still running a year later.
The architecture of observability, the cryptographic primitives that make logs trustworthy, and the question almost no one asks until it’s too late: when the lawyers arrive, what can your system actually prove?
A note on length. This is a long-form reference post — about a 25-minute read end-to-end. It is structured so you can also dip into the parts you need: the “Eat This, Not That” summary near the bottom is the screenshot version of the whole argument; the architecture and workflow diagrams in the middle are the reference artefacts most teams will return to. Engineers building the pipeline should read it linearly. Architects and CISOs reviewing a vendor’s audit posture can probably start at “Who Owns the Audit System” and work outward.
Imagine a story — composite, but unfortunately representative.
A mid-sized health system deploys an AI triage tool that flags potential sepsis cases in real time. It spans about 400 beds, integrated with the EHR, and includes a clinician-approval step before any care pathway is triggered. The deployment goes well. Sepsis flagging improves. Time-to-antibiotics drops. The board congratulates itself.
Eighteen months later, a family files suit. Their relative was admitted, the AI did not flag sepsis, and by the time the team caught it, it was too late. The complaint asks a simple question: on the morning of the admission, what data did the AI have, what did it decide, and why?
The CTO turns to her team. The team turns to the observability stack. The observability stack returns: API call counts, latency distributions, model uptime, and a structured log of inference requests with timestamps. What it does not return is the input payload that was actually scored at 06:42 that morning, the prompt template that was active in production at the time, the version of the retrieval index, the guardrail configuration, or the exact model checkpoint the inference ran against. The retention window on raw payloads expired at 90 days as a cost-saving measure two engineering quarters ago. Nobody’s quite sure which version of the model was running that morning because the vendor pushed a quiet update to its hosted endpoint.
The legal team asks a question the CTO has not been asked before: Can you prove what your system did, on what basis, eighteen months ago? And the honest answer is no.
That conversation — or some version of it — is going to happen at every organization deploying AI in regulated workflows over the next five years. Whether your team can answer the question depends on architectural choices made before the system shipped, not after. This post is about how to make those choices.
Observability in Regulated AI
In ordinary application observability, the question you’re trying to answer is “Is the system healthy and fast?” Logs, metrics, and traces are designed for that question. Engineers grep through them, dashboards summarise them, alerts fire on them, and after some retention period — a few weeks, maybe a few months — they’re aged out because the storage bill says so.
Regulated AI observability is doing something fundamentally different. It is doing everything the operational stack does, plus answering a separate set of questions that ordinary observability is not designed for.
The five questions that define the difference:
Reconstructability. Given an arbitrary historical decision, can you reconstruct exactly what the system saw, which model version produced the output, which retrieved context was used, which guardrails were active, and what the output and downstream action were? Three years from now. Two architectural rewrites later. After the vendor has been acquired. After the engineer who built it has left.
Integrity. Can you prove that the record of that decision has not been altered since it was written? Not “we trust our cloud provider’s access logs” — prove, in a way that survives a hostile party arguing the logs were modified after the fact.
Completeness. Can you prove no records were silently dropped, lost, or excluded? An audit trail with gaps is worse than no audit trail at all, because it manufactures plausible deniability for the wrong party.
Privacy compliance. Can you maintain reconstructability without retaining personal data beyond what the regulations permit? GDPR, HIPAA, the DPDP Act, and the Data Privacy Framework — they all impose minimisation requirements, and none of them cares that you needed the data for an audit trail.
Longevity. Can the system answer these questions across timeframes that exceed your software stack’s natural lifecycle? Most clinical liability cases surface 2-7 years after the fact. Most banking disputes fall within statutory limitation periods of 3-10 years, depending on jurisdiction. Your audit trail has to outlive the framework you wrote it in.
These are not the same as the operational concerns that traditional observability stacks are built for. They overlap — both involve writing structured data — but the storage architecture, retention policy, integrity guarantees, schema design, and access controls differ. Treating them as the same problem is the most common mistake teams make when shipping AI into regulated environments.
One caveat before going further. An audit trail does not, by itself, “hold up in court” — courts decide admissibility on multiple factors, including chain of custody, expert testimony, jurisdiction, and whether your organisation followed its own documented processes consistently. What a well-designed audit trail does is narrower and more important: it gives the organization a defensible, reconstructable record of what the system did and why. That record then becomes the raw material your legal team works with. The architecture in this post is what makes that raw material exist. What a court or regulator does with it is a different conversation, governed by different specialists, in a different room.
What to Capture (and What Not To)
Most teams overcollect at the wrong level of detail and undercollect at the right one. They store gigabytes of HTTP-level logs and have no record of which prompt template version was active. They retain raw model outputs forever and forget to capture the retrieval scores that produced them.
The right unit of capture is not the API call. It is the decision. Each AI decision — every meaningful inference that affects a patient, transaction, customer, or downstream action — gets a single audit-grade record with a known schema. Operational telemetry sits separately; it can have its own retention, its own store, its own access patterns. Mixing them is what creates both the cost problem and the auditability problem.
If you remember nothing else from this section, remember the schema below. It is the minimum viable audit-grade record. Eight field groups, each one answering a question that a regulator or court will eventually ask.
A few notes on the design choices that aren’t obvious from the schema diagram.
Hashes over payloads. Wherever possible, the audit record stores a hash/token-id of the input payload and a reference to where the raw payload lives — not the raw payload itself. This serves three purposes simultaneously. It keeps the audit-store size manageable. It allows independent retention policies for the audit metadata (long, sometimes permanent) and the raw payloads (often shorter, governed by privacy law). And it allows raw payloads to be encrypted, key-rotated, or even deleted on legal request without compromising the integrity of the audit record — because the hash still proves what was scored, even if the original is gone.
Reviewer dwell time. The review_dwell_ms field is non-obvious but worth capturing on every reviewed decision. Dwell time alone doesn’t prove cognitive engagement — a reviewer can stare at a screen for 60 seconds without thinking — but it is one of the few signals that help detect the opposite: instant approvals, rubber-stamp patterns, and reviewer fatigue at scale. Combined with output complexity, override rates, and downstream outcome correlation, it’s a corner of the picture that’s hard to fake.
Guardrail evaluations, not just guardrail config. Don’t just capture which guardrails were configured. Capture what each one evaluated on this specific request. “Toxicity filter: pass (score 0.04)” is auditable; “toxicity filter: enabled” is not.
Downstream system references. When an AI decision triggers a downstream action — a prescription order, a payment release, a flag in a CRM — capture the IDs of the downstream artefacts that resulted. This is what lets you answer the question “this transaction was disputed; what AI decision led to it?” without relying on log correlation across systems that may have aged out.
What not to capture, equally important:
Don’t capture intermediate token streams unless you have a specific use case for them. Token-level logs balloon storage and rarely answer questions you’ll be asked.
Don’t capture personal data unnecessarily in the audit metadata layer. The audit record should reference which patient (by stable internal ID) and which transaction (by stable internal ID), not the patient’s name, address, or transaction amount. The raw payload — which can contain those details — lives in a separate, more tightly controlled store.
Don’t capture vendor API metadata that’s likely to change schema on you. If you’re using a hosted model, capture the model version and your prompt, not the entire vendor request/response envelope. Vendor envelopes are not stable; your audit trail needs to be.
The Immutability Layer
Capturing the right data is the easier half. Proving that what you captured is what was actually written at the time, and not edited later by someone with database access, is the harder half. This is where most teams quietly fail their first audit.
The naive approach is “use immutable storage.” S3 with object lock, or Azure Blob with immutability policies, or any of the cloud-native WORM (Write Once, Read Many) options. This is fine as far as it goes, but it has two problems. First, it’s expensive at the volumes regulated AI generates — billions of records over multi-year retention add up fast. Second, it depends on trusting the cloud provider’s enforcement of the immutability policy, which an aggressive opposing counsel can challenge.
The better approach is to layer cryptographic integrity on top of cheaper storage. The pattern is well-established outside the AI world — it’s how banking transaction logs, blockchain anchoring services, and certificate transparency logs work — but it’s underused in AI observability.
Three layers, each cheap, each adding a property that the previous layer can’t provide alone.
Layer 1: hash chain. Each audit record contains the hash of the previous record. Standard append-only-log pattern. The cost is one extra hash field per record. The benefit is that any tampering — modifying an old record, deleting a record from the middle, inserting a record after the fact — breaks the chain at the point of tampering and every hash downstream from it. You can detect tampering by re-walking the chain.
Layer 2: Merkle anchor. Periodically (every N records, every T minutes, your choice — typical values: every 1,000 records or every 10 minutes), compute a Merkle root over the batch of records since the last anchor. A single 32-byte hash now cryptographically commits to thousands of records. This is the unit you’ll publish externally, which keeps the external publication cost trivially small even at high record volumes.
Layer 3: external witness. Publish the Merkle root somewhere outside your own infrastructure, so that even an adversary with full access to your systems cannot rewrite history without leaving evidence. Four common patterns:
A WORM/object-lock store paired with independent timestamping. Cloud object storage (S3 Object Lock, Azure Immutable Blob, GCS Bucket Lock) configured for write-once retention, with the Merkle root co-signed by an external timestamp authority.
Pros: cheap, well-understood operationally, the timestamp authority keeps the integrity claim defensible even if someone challenges the cloud provider’s WORM enforcement.
Cons: You have to operate the timestamping integration yourself.
A managed confidential ledger like Azure Confidential Ledger.
Pros: easy to use, integrates with cloud-native deployments, backed by confidential computing enclaves that further raise the bar for tamper resistance.
Cons: still inside the cloud provider’s trust boundary, which a hostile opposing party may argue against.
An RFC 3161 timestamp authority. A mature, decades-old standard used in document and code signing, defined by the IETF in RFC 3161. A trusted third party signs your hash with a timestamp.
Pros: legally well-understood, accepted in most jurisdictions, and auditor-friendly.
Cons: requires choosing and trusting a TSA vendor.
A public chain anchor via something like OpenTimestamps. Anchors your hash to Bitcoin or another widely-witnessed chain.
Pros: maximally adversarial-resistant; nobody can rewrite Bitcoin’s history.
Cons: Regulated industries are sometimes squeamish about the optics of “we use blockchain,” even when the use case is straightforward.
For most regulated AI deployments, an immutable store, along with independent timestamping, is the pragmatic default. A managed confidential ledger is a good option where the trust boundary and cloud dependency are acceptable. The public-chain option is for the genuinely adversarial cases. Pick one and document the choice; switching later is hard.
Who Owns the Audit System?
“The Vendor Handles It” Is the Wrong Answer
Up to this point, I’ve described the audit pipeline as a single thing. In practice, the most consequential question is who owns it. The default assumption — that the AI application or vendor handles audit, that observability is a feature of the platform — quietly fails at the worst possible moment.
The short answer: in a regulated industry, the audit system is owned by the regulated entity, not the AI application/system. The hospital owns it, not the clinical AI vendor. The bank owns it, not the SaaS underwriting platform. The insurer owns it, not the claims-triage tool. AI applications produce records into the enterprise’s audit fabric; they do not constitute it.
The reason is a property of regulatory liability that engineers often miss. When a regulator opens an investigation into an adverse outcome, the question they ask is not “what does your AI vendor’s audit system show?” It is “What does your audit show?” The regulated entity is on the hook. They can pursue the vendor contractually after the fact, but they cannot delegate their regulatory obligation. An audit system that lives only inside the vendor is, from the regulator’s perspective, not your audit system at all. It’s a third-party assertion you’ll be asked to corroborate from your own records.
This has architectural consequences that compound quickly:
Records must leave the AI application’s trust boundary. Audit records produced by the AI cannot live solely on the AI vendor’s infrastructure. They must be transported into the enterprise’s own audit fabric, signed by the producing application, and stored under the enterprise’s controls. If the vendor disappears tomorrow — acquired, bankrupt, breached, contract terminated — your audit obligations don’t disappear with them.
The schema is the contract, not the product. When you procure an AI application, the audit-record schema becomes part of the contractual artefact set. The vendor must produce records that conform to your schema, on your transport, signed with credentials you control. If the vendor cannot or will not do this, that is a procurement failure, not a technical detail to be negotiated later.
Internal AI applications get the same treatment as vendor ones. This is the discipline that’s hardest to enforce. When the team next door builds an internal AI tool, the temptation is to let them use whatever logging library they prefer and skip the formal audit pipeline. Don’t. The discipline only works if every AI producer — internal, vendor, hybrid — produces into the same audit fabric using the same schema.
The SaaS Question
Most regulated AI in production today is delivered as SaaS. This is not a problem in principle, but it makes the ownership question sharper.
When an AI application is delivered as SaaS, the inference happens on the vendor’s infrastructure, the model weights are the vendor’s, the prompt templates are sometimes the vendor’s, and the retrieval indices may be the vendor’s. The vendor has every operational reason to want to own the audit trail too — it’s where their telemetry lives, it’s where their improvement signals come from, it’s where they can demonstrate value back to the customer. Most SaaS contracts default to the vendor owning the audit logs.
This default is wrong for regulated buyers. Here’s what the contract and the architecture have to enforce instead:
The vendor produces audit records on the customer’s behalf. The records belong to the customer the moment they’re produced. The vendor may keep a copy for their own operational purposes (with appropriate data agreements), but the authoritative record lives in the customer’s audit fabric, not the vendor’s.
Records are produced over a customer-controlled channel. Even though the inference occurs on the vendor’s infrastructure, the audit record is signed by the application instance using credentials the customer issued (typically through a workload identity system like SPIFFE) and transported over a connection that the customer’s network controls. The vendor cannot quietly stop sending records, replay old records, or rewrite records in flight without leaving evidence.
Schema conformance is a contractual obligation. Vendors who want to sell into regulated industries have to produce records that match the customer’s audit schema, including the integrity envelope. This is one of the most common procurement gaps; it should be a non-negotiable line item before a contract is signed.
Customers can independently verify. The customer’s audit fabric must be able to verify, without consulting the vendor, that records are well-formed, signed by the correct producer, in correct sequence, and anchored. If verification depends on calling the vendor’s API, the verification is not independent.
Vendor model and policy updates produce audit events. When the vendor pushes a model update — or a prompt template change, retrieval corpus refresh, guardrail policy revision, routing rule change, or any threshold adjustment — that update itself becomes an auditable event. All of these can shift behaviour as much as a weights update can, and customers often discover the change only when output drifts. The customer’s audit fabric should capture which model version, prompt version, and policy configuration were active for each decision, with sufficient precision to enable a “before update” and “after update” comparison months later. Without this, the most consequential class of regulatory questions (“did the AI behave consistently before and after the change?”) becomes unanswerable.
The hard truth is that many SaaS AI vendors are not yet ready to meet these requirements. Their audit features are designed for their own operational needs, not for regulated customer evidentiary needs. This is a market gap that regulated buyers are increasingly closing through procurement leverage. If your vendor cannot meet these requirements today, the right move is to include them in the contract anyway, with a timeline, and make them a renewal condition.
Can Hospitals and Banks Afford a Different Audit System Per AI Vendor?
No. They cannot. And this is the operational truth that drives the whole architecture.
A typical mid-sized hospital today has anywhere from eight to twenty AI applications in some stage of deployment — sepsis detection, radiology triage, ambient documentation, billing copilots, clinical decision support, medication-error checking, scheduling optimisation, scribing, and so on. A bank has a similar or larger spread across credit decisioning, KYC, AML, fraud detection, customer service automation, and document understanding.
If each of those AI applications had its own audit system, the regulated entity would be operating eight to twenty different audit pipelines, with eight to twenty different schemas, eight to twenty different retention policies, and eight to twenty different reconstruction interfaces. When the regulator asks “show me every AI decision made about this patient between June and August,” the compliance team would have to query eight to twenty different systems, normalise the outputs, and hope the timestamps are reconcilable. That is an operational impossibility. It is also exactly the situation many enterprises are sliding into by default.
The alternative, and the only architecture that scales, is a single enterprise-owned audit fabric that every AI application — internal or vendor, in-house or SaaS — produces records into. The schema is owned by the enterprise. The transport is owned by the enterprise. The storage is owned by the enterprise. The reconstruction interface is owned by the enterprise. AI applications are producers; the audit fabric is the system of record.
This is the architecture that the rest of this post describes. The ownership question is what makes it real.
Seven tiers, each with its own technology choices, each owned and operated by the enterprise. Producers — whether internal teams or external vendors — conform to the published schema and produce into Tier 2’s transport. Everything from there is the enterprise’s responsibility. This is the architecture that lets a hospital with fifteen AI applications still answer a single regulatory question with a single query.
Transport Security: The Most-Attacked, Least-Discussed Layer
There is an obvious attack on the architecture above that nobody likes to talk about. Audit records are most vulnerable in the gap between when they are produced (inside the AI application) and when they are sealed into the chain (at Tier 4). If an attacker can tamper with records in that gap, the entire integrity story downstream becomes fiction. The hash chain is faithfully recording records that were already corrupted before they arrived.
This is why transport security in audit pipelines is a different problem from transport security in operational telemetry. For operational telemetry, you mainly care that data gets there; the occasional dropped span doesn’t matter. For audit records, you care that every record arrives in the form it was produced, is signed by the producer identity, is deduplicated, and is ordered within the relevant producer or decision stream — with globally consistent anchoring across streams handled at the Merkle batch layer. Any of those properties failing breaks the evidential value.
The minimum controls:
Mutually-authenticated transport (mTLS everywhere). The producer authenticates the receiver, and the receiver authenticates the producer. No anonymous publishers. No shared bearer tokens. This eliminates the “we accepted records from an attacker who claimed to be the AI app” failure mode.
Workload identity, not service accounts. Use a workload identity system (SPIFFE/SPIRE in Kubernetes-heavy environments, cloud-native equivalents like AWS IAM Roles for Service Accounts or Azure Workload Identity elsewhere) so that each AI application instance has a verifiable cryptographic identity. The signature on the audit envelope is verifiable against that identity. If the AI application is compromised and an attacker tries to produce records from a different identity, the signature check fails.
Signed envelopes, signed inside the application boundary. The signature on the audit envelope is computed within the producing application, using a key that the application controls, before the record leaves. If signing occurs at the network edge (a sidecar or gateway), then anyone who can inject between the application and the edge can produce unsigned-but-accepted records. Sign at the source.
Idempotent sequence IDs and replay detection. Every record has a decision_id that is unique identifier within the producing application. The gateway dedupes on this ID. An attacker who replays records will produce duplicates that the gateway rejects. Without this, an attacker who captures a legitimate record can replay it to manufacture false evidence.
At least once delivery, not at most once. The transport must guarantee delivery and retry on failure, with the gateway’s dedupe handling the resulting duplicates. The opposite (at-most-once) silently drops records in the face of transient failures, and silent loss is the worst possible failure mode for an audit system.
Backpressure that fails closed, not open. If the transport pipeline is overloaded and cannot keep up, the producing AI application must either block on the audit submission or refuse the inference. The pattern that must not happen is “fire and forget the audit, return the answer to the user.” That pattern produces actions without records, which is the most legally damaging failure mode possible. Fail closed: if you can’t produce the audit, you don’t produce the action.
These are not unusual controls. They’re standard for high-integrity transactional systems — banks have used them for decades to record transactions. They are still missing from most AI observability deployments because the deployments grew out of an operational logging culture, where dropped records are an inconvenience rather than a liability.
The Audit Record Workflow, End to End
Putting it all together, this is what the journey of a single audit record looks like — from the moment the model produces an output to the moment the record is permanent, anchored, and queryable.
Read this flow as a chain of integrity properties. Each stage adds or preserves a property; the combination is what makes the final record evidential rather than merely informational.
A record that has only some of these properties is not partial evidence — it’s fragile evidence, in ways that can be hard to spot until they’re tested under adversarial conditions. The point of the architecture is that every stage has its own attacker profile and its own control, and the controls compose. If a record is forged outside the producer identity, the gateway’s signature verification catches it. If the producer itself is compromised, no single signature check will save you — the control then shifts to key isolation, runtime attestation, anomaly detection, sequence monitoring, and downstream reconciliation against the actions the AI actually took. If the gateway is compromised, the external witness catches the silent rewrites. If the external witness is compromised, choose two witnesses. The architecture degrades gracefully under partial failure, which is what “evidential” actually means in operational terms.
The combined architecture gives you something specific: the ability to take any historical audit record, walk the hash chain to its enclosing Merkle batch, fetch the externally-witnessed root, and produce a cryptographic proof that the record existed in its current form at the time of anchoring. That is the kind of evidence package a regulator, auditor, or court can evaluate.
OpenTelemetry: Yes, But Not For This
So far, this post has built up a custom-looking pipeline: producer SDKs, signed envelopes, gateways, hash chains, Merkle anchors, and immutable storage. A reasonable question at this point is, “Doesn’t OpenTelemetry already do most of this?” The answer is no, and the why is worth understanding clearly — because OTel is going to be in your stack anyway, and the question isn’t whether to use it but where to draw the line between what OTel handles and what the audit fabric handles.
OpenTelemetry is the right answer to most observability questions in 2026. It is not, by itself, the right answer to regulated AI audit trails. The distinction matters because most teams default to OTel and end up with an audit posture that is operationally excellent but legally indefensible.
What OTel does well: distributed tracing across the AI request path, latency and throughput metrics, correlation IDs that link your gateway to your model serving layer to downstream actions, and structured event emission in a well-understood vendor-neutral format. For operational observability of AI systems, OTel is excellent — and the emerging OpenTelemetry semantic conventions for GenAI (covering attributes like gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens, etc.) make it easier than before to consistently capture LLM-specific telemetry, even though significant portions of those conventions are still in active development.
What OTel does not do: provide immutability, provide cryptographic integrity guarantees, enforce retention policies that distinguish operational telemetry from audit-grade records, or guarantee that what you emitted survives in the form you emitted it.
The right pattern is to use OTel as the instrumentation and correlation layer, and to route audit-grade records to a separate immutable sink that lives outside the trace and metrics pipeline.
In practice, this looks like:
Your application emits OTel spans for every model call, with the GenAI semantic-convention attributes attached. These flow to your standard observability backend (Datadog, Honeycomb, New Relic, your self-hosted Jaeger/Tempo, whatever).
For decisions that meet the audit-grade threshold (typically: any decision that affects a patient, transaction, customer, or downstream action), the application also emits a separate audit record — typically as a structured event to a dedicated Kafka/Kinesis topic — that flows into the immutable audit pipeline described above.
The two records share a correlation ID (the OTel trace ID is fine for this), so an investigator can pivot between operational telemetry and audit evidence.
Two failure modes to avoid:
Don’t try to make your trace store the audit store. APM and tracing platforms are designed for short-retention, high-cardinality, mutable data. They will happily lose your spans, sample them, age them out, or schema-evolve them under you. None of those behaviours is compatible with audit requirements.
Don’t double-write everything to both stores. Decide which records cross the audit-grade threshold and route only those records. A retrieval that returns no results and triggers no action is operational telemetry; the same retrieval that grounds a clinical recommendation is an audit event. Same span, different routing.
The PII Problem (What “Masked” Means In Court)
This is the section where most AI observability discussions go quiet, because the honest answer is uncomfortable.
Regulated AI systems process personal data. The audit trail therefore captures, references, or is otherwise entangled with personal data. Privacy law (GDPR’s Article 5, HIPAA’s minimum necessary standard, the DPDP Act’s purpose limitation, every analogous regime) requires that personal data be retained only as long as necessary for the purpose collected, in the minimum amount necessary, with appropriate safeguards. None of those laws care that your audit retention requirements are longer than your data retention requirements.
You cannot solve this by “just masking everything.” There are at least four different things that get called masking, and they have very different properties.
Redaction (irreversible). Replace Nitin Mallya, Aadhar number 9876 5432 1000 6723 with [NAME], [AADHAR NUMBER]. Easy. Cheap. And often fatal to your audit trail’s reconstructability, because if a court asks “what did the AI decide for Nitin on June 14th?”, you may not be able to answer from the audit trail itself. You’ve made a record of some decision involving some patient, which is not what a regulator is asking for.
Hashing (deterministic but not reversible). Replace 9876 5432 1000 6723 with sha256("9876 5432 1000 6723"). Better. The same patient now produces the same hash across records, so you can correlate decisions for the same person without storing the identifier. But there’s a gotcha: hashes of small identifier spaces (account numbers, anything with limited entropy) are trivially reversible by an attacker who can rainbow-table the input space. For most regulated identifiers, raw hashing is barely better than plaintext from a privacy standpoint. You need a salt or HMAC key, stored separately, with its own access controls.
Tokenisation (reversible, with controlled access to the mapping). Replace 9876 5432 1000 6723 with token tok_8a2f3c…, and store the mapping tok_8a2f3c → "9876 5432 1000 6723” in a separate, tightly access-controlled token vault. This is the pattern that actually works in regulated environments. The audit record contains the token, which is meaningless without the vault. The vault has its own access controls, audit logs, and can be subject to legal hold. When a court asks “what did the AI decide for patient X?”, you authorise a single, logged dereferencing of X’s token to find their records — and you have a logged record of who looked, when, and why.
Format-preserving encryption. Replace 9876 5432 1000 6723 with 6789 2354 0001 3276 (still sixteen digits, still passes type validation, but encrypted under a key you control). Useful when downstream systems need data of the right shape but not the actual value. More complex than tokenisation; rarely worth the complexity unless you have a specific schema-compatibility constraint.
For audit trails that must support lawful reconstruction, the right default is tokenisation, with the token vault treated as a first-class compliance artefact: separately encrypted, separately access-controlled, separately backed up, with its own audit log of every dereferencing operation.
Now the part nobody likes to talk about.
What “masked” means when the litigation arrives. When opposing counsel deposes your CTO and asks “did your AI system make a decision involving my client on June 14th, 2024?”, the answer your team gives depends entirely on which masking strategy you chose.
If you redacted: your honest answer is “we don’t know.” That answer is not a defence. In some jurisdictions, the inability to produce records that you should reasonably have kept is itself adverse to your case. Spoliation is the legal term, and it has consequences.
If you hashed: your honest answer is “we can check, but it depends on the entropy of the identifier and the strength of our salt.” This is a fragile answer to give in court.
If you tokenised: your honest answer is “yes, here are the records.” The vault dereferencing produces evidence. The vault’s own audit log proves the dereferencing was authorised and lawful.
The choice of masking strategy is therefore not just a privacy choice. It is a litigation-readiness choice. Teams that redact-by-default are choosing privacy maximalism at the cost of being unable to defend themselves when the case comes. Teams that tokenise correctly are choosing a strategy that satisfies privacy regulators (the audit store contains no decryptable PII) and preserves their ability to respond to lawful production demands (the vault is the controlled choke point).
This nuance does not show up in the privacy-by-design literature. It shows up in the discovery phase of every AI lawsuit that has yet to be filed.
Retention
One more piece. How long do you keep all of this?
This is not an engineering decision. It is a legal and compliance decision that engineering implements. The single most common pathology in regulated AI observability is engineering deciding retention based on storage cost, then discovering after a lawsuit that legal would have set retention to a longer duration.
The right default architecture is tiered retention, set by record type, governed by legal and compliance:
Operational telemetry (OTel spans, metrics, ops logs): days to weeks, set by SRE for incident response needs.
Audit metadata records (the structured records described above, minus the raw payloads): typically the longest of (regulatory mandate for the industry, statute of limitations for likely litigation, internal policy). For healthcare AI in most jurisdictions, this means 6-10 years minimum. For banking AI, often longer.
Raw payloads (the actual inputs and outputs the audit metadata references): governed by data minimisation requirements; often much shorter than audit metadata. The hash in the audit record proves what was scored, even after the raw payload is deleted.
Token vault: governed by the same regime as the underlying personal data, with the additional constraint that it must outlive the audit records that reference it (otherwise the audit records become unreadable).
Legal hold overrides everything. When a litigation hold notice arrives, deletion stops for everything in scope, regardless of what the default retention policy says. The system must support this as a first-class operation, not as a panicked all-hands at 11pm on a Friday.
The other thing engineering does not get to decide: deletion. In regulated environments, “we deleted the data to save cost” is not a defence; it is an admission. Any deletion policy must be reviewed by legal, executed automatically by the system (not by engineers running scripts), and itself logged in the audit trail. The fact of deletion, the policy that authorised it, and the records affected — all of it goes in the audit trail.
Eat This, Not That
The whole architecture in one image, for the people who got this far.
What This Buys You
A regulated AI observability stack built this way is not just a compliance artefact. It is a system property.
It buys you the ability to answer, with evidence, the question that started this post: “what did the AI decide on the morning of the admission, and on what basis?” It buys you the ability to defend that answer in front of a regulator who has read your validation protocol and a court that has not. It buys you the ability to detect drift, debug failures, and reconstruct incidents long after the team that built the system has moved on. It buys you the ability to comply with privacy law and litigation requirements simultaneously, which most teams treat as a contradiction and is not.
It also buys you the right to deploy AI in workflows where the cost of being wrong is asymmetric — which, as Part 1 of this series argued, is the only kind of deployment that actually moves the needle for regulated industries.
The architecture is not free. Engineering effort is real. The token vault is a non-trivial system. The Merkle anchoring requires choosing and operating a witness. The schema discipline requires governance. None of this is what your data engineers signed up for when they joined.
But the alternative is the conversation the CTO had at the start of this post. Multiplied across an industry that is now deploying AI into the workflows that matter. The teams that build the audit trail right will at least enter their first lawsuit with evidence instead of explanations. The teams that don’t will become the case study that justifies the next round of regulation.
Build the audit trail. Build it now. Build it before the lawyers arrive — because they will.
FAQ: Regulated AI Audit Trails
Ten questions a senior practitioner is likely to ask after reading this article. Answers are calibrated for technical leaders, architects, and CISOs — not for cryptography specialists, but not for beginners either.
What is a Merkle tree, and why does this architecture need one?
A Merkle tree is a way of producing a single short hash — typically 32 bytes — that cryptographically commits to a large set of records. You hash each record individually, pair the hashes and hash the pairs, then pair those and hash again, and so on until you reach a single root hash. If any record in the original set is altered, the root changes. If the root hasn’t changed, the records haven’t either.
The architecture needs Merkle trees for one practical reason: cost. Without them, you would either have to publish every audit record to an external witness (expensive and slow at the volumes regulated AI generates) or trust that your own storage layer hasn’t been tampered with (defeats the point). With a Merkle tree, you batch thousands of records together and only publish the root externally. A 32-byte hash now stands as cryptographic evidence for the integrity of the entire batch. The maths means you can prove any individual record’s inclusion in the batch with a small “Merkle proof” — a handful of hashes — without needing the rest of the batch.
Merkle trees are not exotic. They are how Bitcoin organises transactions in a block, how Git tracks file changes, and how Certificate Transparency logs prove the integrity of TLS certificates issued across the public internet. The pattern is decades old and well-understood. The only new thing is applying it to AI audit records.
Why OpenTimestamps vs. “putting things on the blockchain”?
OpenTimestamps is a free, open protocol that lets you anchor a hash to the Bitcoin blockchain without paying transaction fees, running a node, or publishing anything sensitive. It works by aggregating large numbers of submitted hashes into its own Merkle tree, and committing only the root of that tree to Bitcoin. Each user gets back a small proof file that, combined with the public Bitcoin blockchain, proves their hash existed at a specific point in time.
The distinction from “putting things on the blockchain” matters. People hear “blockchain” and imagine storing the actual data on-chain — which would be expensive, slow, and would expose sensitive content publicly. OpenTimestamps does the opposite: nothing about your audit records goes anywhere near Bitcoin. Only a hash of a hash of a hash, with no way to reverse it back to your data, ever touches the public chain. What you get is a proof of existence — evidence that this hash existed at this time, witnessed by an entire global network — without any data exposure.
For most regulated organisations, OpenTimestamps is the maximally adversarial-resistant external witness. Nobody can rewrite Bitcoin’s history to falsify your audit trail. The trade-off is operational complexity (you need to manage the proof files) and the optics question — some regulated industries are still squeamish about “blockchain” in any form, regardless of the technical reality.
What is RFC 3161, and why should I trust it?
RFC 3161 is an IETF standard from 2001 that defines how to get a trusted third party — called a Time Stamping Authority, or TSA — to digitally sign a hash with a precise timestamp. You send the TSA a hash, they sign it along with the current time using their private key, and the resulting signed object proves that this hash existed at this time, attested to by this TSA. You can verify the signature later using the TSA’s public certificate without contacting the TSA again.
Trust comes from the same place it comes from for TLS certificates: a chain of cryptographic signatures back to a root authority that auditors and courts already accept. Most national post offices, several governments, and a number of commercial vendors operate RFC 3161 TSAs. The standard has been used in regulated industries for over two decades — code signing, document signing, e-invoicing, court-admissible electronic evidence — and the legal weight of an RFC 3161 timestamp is well-understood in most jurisdictions.
For AI audit trails, RFC 3161 is the boring, mature, defensible option. It is what your legal team will be most comfortable with, because they have already seen it accepted in non-AI contexts. The cost is choosing a TSA vendor and integrating with their API, both of which are routine.
What’s the difference between WORM storage and a managed ledger?
WORM (Write Once, Read Many) storage is a property of an object store: once you write a file, you cannot modify or delete it until a configured retention period expires. AWS S3 Object Lock, Azure Immutable Blob Storage, and Google Cloud Storage Bucket Lock all implement WORM mode. The cloud provider enforces the immutability — your application code cannot bypass it.
A managed ledger is a different category of service. Azure Confidential Ledger is the canonical example. It provides an append-only data structure with built-in cryptographic integrity (hash chain, Merkle proofs), runs inside hardware-secured enclaves, and produces verifiable receipts for every entry. The provider gives you not just immutable storage but also the integrity proofs as a service.
The architectural difference: WORM gives you immutability, but you have to build the integrity layer (hash chain, Merkle anchoring) yourself on top. A managed ledger gives you both. The trade-off is cost (managed ledgers are typically more expensive per write than object storage), trust boundary (you’re trusting the cloud provider’s enclave attestation rather than your own cryptography), and lock-in (managed ledgers don’t have a portable standard — you can’t easily migrate from one provider’s ledger to another).
The pragmatic default for most regulated AI audit fabrics is WORM object storage paired with independent timestamping (RFC 3161 or OpenTimestamps). Managed ledgers make sense when the operational simplicity is worth the cost premium and the cloud-trust dependency is acceptable.
What is SPIFFE/SPIRE, and why not just use API keys?
SPIFFE (Secure Production Identity Framework For Everyone) is an open standard for issuing cryptographic identities to software workloads — services, containers, functions — automatically, at runtime, without humans handling secrets. SPIRE is the reference implementation. Together they let every running instance of an AI application have its own short-lived, verifiable cryptographic identity, rotated continuously, without any team ever needing to manage an API key.
The alternative — API keys — has three problems that matter for audit. First, API keys identify the application, not the instance; if you have ten copies of an AI service running, they all sign records with the same key, so an attacker who compromises one instance can produce records that are indistinguishable from any of the others. Second, API keys are long-lived; if one leaks (and they do leak — into logs, into git history, into screenshots), the attacker has months or years before rotation. Third, API keys are bearer tokens; anyone who holds the token can act as the identity. There is no cryptographic proof of who is currently using the key.
SPIFFE solves all three. Each instance has its own identity. Identities are short-lived (typically rotated every few hours). Authentication uses asymmetric cryptography, so possessing a SPIFFE identity means controlling the private key, not just holding a token someone copied. For audit records, this means the signature on each record traces to a specific instance, at a specific time, with cryptographic guarantees that are dramatically stronger than “we trust whoever sent us a valid API key.”
You don’t strictly need SPIFFE — cloud-native equivalents (AWS IAM Roles for Service Accounts, Azure Workload Identity, GCP Workload Identity Federation) provide similar guarantees with provider lock-in. The principle matters more than the implementation: workload identity, not service accounts; short-lived credentials, not long-lived secrets; per-instance attribution, not application-wide.
What does “fail closed” actually mean in production?
Fail closed means: if the system cannot perform the action safely, it does not perform it. Fail open means: if the system cannot perform the action safely, it performs it anyway and hopes for the best.
In the context of audit pipelines, fail closed means: if your AI application cannot successfully write an audit record (transport is down, gateway is unreachable, signing key is unavailable), the application blocks the inference or refuses to return the answer — until the audit can be written. Fail open means the application returns the answer to the user and tries to write the audit later, accepting silent loss as a possibility.
Most operational systems default to fail open because it improves availability. For audit pipelines in regulated AI, fail open is the worst possible failure mode: it produces actions without records. An AI agent took an action, the user saw it happen, the downstream system was updated — but there is no audit trail of the decision. From a regulator’s perspective, this is indistinguishable from the system having taken an unauthorised action with the team trying to cover it up. Even if the cause was a benign infrastructure hiccup, the absence of evidence is itself adverse to the organisation’s case.
In production, fail closed is implemented as: the audit submission is on the synchronous path of the inference. If the submission fails after retries, the inference returns an error to the user (or queues for human review, depending on the workflow). The team will hate you when the audit pipeline has an outage and the AI features start failing. They will hate you less than a regulator finding gaps in your audit trail.
How is this different from logging?
Logging is for engineers debugging the system. Auditing is for proving what the system did to someone who wasn’t there.
The differences cascade through the architecture. Logs can be edited or deleted (often by the same engineers who write them); audit records cannot. Logs can be sampled or dropped under load; audit records must be guaranteed to arrive, in full, in order, exactly once. Logs are typically kept for weeks; audit records are typically kept for years. Logs use whatever schema the developer thought useful; audit records conform to a published schema that legal and compliance have signed off on. Logs are accessed by anyone on the team with the right roles; audit records have access controls, dereferencing audit logs, and legal-hold overrides.
Most teams treat AI audit as “structured logging with a longer retention.” That treatment fails the first time someone asks “prove that nobody on your team modified this record.” Logs cannot prove that. Auditing is logging plus integrity, plus governance, plus retention discipline, plus access controls, plus the cryptographic infrastructure to defend the record’s authenticity. The architecture in the main post is what gets you from one to the other.
Can we use blockchain for the whole audit trail instead?
In principle, yes. In practice, no.
Blockchains have several properties that look attractive for audit: immutability, cryptographic integrity, distributed witness, well-understood verification. But they have several properties that disqualify them for regulated AI audit at scale.
Cost is the first problem. Public blockchains charge per write, often substantially. Writing every audit record to a public chain would bankrupt the audit budget within weeks of going to production. Private or permissioned chains (Hyperledger, Quorum, etc.) avoid the per-write fees but lose the adversarial-resistance property — they’re now back inside your trust boundary, with all the same questions you’d have about WORM storage but with much more complex operations.
Privacy is the second problem. Once data is on a public chain, it is on the chain forever, visible to everyone, regardless of what privacy law says. You cannot delete it on a GDPR erasure request. You cannot tokenise it after the fact. Hashes of personal data, written carelessly, can be reversed by anyone with the patience to brute-force a small input space. The chain is the worst possible place to store anything that touches PII.
Throughput is the third. Public chains process tens of transactions per second. A regulated AI deployment may produce hundreds or thousands of audit records per second. The mismatch is several orders of magnitude.
The right pattern is what the main post describes: keep the audit records in your own infrastructure, build a hash chain locally, batch the chain heads into Merkle roots, and only commit the Merkle roots to a public chain (via OpenTimestamps) or a managed ledger. The chain is used as a witness, not as a database. This gets you the integrity property without the cost, privacy, or throughput problems.
Building Blocks: Trillian vs. Sigstore Rekor vs. Build Your Own?
All three solve the same problem: append-only, ordered, cryptographically verifiable storage for audit records. The differences are in maturity, intended use case, and operational footprint.
Trillian is Google’s open-source verifiable log implementation. It is what powers Certificate Transparency — the global infrastructure that monitors TLS certificate issuance to detect rogue certificate authorities. It is battle-tested at internet scale, well-documented, and designed to be operated by people who take audit infrastructure seriously. The downside is that it’s a significant operational commitment; running Trillian well requires real expertise.
Sigstore Rekor is part of the Sigstore project, originally designed for software supply-chain transparency (signing open-source artefacts, recording attestations). It is built on the same verifiable-log primitives as Trillian but with a more opinionated API, easier deployment, and a smaller operational footprint. For organisations that want a verifiable log without operating infrastructure at the Trillian level of seriousness, Rekor is the more pragmatic choice.
Rolling your own is the right choice when your scale is small, your team has the cryptographic expertise to build it correctly, and you have specific requirements that don’t fit either Trillian or Rekor. The risk is that hash-chain writers are easy to write and hard to write correctly; subtle bugs around concurrent writes, replay handling, or signature verification can quietly corrupt the integrity of the entire chain. If you go this route, treat it as security-critical code, with the review and testing discipline that implies.
The pragmatic default for most regulated AI audit fabrics is Sigstore Rekor — it gets you most of what Trillian provides at a fraction of the operational complexity, and it has a healthier ecosystem than custom code.
How does this interact with GDPR’s right to erasure?
This is the question with no clean answer, and any vendor or consultant who tells you otherwise is selling something.
GDPR Article 17 grants data subjects the right to have their personal data erased under certain conditions. Audit retention requirements — for regulated industries, often six to ten years or more — create personal data that the organisation has a legal obligation to keep. The two regimes can collide directly.
The architecture in the main post is designed to handle this collision in the cleanest way the law allows, but it does not eliminate the tension. The pattern: personal data lives in the raw payload store and the token vault, not in the audit-record metadata. The audit records themselves contain only tokens and hashes. When an erasure request arrives and is determined to be valid (which is a legal determination, not a technical one), the personal data in the raw payload store can be deleted, and the token vault entries for that data subject can be deleted. The audit metadata records remain, with their tokens now pointing at vault entries that no longer exist. The records still prove what happened — some AI decision involving some (now-erased) data subject — but the personal connection is severed.
This is sometimes called “crypto-shredding” — using key destruction or vault-entry destruction to render previously-encrypted data effectively unrecoverable. Whether it satisfies GDPR Article 17 in any specific case is a legal determination that depends on the jurisdiction, the nature of the regulated retention obligation, the specific data, and how courts and Data Protection Authorities have interpreted “erasure” in similar cases. In some regulated contexts, the regulated retention obligation overrides the erasure right. In others, it doesn’t.
The architectural answer is therefore: build the system so that erasure is possible without breaking the audit trail. Whether to actually exercise that capability in any specific case is a question for legal counsel, not engineering. Engineering’s job is to make sure the choice is available.
This blog post is about building AI systems for regulated industries — healthcare, banking, insurance, and other places where “ship fast and iterate” gets you a subpoena.
The Air Canada Precedent
In February 2024, a man named Jake Moffatt asked an Air Canada chatbot about bereavement fares. The chatbot told him he could book a regular ticket and apply for a bereavement refund within 90 days. He did. Air Canada refused the refund, citing its actual policy, which the chatbot had got wrong. Moffatt took them to a small-claims tribunal in British Columbia. Air Canada argued, in essence, that it should not be liable for what its chatbot said — that the chatbot was a separate informational source, distinct from Air Canada itself. The tribunal disagreed. It ruled in favour of Moffatt and ordered Air Canada to honour what its chatbot had said. (Moffatt v. Air Canada, 2024 BCCRT 149)
The amount Moffatt was awarded was $812.02 Canadian. Legally, it was a small contract decision in one Canadian province — not a sweeping precedent on AI liability, no matter how it was reported. But as a signal of how courts and tribunals are starting to treat the output of AI systems, it is hard to ignore. A company saying “the chatbot did it, not us” is not a defence anyone wants to test in front of a regulator with broader powers.
Most AI commentary you’ll read online is written by, and for, people building things where the cost of being wrong is annoying. A chatbot gives a bad recipe. A coding assistant suggests a deprecated function. A marketing tool writes a weird subject line. The user shrugs, regenerates, and moves on. Air Canada’s mistake — and the reason it’s a useful starting point — is that it sat exactly on the boundary between annoying and legally consequential, and a tribunal decided which side of that boundary it was on. For about $1,000 and one customer.
Now, picture the same incident in a hospital. Or a bank’s payment system. Or a clinical trial recruitment platform. The boundary doesn’t exist. There is only the legally consequential side.
This series is for rooms where only the legally consequential side is present.
The Asymmetry
The defining feature of regulated AI is that the cost of being wrong is asymmetric.
Tens of thousands of correct outputs get you no upside. The system is supposed to work. Nobody throws a parade when a clinical decision support tool flags the right drug interaction or a payment-routing model correctly classifies a transaction. That’s the baseline. That’s why you bought the product.
One catastrophically wrong output, on the other hand, gets you front-page news, a regulator’s attention, and a board meeting nobody wants to attend. A clinical decision support system that recommends a contraindicated medication doesn’t just embarrass the vendor — it can harm a patient, trigger a reportable safety event, open a liability case, and require regulatory impact assessment or submission review. A KYC model that misclassifies a high-risk transaction in a CBUAE-regulated payment hub doesn’t just create a refund ticket — it can trigger a regulatory inquiry, a suspicious activity report, and a multi-million-dirham penalty. An underwriting model that produces disparate outcomes across protected classes doesn’t just lose customers — it invites a discrimination suit and a regulator’s audit of every other model on your shelf.
The asymmetry is structural. The downside dominates the expected value calculation in a way that no upside can offset. This changes everything about how the AI gets built. Not the model selection. Not the prompt engineering. Not the RAG architecture. Everything.
Regulator in the Room (Physics Constraints)
Five things change the moment your AI system enters a regulated industry. None of them is purely technical — but every one of them changes the architecture.
The regulator has veto power, regardless of market success. In consumer AI, the user is the customer; if they don’t like it, they leave. In regulated AI, the regulator sits behind the user with a different kind of power — not a vote with their wallet, but the authority to halt your product, mandate a recall, or refer your conduct for investigation. They have read your incident reports. They have read your vendor’s incident reports. They have a copy of your validation protocol, and they remember the version number. The user can love your product. The regulator can shut it down.
Documentation is the deliverable, not the overhead. A clean GitHub repo and a working demo are not a product in healthcare or banking. The product is the system (model) plus the evidence file, which includes the validation protocol, training data lineage, failure mode analysis, change control records, and post-market surveillance plan. In FDA-regulated MedTech, this is literally called the Design History File. In banking, it’s called Model Risk Management documentation under SR 11-7. The model is maybe a fifth of what you’re actually building. The rest is the case you’ll need to make to a regulator who has not yet decided to trust you.
Failure modes are first-class architectural concerns, not edge cases. When wrong answers can hurt people, “we’ll handle that in v1.1” is not an answer. The failure mode taxonomy gets defined before the happy path is built, not after. This is the IEC 62304 mindset — every software item gets a safety classification before a single line of code is written. You inherit the discipline whether or not you adopt the standard, because the alternative is discovering your safety class through litigation.
Auditability is non-negotiable. Every AI decision must be reconstructable, not just logged. The difference matters. A log says “the model returned X.” An audit trail says “the model returned X because it received inputs A, B, C; retrieved documents D, E, F from the knowledge base version dated Y; was running model checkpoint Z under prompt template version P; with these guardrails active; and here is the cryptographic evidence that none of this has been altered since.” If you can’t reconstruct it three years later when the case comes to court, you don’t have an audit trail. You have a hope dressed as a log file.
Change is governed, not continuous. The Silicon Valley default is “deploy ten times a day.” The regulated-industry default is “every change to a clinical algorithm requires impact analysis, validation, and possibly a regulatory submission.” When a foundation model vendor pushes a quiet weights update, that is not merely a feature update — depending on the intended use, the risk classification, and the impact on validated performance, it may constitute a regulated change requiring impact analysis, revalidation, and possibly submission review. Most AI vendor contracts don’t even tell you when this happens. That is a procurement problem dressed as a technical convenience.
These five constraints are not bugs to be optimised away. They are the physics of the environment. Trying to build regulated AI without internalising it is like trying to build a bridge without internalising gravity.
Disclaimer (in the middle)
A few things worth saying before going further.
This series is opinionated about the contexts where these patterns matter — production AI in healthcare, banking, insurance, and regulated MedTech, where wrong outputs reach real customers, patients, or transactions. It is not a claim that every AI system needs the full playbook. Internal research sandboxes, exploratory prototypes, and tools used by small numbers of trained domain experts in controlled conditions can reasonably operate with lighter scaffolding. The cost-benefit changes when the blast radius is bounded by scope rather than by architecture.
It is also not a substitute for jurisdiction-specific legal review. Regulatory regimes vary significantly by country, industry, and risk classification. The patterns in this series sit at a level of abstraction common across most regulated environments — but the specific obligations under FDA AI/ML guidance, EU AI Act, EU MDR, RBI circulars, CBUAE regulations, SR 11-7, GDPR, HIPAA, and their many cousins are not interchangeable, and any actual implementation needs counsel who specialises in your specific regime.
What this series is is a synthesis of architectural patterns that keep proving themselves across regulated environments — patterns that map well to most of the major frameworks, even where the specifics differ. Use them as starting points, not as legal cover.
Audience
If you are building AI inside a hospital system, a bank, an insurer, or a regulated MedTech firm, this series is for you. If you are an enterprise architect being asked to put guardrails around a foundation model that’s already in someone’s pilot, it is for you. If you are a CISO trying to figure out what your model risk surface looks like now that half your business units have wired in OpenAI, it is for you. If you are in regulatory affairs and you’ve just been told there’s a new AI feature in the next release and you need to figure out what that means for your submission package, it is especially for you.
If you are reading this and thinking, “We already deployed without most of this in place,” you are not alone. Most enterprises are past the greenfield-design moment. They are dealing with deployed systems, vendor lock-in, and audit questions arriving faster than the architecture can answer them. The retrofit playbook is real, and it is coming.
The shift you are navigating is this: the product is no longer the model. The product is the model plus the evidence that it behaved safely, consistently, and under control. Building for that requires a different set of architectural primitives than building a clever chatbot. The patterns below are drawn from more than two decades of building software in industries — clinical IT, healthcare service intelligence, regulated payment infrastructure — where being wrong is expensive in ways that matter. They have all earned their place by surviving contact with auditors, regulators, and the occasional lawyer.
Six Patterns for Regulated AI
The patterns themselves emerged from specific systems: clinical IT, payment infrastructure, MedTech architectures, and knowledge graphs for regulated workflows. Across those environments, six patterns kept reappearing as the difference between AI that ships and AI that survives. Each will get its own deep-dive post in this series, with concrete eat-this-not-that guidance. Here is the map.
Pattern 1 — Audit Trail
Every decision must be reconstructable, not just logged.
The minimum viable audit trail in regulated AI captures the inputs, the model version, the prompt template version, the retrieved context (with knowledge-base snapshot version), the active guardrails, the output, the human review action, if any, and a tamper-evident anchor — typically a hash chain or Merkle anchor written to an append-only ledger — that proves none of it has been altered. Three years from now, you must be able to answer: “Why did the system make this specific decision on this specific date for this specific patient or transaction?” — and back it with evidence.
Pattern 2 — Bounded Autonomy
Agents operate inside an architecturally enforced perimeter.
Most agentic AI demos give the agent the keys to the kingdom and trust the system prompt to behave responsibly. In regulated industries, this amounts to malpractice (a strong statement, apologies). Bounded autonomy means the agent has a hard-coded, externally enforced perimeter on its normal operation: which tools it can call, which datasets it can read, which actions it can take, which thresholds trigger mandatory human review, and what the maximum consequence (financial or clinical) of any single decision can be. The boundaries live in the architecture, not in the prompt.
A payment agent that could move ten million dollars but is architecturally limited to ten thousand without a second human approval is bounded autonomy. A payment agent that’s been told in its system prompt to be careful is a wish.
Pattern 3 — Human Review Quality
Review is a designed intervention, not a checkbox.
“Human-in-the-loop” has become the most abused phrase in regulated AI. It often means a tired clinician clicks “approve” on 200 AI recommendations a day without reading them, or an ops (maker/checker) analyst rubber-stamps fraud flags faster than the model produces them. That is not human-in-the-loop. That is human-as-rubber-stamp, and it is worse than no review because it manufactures a paper trail of false attention.
Human review done right specifies which decisions need review, what information the reviewer needs to make the decision well, how much time they need, what training they need to interpret the AI output, and how the system measures whether reviews are happening with cognitive engagement or in autopilot. If you don’t measure the quality of the review, you don’t have control, only a liability shield.
Pattern 4 — Evidence-Grade Evaluation
Evals built to clinical-trial standards, not sprint-demo standards.
The eval suite that gets your model into a board deck is not the eval suite that gets it past a regulator. Evidence-grade evaluation is structured the way clinical trials are structured: pre-registered protocols, defined endpoints, statistical power calculations, sub-group analysis (does it perform equally well across demographics, geographies, and edge cases?), failure mode classification, and a clear separation between development data and validation data with a documented chain of custody.
If your evaluation can be summarised as “we ran 500 test cases and got a 94% pass rate,” you do not have evidence.
Pattern 5 — Data & Model Lineage
Every output traceable to every artefact that shaped it.
When a regulator asks, “What data trained this model?” the right answer is not “publicly available text from the internet.” The right answer is a documented chain: training data sources with licensing information, fine-tuning datasets with version hashes, retrieval index snapshots with timestamps, prompt templates with version control, and guardrail configurations with effective dates. For every output the system produces, you should be able to walk backwards to every artefact that contributed to it.
This is also where vendor risk lives. If your foundation model vendor cannot tell you what their training data was, you have inherited their problem. In a regulated context, that may be unacceptable. This is why regulated industries are looking at smaller, sovereign, auditable models, even at a capability cost.
Pattern 6 — Failure Containment
Designed for graceful failure, not heroic prevention.
Bounded Autonomy is about the perimeter within which the system operates when things are normal. Failure Containment is about what happens when things are not normal — when the model is wrong, the inputs are adversarial, the data drifts, or the guardrails are bypassed. The two patterns sit on either side of the same coin.
Containment means the system has a defined behaviour when uncertainty exceeds a threshold (refuse, escalate, defer), hard limits on consequential actions (rate limits, value limits, irreversibility limits), detection mechanisms for known failure modes (drift, bias, hallucination, prompt injection), and rollback procedures that work fast — measured in minutes, not change-management cycles.
In MedTech, this is the FMEA mindset. In banking, it is the circuit breaker mindset. In both cases, the assumption is that the system will fail, and the engineering goal is to ensure that failures are detected, contained, and reversible before they become harmful.
Why Now?
Two years ago, the AI conversation in regulated industries was theoretical. Healthcare was watching. Banking was piloting. Insurance was modelling.
That has changed. The FDA now maintains a public list of authorised AI/ML-enabled medical devices that has grown into the many hundreds and continues to expand. Agentic payment and operations workflows are moving from controlled pilots toward supervised deployment in regulated banks. AI-assisted underwriting is being approved by insurance regulators, with conditions. The demos are becoming products. The products are becoming infrastructure. The infrastructure is now being audited.
And the playbook for how to do this safely, at scale, with evidence — that playbook is mostly being written behind NDAs, inside large enterprises, by teams who don’t have time to talk about it. The publicly available AI commentary continues to be dominated by use cases where the cost of being wrong is a refund, not a recall.
This series is an attempt to fill some of that gap. Not exhaustively — no series can — but with enough specificity. The bridge between AI demos and AI infrastructure runs through these six patterns. The teams that build the bridge will earn the right to ship AI into the systems that matter. The patterns are how you build the bridge.
The rest of this series will go deep on the six patterns — and close with the retrofit problem most enterprises eventually face:
Post 2 — The Audit Trail That Holds Up in Court. What to capture, how to anchor it, what tooling actually works, and the eat-this-not-that of audit architecture.
Post 3 — Bounded Autonomy: Building the Cage Before You Build the Agent. Architectural patterns for blast-radius control, with worked examples from payment workflow design.
Post 4 — Human Review, Without the Theatre. How to design review steps that survive a deposition.
Post 5 — Evals That Pass Regulators, Not Just Demos. Borrowing from clinical trial methodology to build evidence-grade evaluation pipelines.
Post 6 — Lineage as a First-Class Citizen. Tracking every artefact that shaped an AI output, from training data to the prompt version.
Post 7 — Designing for Failure Before You Design for Success. FMEA-thinking for AI systems, with a containment pattern catalogue.
Post 8 — When You Inherit the Problem. The retrofit playbook for AI systems already in production — vendor lock-in, missing lineage, contractual indemnities, and what to do when the business won’t let you turn it off.
Each post will be opinionated (sorry), specific, and prescriptive. Less “it depends,” more decision patterns, trade-offs, and concrete defaults. Vendor-agnostic by default. Just the patterns that have worked — and the ones that have failed — in the kinds of environments where being wrong has lawyers attached.
Data analysts look for descriptors in data to generate insights.
For a Data aggregator, descriptive attributes of data like size, speed, heterogeneity, lineage, provenance, and usefulness are essential to decide the storage infrastructure scale, data life cycle, and data quality. These aggregator-oriented descriptions are black-box perspectives.
For a Data analyst, descriptive statistics, patterns, and relationships are essential to generate actionable insights. These analyst-oriented descriptions are white-box perspectives. The analysts then use inferential methods to test various hypotheses.
Descriptive Statistics
Data analysts usually work with a significant sample of homogenous records to statistically analyze features. The typical descriptive statistics are – measures of location, measures of center, measures of skewness, and measures of spread.
E.g., A 23 member cricket team of three different states has players of the following ages:
Numbers represented this way does not help us detect patterns or explain the data. So, it’s typical to see the tabular distribution view:
AGE
Karnataka
Kerala
Maharashtra
19
2
2
6
20
3
3
5
21
4
3
4
22
5
4
3
23
4
5
2
24
3
6
2
25
2
0
1
Age Distribution of State Players
This distribution view is better. So, we would like to see measures of center for this data. These are usually – MEAN, MEDIAN, and MODE.
MEAN is the average (Sum Total / # Total)
MEDIAN is the middle number
MODE is the highest frequency number
Measure
Karnataka
Kerala
Maharashtra
MEAN
22
22.1
21
MEDIAN
22
22
21
MODE
22
24
19
Measures of Center
This description is much better. So, we would like to see this graphically to understand the skewness.
Measuring skewness
The age distribution is symmetrical for Karnataka, skewed to the left for Kerala, and skewed to the right for Maharashtra. The data analyst may infer that Karnataka prefers a good mix of ages, Kerala prefers player experience, and Maharashtra prefers the young.
The data analyst may also be interested in standard deviation, i.e., a measure of spread. The standard deviation symbol is sigma (σ) for a sample and is the MEAN distance from the mean value of all values in the sample. Since a distance can be positive or negative, the distance is squared, and the result is square-rooted.
Measure
Karnataka
Kerala
Maharashtra
Standard Deviation
1.8
1.7
1.8
Measure of Spread
In our example, a measure of location (quartiles, percentiles) is also of interest to the data analyst.
Percentile
Karnataka
Kerala
Maharashtra
25 Percentile
21
21
19.5
50 Percentile
22
22
21
75 Percentile
23
23.5
22
100 Percentile
25
24
25
Measure of Location
The table above shows that the 50 percentile value is the median, and the 100 percentile is the maximum value. This location measure is helpful if the values were scores (like in an exam).
Combining statistics and display to explain the data is the art of descriptive statistics. There are several statistics beyond the ones described in this short blog post that could be useful for data analysis.
Time-series Data Patterns
The time-series data has trends, variations and noise.
A trend is the general direction (up, down, flat) in data over time.
Cyclicity variation is the cyclic peaks and troughs in data over time.
Seasonality variation is the periodic predictability of a peak/trough in data.
Noise is meaningless information in data.
The diagrams below provide a visual explanation:
“Ice cream sales are trending upward,” claims an excited ice-cream salesman.
“Used Car value is trending downward,” warns the car salesman
“Every business has up and down cycles, but my business is trending upwards,” states a businessman.
“It’s the end of the month, so, Salary and EMI season in user accounts, so the transaction volume will be high,” claims the banker.
“There is some white noise in the data,” declared the data scientist.
Data Relationships
Data analysts seek to understand relationships between different features in a data set using statistical regression analysis.
There could be a causal (cause and effect) relationship or simply a correlation. This relationship analysis helps to build predictors.
A simple measure of linear relationship is the correlation coefficient. The measure is not relevant for non-linear relationships. Correlation coefficient of two variables x and y is is calculates as:
It’s a number that in the range [-1,1]. Any number closer to zero implies no correlation, and closer to either extremity means higher linear correlation.
Negative one (-1) means negatively linearly correlated
Positive one (1) means positively linearly correlated
Zero (0) means no correlation
Example: Let’s take this random sample.
X
Y
Y1
Y2
Y3
1
3
-3
83
-100
2
8
-8
108
250
3
15
-15
146
-50
4
24
-24
98
150
5
35
-35
231
-50
6
48
-48
220
155
7
63
-63
170
-125
8
80
-80
100
-150
9
99
-99
228
-12
10
120
-120
234
190
Sample Data
X and Y
X and Y1
X and Y2
X and Y3
1
-1
0.6
0
Correlation coefficient
Visually, we can see that as X increases, Y increases linearly, and Y1 decreases linearly. Hence, the correlation coefficient is positive (1) and negative (-1), respectively. There is no linear relation between X and Y3, and hence, the correlation is 0. The relationship between X and Y2 is somewhere in between with a positive correlation coefficient.
Scatter plot X against (Y, Y1, Y2, Y3)
If X is the number of hours bowler practices and Y2 is the number of wickets, then the correlation between the two can be considered positive.
If X is the number of hours bowler practices and Y3 is the audience popularity score, then the correlation between the two can be considered negligible.
If X is the number of years a leader leads a nation, and Y or Y1 is his popularity index, then the correlation between the two can be considered linearly increasing or decreasing, respectively.
Summary
Data analysts analyze data to generate insights. Insights could be about understanding the past or using the past to predict the (near) future. Using statistics and visualization, the data analysts describe the data and find relationships and patterns. These are then used to tell the story or take actions informed by data.
Digital Journey 