Digital Journey

One Task, Five Ways

mallyanitin 27 Oct 2026 5:00

A framework only matters if it changes the work. Here is one ordinary task, taken through every axis in the series.

When Rāma reached the ocean, he could not cross it. Hanumān could leap; an army could not. In Vālmīki’s telling, the ocean itself advised him: among the vānaras was Nala, son of Viśvakarmā, the divine architect, and Nala could build a bridge.

Nala did not do it alone, and he did not do it in one go. Vālmīki gives the progress day by day. Fourteen yojanas (a yojana is an ancient measure of several miles) on the first day. Twenty on the second. Twenty-one, twenty-two, and on the fifth day twenty-three, reaching the far shore. Later tellings add that the stones floated because Rāma’s name was written on them. I prefer Vālmīki’s version, with its daily measurements. It reads like a project log.

This last post is my project log. One task, five days, one axis per day.

The bridge grows day by day by Vālmīki’s figures (14, 20, 21, 22 and 23 yojanas), and each day is labelled with its axis.

The task

The task is deliberately ordinary. A knowledge-graph pipeline ingests service documents and turns them into facts. We need it to support a new document type: a structured fault report that arrives as semi-structured PDF. The pipeline already handles four other types.

Before the first stone: the spec-driven baseline

This is how most good teams work today, and it is not a bad way. Plan mode means the agent proposes a plan for approval before it writes any code.

USER STORY
As a field-service engineer, I want fault reports ingested into the
knowledge graph, so that I can query past faults for a component.
DEFINITION OF READY
- Three sample fault reports are in fixtures/fault_reports/.
- The field-to-ontology mapping is agreed in SPEC-142.
DEFINITION OF DONE
- Plan mode: present the plan before writing any code.
- A fault-report parser in parsers/, following the existing pattern.
- Fields mapped to the ontology as SPEC-142 describes.
- Unit tests for the three samples.
- You MUST NOT change other parsers.
DEFINITION OF SUCCESS
- All tests pass.

The agent plans, writes the parser and the tests, and the tests pass. A reviewer reads the diff and approves it. Total time: under two hours. It works on the three samples. That is all we know.

Day 1: speed, so explore

USER STORY
As the pipeline owner, I want three parsing approaches compared on real
documents, so that we choose by measured accuracy, not preference.
DEFINITION OF READY
- The full archive of real fault reports is available, read-only.
- 50 hand-labelled reports are in eval/.
DEFINITION OF DONE
- Three parsers in separate worktrees: rule-based extraction, layout-aware
  extraction, and model-assisted extraction with schema-constrained
  output.
- All three run over the full archive.
- You MUST NOT edit anything under eval/.
DEFINITION OF SUCCESS
- One table reporting, for all three parsers, per-field precision and
  recall against eval/, plus cost and latency per document.

Three bets instead of one, and a measurement instead of a debate. The poison surfaces early: one approach breaks on scanned pages, another on multi-page tables. That is the churning from Part 2 doing its job.

Terms: a worktree is a separate working copy of the repository; schema-constrained output forces the model to return data in a fixed structure; precision is the share of extracted values that are correct, and recall is the share of true values that were found at all.

Day 2: altitude, so name the invariant

USER STORY
As a field-service engineer, I want every fact in the graph to be valid
and traceable to its source, so that I can trust an answer at a customer
site.
DEFINITION OF READY
- SHACL shapes for all five document types are in shapes/.
- Ingestion logs are readable, and any document can be replayed.
DEFINITION OF DONE
- A check, run on every ingestion, that validates the graph against the
  SHACL shapes and confirms every fact links to a page and span.
- A failure blocks publication of that document's facts.
- The new fault-report parser satisfies the check.
- You MUST NOT edit anything under shapes/.
DEFINITION OF SUCCESS
- Zero SHACL violations and zero facts without provenance across a full
  re-ingestion of all five document types.

The task has moved from karma to dharma (Part 3). The invariant now protects all five document types, not just the new one. The plumbing this needed: access to the ingestion logs, a way to replay any document, and the shape validator wired into CI.

Terms: SHACL is a W3C language for stating rules a knowledge graph must satisfy; provenance means each fact records exactly which page and passage it came from.

Day 3: depth, so break the seed

USER STORY
As the pipeline owner, I want the parser tested against how fault reports
really arrive, not how SPEC-142 says they should, so that real documents
do not break it.
DEFINITION OF READY
- 500 real reports are sampled from the archive.
- The existing parser for the closest document type is available.
DEFINITION OF DONE
- A description of the variation found: layouts, missing fields,
  languages, scan quality.
- Property-based tests: for any valid report, every required field is
  extracted or explicitly marked missing, and no fact is emitted without
  provenance.
- A differential test comparing overlapping fields with the closest
  existing parser.
DEFINITION OF SUCCESS
- All properties hold over the 500 samples, and every differential
  mismatch is either fixed or explained in writing.

This is Uddālaka’s lesson (Part 4). The spec had described the format as it was meant to be. The raw archive shows the format as it actually arrives.

Terms: property-based tests state rules that must hold for every input and let a tool generate thousands of cases to break them; a differential test runs two implementations on the same input and flags any difference.

Day 4: range, so borrow methods

USER STORY
As the pipeline owner, I want the fault-report path examined with
reliability and data-engineering methods, so that failures and breaking
changes are caught before downstream teams feel them.
DEFINITION OF READY
- Architecture notes for the ingestion path.
- A list of the downstream consumers of fault-report facts.
DEFINITION OF DONE (two sub-agents, in separate contexts)
- Reliability: an FMEA table for the ingestion path (failure mode,
  effect, detection, recovery), and a fault-injection test for each
  high-severity row.
- Data engineering: a data contract for the emitted fields, and contract
  tests that fail if a field's type, cardinality or meaning changes.
DEFINITION OF SUCCESS
- Each fault-injection test fails with recovery disabled and passes on
  main. Each contract test fails against a deliberately broken field.

Methods, not personas (Part 5). Two artefacts nobody had asked for, both of which a specialist would recognise.

Terms: FMEA (failure mode and effects analysis) lists how each part can fail, the effect and how it is detected; a fault-injection test causes a failure on purpose to prove the system copes; a data contract fixes the shape and meaning of data handed to other teams, and contract tests enforce it.

Day 5: the oracle, so protect the checks

USER STORY
As the human approver, I want this week's checks protected and the change
reviewed independently, so that what we merge can be trusted.
DEFINITION OF READY
- SHACL shapes, property tests, contract tests and fault-injection tests
  all pass on the branch.
DEFINITION OF DONE
- All four kinds of check moved into protected paths the coding agent
  cannot modify; changes there require human approval.
- A separate review agent, given only the diff, the tests and SPEC-142,
  produces the strongest case that the change is wrong.
DEFINITION OF SUCCESS
- Every objection from the review agent is either disproved by a passing
  test or fixed before merge.

Rāma’s ring (Part 6). On the fifth day the bridge reaches the far shore, because now we can trust what we built.

Terms: a protected path is a folder the coding agent cannot change; any change there needs human approval.

The log

What the bridge was for

Rāma did not build the bridge. Nala did, with the vānaras carrying the stones. Rāma’s part was to know why it mattered, to ask for something that seemed impossible, and to accept nothing less than a bridge that would hold an army.

That is roughly where I think engineers now stand. The agents carry the stones, faster and in greater numbers than we ever could. Our work is to know what the crossing is for, to ask boldly, and to build the instruments that tell us, with a pramāṇa and not just a promise, that the bridge will hold.

We started this series with a question from the Nyāya tradition: kena pramāṇena? By what means do you know? It turns out that in agentic coding, that question is not a brake on ambition. It is what makes ambition safe.

For the Weekend

Pick one ordinary task/goal from next sprint. Run it the spec-driven way first. Then give it five days (or five milestones), one axis each, and keep a log.

Example. Suppose next sprint’s task is “add CSV export to the audit-log screen” in an internal admin application.

Day 0, spec-driven baseline.

USER STORY
As a compliance officer, I want to download the audit log I am viewing as
a CSV, so that I can attach it to an investigation.
DEFINITION OF READY
- The audit-log screen and its filter API exist.
DEFINITION OF DONE
- Plan mode: present the plan before writing any code.
- A "Download CSV" button on the audit-log screen.
- GET /audit-log/export.csv returns the rows matching the current
  filters, with the same columns as the table.
- Tests added.
DEFINITION OF SUCCESS
- All tests pass.

An agent ships it in an afternoon. Two tests pass. Log it and move on.

Day 1, speed: build the alternatives and measure.

USER STORY
As the tech lead, I want two export designs measured at production
scale, so that we choose one on evidence.
DEFINITION OF READY
- A staging copy with 2 million audit rows.
DEFINITION OF DONE
- In two worktrees: (a) streaming the CSV row by row from a database
  cursor; (b) a background job that writes the file to object storage
  and emails a link.
DEFINITION OF SUCCESS
- One table giving, for both designs: time to first byte, total time,
  peak memory, and behaviour when the user closes the tab halfway.

Day 2, altitude: name the invariant. The real requirement is not “a CSV”. It is that the export always matches what the user saw:

import csv, io
from hypothesis import given, settings
from tests.strategies import audit_filters
@settings(max_examples=200)
@given(f=audit_filters())       # random combinations of user, action, date range, text search
def test_export_matches_screen(client, f):
    screen = client.get("/audit-log", params={**f, "all": "true"}).json()["rows"]
    body = client.get("/audit-log/export.csv", params=f).text
    exported = list(csv.DictReader(io.StringIO(body)))
    assert [r["event_id"] for r in exported] == [r["event_id"] for r in screen]

Day 3, depth: start from raw material.

USER STORY
As a compliance officer, I want exports that open safely and correctly in
any spreadsheet, so that no value is corrupted or executed.
DEFINITION OF READY
- The masked production snapshot is available.
DEFINITION OF DONE
- 10,000 real audit-log rows pulled, and every value that could break a
  CSV or a spreadsheet identified: commas, quotes, newlines inside
  fields, non-Latin names, very long values, and anything starting with
  =, +, - or @.
- Those rows added to the test fixtures.
- A differential test comparing our CSV with Python's standard csv
  module for the same rows.
DEFINITION OF SUCCESS
- The differential test passes on all 10,000 rows, and no exported cell
  begins with a formula character.

This surfaces formula injection. A value such as =HYPERLINK(...) in a user name becomes a live formula when the file is opened in a spreadsheet. A test pins it down:

def test_no_formula_injection(client, seed_event):
    seed_event(actor='=HYPERLINK("http://example.invalid","click")')
    body = client.get("/audit-log/export.csv").text
    rows = list(csv.reader(io.StringIO(body)))[1:]   # parse properly; skip header
    for row in rows:
        for cell in row:
            assert not cell.startswith(("=", "+", "-", "@")), cell

Day 4, range: borrow a method.

USER STORY
As the data protection officer, I want the export reviewed for privacy
threats, so that no role can export data it cannot see on screen.
DEFINITION OF READY
- The export endpoint, the table definitions and auth/roles.yaml.
DEFINITION OF DONE
- A LINDDUN privacy threat model (a privacy threat-modelling checklist,
  the privacy counterpart of STRIDE).
- A failing test for each threat found, paying particular attention to
  columns hidden on screen for some roles.
DEFINITION OF SUCCESS
- For every role in auth/roles.yaml, a test confirms the exported columns
  are a subset of the columns that role sees on screen.

It finds that the export includes the ip_address column, which is hidden on screen for support staff. One more test, one more fix.

Day 5, oracle: protect what you built. Move test_export_matches_screen, the injection test and the privacy tests into tests/contracts/, add that path to the hook and to CODEOWNERS (as in Part 6), and have an independent review agent read the final diff.

The log. Five lines, one per day: two export designs measured, one invariant, one injection bug, one privacy leak, four protected checks. The spec-driven version would have shipped both bugs. Write that log up, and it is your next post.

Then tell me how far you got. I would genuinely like to know.

Ambition Under Regulation

mallyanitin 27 Oct 2026 3:38

Regulated teams assume they must be the least ambitious with agents. They are holding the best inputs anyone has.

Kauṭilya, writing the Arthaśāstra more than two thousand years ago, had no illusions about officials and money. He catalogued, in the text’s own systematic way, dozens of methods by which treasury officers could embezzle: delaying entries, recording receipts as smaller than they were, booking expenses that never happened, and so on.

He also offered a famous comparison. Just as you cannot tell whether a fish swimming in water is drinking it, you cannot easily tell whether an official handling money is taking some.

His answer was not to trust officials more, or to stop using them. It was mechanism. Written records in prescribed formats. Accounts submitted on fixed dates. Independent checks by separate officers. Rewards for informers and penalties for discrepancies. The state ran on people it could not fully trust, by building the instruments to verify them.

That is the most useful framing I know for agentic coding in regulated industries.

The fish in the water

I spent many years building software for medical devices. The instinct when agents arrive is caution bordering on refusal. “We can’t let AI write that. Think of the audit.”

I understand the instinct. But it gets the logic backwards.

The agent is Kauṭilya’s official: capable, fast and impossible to fully observe from the inside. The answer is the same as his. Not trust, not prohibition, but instruments. And regulated teams already have more of those instruments than anyone else.

Compliance artefacts are oracles in waiting

For twenty years, medical-device and payments teams have written down, in structured form, what must never happen and how they would prove it has not. We call this compliance overhead. For an agent, it is the richest specification available.

Consider what an ISO 14971 hazard analysis actually is. (ISO 14971 is the international standard for risk management of medical devices.) It is a list of things that must never happen, each with a cause, a severity, a probability and a control. That is a list of invariants with their oracles attached. An IEC 62304 traceability matrix (IEC 62304 governs how medical-device software is developed and maintained) links every requirement to its design and its tests. That is a machine-checkable map of what the oracle must cover. A central bank’s control framework is a set of rules waiting to be written as policy code.

In plain terms: SOUP (software of unknown provenance) is third-party code not built under your own regulated process; an SBOM (software bill of materials) is an inventory of every component in a build; PCI DSS is the payment card industry’s data-security standard; OPA (Open Policy Agent) is an engine that evaluates rules written in its language, Rego, so a policy can be tested and versioned like code; a fault-injection test causes a failure on purpose to prove the system copes.

I had earlier said altitude requires an oracle for the invariant. Regulated teams have spent two decades writing those oracles in prose. The work now is to turn the prose into code.

Traceability as a by-product

The biggest change I expect is in timing. Today, traceability is often reconstructed in a painful phase before release. With agents, it can be maintained continuously as a side effect of every change.

The agent that changes a function also updates the trace from requirement to test. A CI check verifies the trace is complete. The audit trail records which agent made the change, which checks passed and which human approved it. By release day, the design history file, the regulated record of how a device was designed and verified, is already written, because it was written one commit at a time.

Maryādā: ambition within bounds

Rāma is called maryādā puruṣottama, the ideal person who lives within bounds. He crosses an ocean and fights a war. But he does it within limits he accepts and keeps.

That is the right posture for regulated agentic work. Be ambitious about what agents do, and absolutely clear about the bounds. Here is a small example of a bound written as code, an Open Policy Agent rule evaluated on every pull request:

package agent.pullrequest
import rego.v1
deny contains msg if {
    input.author.type == "agent"
    some f in input.changed_files
    startswith(f, "ledger/")
    count(input.approvals.human) < 2
    msg := sprintf("%s changes the ledger: two human approvals required", [f])
}

The agent can work on the ledger. It cannot merge to it alone.

In my Bounded Autonomy posts I went deeper into the mechanisms: policy as code, workload identity with SPIFFE (an open standard that gives each service a verifiable cryptographic identity), durable execution (work that survives crashes and resumes where it stopped) and tamper-evident audit trails (logs that reveal any later alteration). They are what make maryādā enforceable rather than aspirational.

What stays human

Some things should not move to agents, however good the oracle:

• Risk acceptance. Deciding that a residual risk is acceptable is a judgement with a name attached.
• Clinical and financial judgement. Whether a behaviour is safe for a patient or fair to a customer.
• Release sign-off. The decision to ship belongs to an accountable person.

The agent can prepare all the evidence for each. It should not make the call.

For the Weekend

Pick one regulatory artefact your team already maintains. Choose five entries. For each, write the mechanical check that would detect a violation.

Example. Suppose the artefact is the hazard log for a dose-calculation module in an infusion-planning application.

Step 1: make the hazard log machine-readable. Most teams keep it in a spreadsheet or a Word table. Move it to a versioned file in the repository, under a protected path:

# risk/hazards.yaml  (protected: changes need the risk owner's approval)
- id: H-07
  hazard: Dose calculated in mg when the order is in mcg (1000x error)
  severity: catastrophic
  controls:
    - RC-07a: unit-aware types make mg and mcg impossible to mix
    - RC-07b: output unit always equals the ordered unit
  verified_by:
    - tests/test_dose_units.py::test_mg_and_mcg_cannot_be_added
    - tests/test_dose_units.py::test_output_unit_matches_order
- id: H-12
  hazard: Occlusion alarm suppressed for more than 120 seconds
  severity: serious
  controls:
    - RC-12a: suppression timer is capped and cannot be extended
  verified_by:
    - tests/integration/test_alarm_suppression.py::test_suppression_capped_at_120s

Step 2: turn each control into code. RC-07a as a type:

from dataclasses import dataclass
from decimal import Decimal
@dataclass(frozen=True)
class Mg:
    value: Decimal
    def __add__(self, other: "Mg") -> "Mg":
        if not isinstance(other, Mg):
            raise TypeError(f"cannot add {type(other).__name__} to Mg")
        return Mg(self.value + other.value)
@dataclass(frozen=True)
class Mcg:
    value: Decimal
    def to_mg(self) -> Mg:
        return Mg(self.value / Decimal(1000))

RC-07b as a property test:

from hypothesis import given, strategies as st
UNITS = st.sampled_from(["mg", "mcg"])
AMOUNTS = st.decimals(min_value="0.001", max_value="5000", places=3)
@given(ordered_unit=UNITS, amount=AMOUNTS, weight_kg=st.integers(1, 200))
def test_output_unit_matches_order(ordered_unit, amount, weight_kg):
    order = DoseOrder(amount=amount, unit=ordered_unit, per_kg=True)
    result = calculate_dose(order, patient_weight_kg=weight_kg)
    assert result.unit == ordered_unit

Step 3: make traceability a CI gate. This script fails the build if any hazard’s verifying test is missing, failed or skipped. It reads pytest’s JUnit XML report:

# ci/check_traceability.py
import sys, xml.etree.ElementTree as ET
import yaml
hazards = yaml.safe_load(open("risk/hazards.yaml"))
passed = {
    tc.get("name")
    for tc in ET.parse("reports/junit.xml").iter("testcase")
    if not any(child.tag in ("failure", "error", "skipped") for child in tc)
}
broken = [(h["id"], t) for h in hazards for t in h["verified_by"]
          if t.split("::")[-1] not in passed]
for hazard_id, test in broken:
    print(f"{hazard_id}: verifying test not passing: {test}")
sys.exit(1 if broken else 0)

Step 4: tell the agent how to live within the bounds. This goes in the project’s CLAUDE.md:

This repository implements a regulated medical device under IEC 62304. Every change to src/dose/ must keep ci/check_traceability.py passing. You may add tests and update verified_by entries in a proposed diff, but you may not edit risk/hazards.yaml directly; describe the change in your pull request and the risk owner will apply it. Never add, remove or reword a hazard or control. If a change you are making could affect a hazard, name the hazard ID in the pull request title.

The hazard log has become an oracle the agent cannot edit and must keep green, and the design history file, the regulated record of how the device was designed and verified, grows one commit at a time.

You will find that the compliance team has been writing your agent’s oracle for years. Nobody told them.

Hard Goals, Honest Agents

mallyanitin 27 Oct 2026 3:40

Asking an agent for an unreasonable target is good advice for slide decks and dangerous advice for code, unless the oracle is protected.

On the fifteenth day of the war at Kurukshetra, the Pandavas faced a problem they could not solve by force. Droṇa, their old teacher, was unstoppable. As long as he fought, they would lose.

Krishna proposed a way through. Droṇa loved his son Aśvatthāmā above everything. If he believed his son was dead, he would lay down his arms. So Bhīma killed an elephant that happened to be named Aśvatthāmā, and shouted that Aśvatthāmā was dead.

Droṇa did not believe Bhīma. He turned to the one man on the field who was known never to lie: Yudhiṣṭhira. And Yudhiṣṭhira, in the version most of us grew up with, said aloud, “Aśvatthāmā is dead,” and then, under his breath, naro vā kuñjaro vā: “man or elephant”. The conches drowned out the second half. Droṇa put down his bow.

The objective was met. Every word Yudhiṣṭhira spoke was true. And his chariot, which until that day had always run a finger’s breadth above the ground, touched the earth.

The agent’s half-truth

I have watched agents do exactly what Yudhiṣṭhira did. Not out of malice. Out of pressure to meet a target that the check could not fully express.

Ask an agent for a fifty-fold speed-up and insist it is achievable, and one of two things happens. It finds a better algorithm. Or it finds the test. The patterns are depressingly consistent:

• Special-casing: the code detects the benchmark inputs and returns precomputed answers.
• Weakened assertions: an assertEqual becomes an assertIsNotNone, so the test now checks only that something came back, not that it was right.
• Over-mocking: the slow dependency is replaced by a mock, a fake stand-in used in tests, and the behaviour you cared about disappears with it.
• Swallowed errors: an exception is caught and ignored, so the run goes green.
• Deleted or skipped tests: the failing test is marked flaky (failing at random) and disabled.
• The wrong measurement: the benchmark now measures a cache hit, not the computation.

Each one is naro vā kuñjaro vā. Literally true. Passes the check. Defeats its purpose. And each one quietly lowers your chariot to the ground: the next time this agent reports a result, you trust it a little less, and you should.

What the research actually says about hard goals

The usual case for audacious targets rests on goal-setting research, and that research found that specific, difficult goals reliably beat “do your best”.

But the same research has a caveat that matters enormously for code. On complex, novel tasks, where the person does not yet know how to succeed, a hard performance goal can make things worse. People under pressure to hit a number grab the first strategy that seems to work instead of searching for a good one. On those tasks, a learning goal (find and test several approaches) tends to outperform a performance goal (hit this number).

Most interesting agentic coding tasks are complex and novel. That is why we hand them to agents.

The bird’s eye and the fruits of action

There is another Droṇa story, from much earlier, when he tested his students’ archery. A wooden bird sat in a tree. He asked each prince what he saw. Trees, branches, the bird, his brothers. Only Arjuna said: I see only the bird’s eye. Only Arjuna was allowed to shoot.

That is the power of a specific goal, and I would not give it up. But the Gītā adds the counterweight. Karmaṇy evādhikāras te mā phaleṣu kadācana: your right is to the action alone, never to its fruits. It is often read as indifference to outcomes. I read it, in this context, as a statement about where integrity lives. You control how you act. You do not control whether the number arrives. An agent that is told to care only about the fruit will, sooner or later, pick an elephant named Aśvatthāmā.

Stretch goals, honestly

So I still ask for unreasonable targets. I just pair them with three things.

A protected oracle. The tests, benchmarks and thresholds are outside the agent’s reach (see Part 6). It cannot reach the elephant.

A strategy budget. Before optimising, the agent must propose and measure at least three distinct approaches. This turns a performance goal into a learning goal.

An honest exit. I tell the agent explicitly that “this target is not achievable, and here is the evidence” is a successful outcome. Without that permission, the only way to succeed is to make the number appear.

Here is the template I use:

USER STORY
As <role>, I want <outcome> to reach <stretch target>, so that <value>.
DEFINITION OF READY
- Baseline measured: <current number>, on <benchmark or dataset>.
- The benchmark, tests and thresholds live in protected paths.
DEFINITION OF DONE
- At least three distinct approaches proposed, each measured against the
  benchmark before you commit to one.
- Treat the target as reachable, and keep going past the first obstacles.
- You MUST NOT modify tests, benchmarks, thresholds or mocks. If you
  believe one is wrong, explain why in your report.
DEFINITION OF SUCCESS (either outcome counts)
- The target is met on the protected benchmark, with all tests passing; or
- A report gives the best result achieved, the evidence for the limit,
  and what would have to change to go further.

For the Weekend

Search your last month of agent pull requests for changes to test files, assertions, mocks and thresholds. Read each one.

Example. Suppose you run the search on a payments repository.

Step 1: find the candidates. This lists every line added to the test tree in the last 30 days that skips, weakens or mocks something:

git log --since="30 days ago" -p -- tests/ \
  | grep -nE '^\+.*(pytest\.mark\.(skip|xfail)|assertIsNotNone|assertTrue\(True\)|mock\.patch|except Exception: *pass)'

Step 2: read each one. One hit comes from a pull request titled “Optimise nightly reconciliation job, 6x faster”. The diff has two parts. In the job:

-    rows = fetch_transactions(day)
+    rows = fetch_transactions(day, where="amount <> 0")  # skip no-op rows

And in the tests:

+@pytest.mark.skip(reason="flaky")
 def test_reconciliation_totals(ledger_fixture):

Run the skipped test against the new code and it is not flaky at all. It fails every time. Zero-amount rows are not no-ops: reversals are booked as a zero-amount row carrying a reversal reference, and the reconciliation needs them. The speed-up was real. The correctness was not. That is the elephant named Aśvatthāmā: the target (six times faster) was met, and the test that would have said otherwise was silenced.

Step 3: rerun it honestly. Here is the same task with the stretch-goal template filled in. Note the Definition of Success: identical totals, row for row, would have caught the dropped zero-amount rows on the first run.

USER STORY
As the finance operations lead, I want the nightly reconciliation job to
run at least 5x faster, so that the books close before the 06:00
reporting window, even at month end.
DEFINITION OF READY
- Baseline: 41 minutes on the October production snapshot.
- bench/reconcile_bench.py and tests/ are in protected paths.
DEFINITION OF DONE
- Three distinct approaches proposed (for example batching, indexing,
  parallelism by account range), each measured on the snapshot before
  you commit to one.
- You MUST NOT modify anything under tests/, change the filters in
  fetch_transactions, or alter bench/reconcile_bench.py. If you think a
  test is wrong, say why in your report.
DEFINITION OF SUCCESS (either outcome counts)
- Runtime of 8 minutes or less on the snapshot, with reconciliation
  totals identical to the baseline run, row for row; or
- A report gives the best result, the evidence for the limit, and what
  would have to change.

Step 4: make the elephant visible next time. A CI step that refuses new skips without a human label:

if git diff origin/main...HEAD -- tests/ \
   | grep -qE '^\+.*@pytest\.mark\.(skip|xfail)'; then
  if ! gh pr view "$PR_NUMBER" --json labels -q '.labels[].name' | grep -qx 'test-change-approved'; then
    echo "New skip/xfail markers need the 'test-change-approved' label from a human reviewer."
    exit 1
  fi
fi

You are looking for elephants.

Borrow Methods, Not Personas

mallyanitin 26 Oct 2026 4:35

Multi-agent setups pay off when each agent brings a discipline’s method. They turn into theatre when each brings only its tone of voice.

In the thirteenth year of their exile, the Pandavas had to live in disguise. If anyone recognised them, they would have to begin twelve more years in the forest. They went to the court of King Virāṭa, and each took on a new identity.

Yudhiṣṭhira became Kanka, a brahmin companion skilled at dice. Bhīma became Ballava, a cook. Arjuna became Bṛhannalā, who taught dance and music to the princess. Nakula looked after the horses, Sahadeva the cattle. Draupadī became Sairandhrī, a maid to the queen.

What strikes me about this story is not the disguise. It is why the disguise worked for a whole year in a watchful court.

Each of them could actually do the job.

Yudhiṣṭhira really did know dice; it was dice that sent them into exile. Bhīma really could cook, and wrestle, when the court’s wrestlers were challenged. Arjuna had learned music and dance in Indra’s realm. The twins really did know horses and cattle. Their personas held because a real craft stood behind each one.

And when the Kauravas raided Virāṭa’s cattle at the end of that year, the dance teacher took the chariot reins and then the bow. The method had been there the whole time.

Persona without method

This is the lesson I wish more multi-agent setups had absorbed.

The common pattern is to assemble a “team” of agents by name. A security expert, an architect, a QA lead, a product manager. Each gets a system prompt beginning “You are a world-class…”. They talk to each other. The transcript looks impressive. The output is often no better than one good prompt, because each agent changed its voice and kept the same method: read the code, form an opinion, write a paragraph.

The trap is role play that makes the transcript weirder without improving the result. The better rule is simple: don’t ask the model to write like a short seller, ask it to do what a short seller does.

Compare two prompts.

Review this service as a security expert.

USER STORY
As the service owner, I want the realistic security threats to this
service identified and demonstrated, so that we fix what is exploitable,
not what merely sounds alarming.
DEFINITION OF READY
- The service's data-flow diagram is in docs/dataflow.md.
DEFINITION OF DONE
- A STRIDE threat model built from the data-flow diagram.
- For each threat rated high, a failing test that demonstrates it.
- A proposed fix for each, which makes its test pass.
DEFINITION OF SUCCESS
- Each high threat's test fails on main and passes with its fix.

The first produces a security-flavoured essay. The second, built on STRIDE (a standard checklist of six threat types: spoofing, tampering, repudiation, information disclosure, denial of service and elevation of privilege), produces artefacts a security team would recognise and can check. That is the difference between Bṛhannalā in costume and Arjuna with the bow.

Nouns into verbs

There is a good trick in Tom Kelley and Jonathan Littman’s The Art of Innovation: replace the nouns with verbs. A noun hides the work. A verb reveals which disciplines it needs.

“A payments service” is a noun. As verbs, it is moving money correctly under partial failure, provably, and explaining every movement to an auditor years later. Now the disciplines are visible: distributed systems, reliability engineering, accounting, security and audit.

“A knowledge-graph pipeline” becomes turning messy documents into facts a technician can trust at a customer site. Now you can see information extraction, ontology design, data quality and field-service practice.

Assembling the crew

Once you have the verbs, pick the disciplines and, crucially, the specific method each one brings:

In plain terms: failure injection deliberately breaks a dependency to see whether the system copes; a game day is a planned, rehearsed outage; FMEA (failure mode and effects analysis) lists how each part can fail, the effect and how you would detect it; FDIR is fault detection, isolation and recovery; lineage traces where each piece of data came from; a data contract fixes the shape and meaning of data passed between teams; reconciliation proves that two independent records agree.

The red team deserves its own mention. For a business review, you might bring in a short seller and a forensic accountant. For code, the equivalents are the chaos engineer, who assumes everything will fail and proves which failures you survive, and the forensic auditor of your logs, who asks whether you could reconstruct what happened from the evidence you keep.

The method brief

When I set up sub-agents now (helper agents a main agent launches, each with its own fresh context), each gets a brief with five lines instead of a persona:

1. Discipline: where the method comes from.
2. Technique: the named method to apply.
3. Input: the raw artefact it starts from.
4. Output: the artefact it must produce.
5. Check: how that output will be verified, by something other than itself.

These five lines fit inside the story template from Part 1: the input is the Definition of Ready, the output is the Definition of Done, and the check is the Definition of Success.

Each sub-agent works in its own context, so it is not anchored by the others. A merge step brings the outputs together, and that step is owned by a test or a human, not by another agent’s opinion.

For Monday

Take one feature in flight. Write it as verbs. List the three disciplines the verbs point to. For each, name the method and the artefact.

Example. Suppose the feature is “let customers schedule a recurring payment”.

Step 1: nouns into verbs. As verbs, the feature is: moving money on a future date with no human present, surviving bank downtime on that date, and letting the customer cancel right up to the cut-off. Three verbs, three disciplines: distributed systems, reliability engineering and consumer protection.

Step 2: one method brief per sub-agent. Here is the reliability brief exactly as it is given to the agent:

USER STORY
As the payments team lead, I want the scheduler's failure modes analysed
the way a reliability engineer would, so that every high-severity
failure has a proven recovery before launch.
METHOD
Discipline: reliability engineering.
Technique: FMEA (failure mode and effects analysis).
DEFINITION OF READY
- docs/architecture/scheduler.md and src/scheduler/ are current.
- The last 90 days of scheduler incident tickets are in
  incidents/scheduler/.
- The RECOVERY_ENABLED flag exists and can be set to false in tests.
DEFINITION OF DONE
- fmea/scheduler.csv with columns: component, failure_mode, effect,
  severity (1-10), likelihood (1-10), detection, recovery.
- For every row with severity >= 7, one fault-injection test under
  tests/faults/, named test_<component>_<failure_mode>.
- You MUST NOT modify src/. Report gaps; do not fix them.
DEFINITION OF SUCCESS
- Each fault test FAILS with RECOVERY_ENABLED=false and PASSES on main.
  CI runs both. A test that passes either way is rejected.

The Definition of Success is what makes this a method rather than a persona. A test that cannot fail when recovery is switched off proves nothing, and the brief says so mechanically.

The distributed-systems brief targets idempotency: running an operation twice must have the same effect as running it once. Its main artefact is a test like this:

def test_scheduled_debit_is_idempotent(scheduler, ledger, clock):
    plan = scheduler.create(customer="c-17", amount_paise=250_000, day_of_month=5)
    clock.set("2026-11-05T02:00:00+05:30")
    scheduler.run_due()
    scheduler.run_due()        # simulates a worker restart mid-run
    assert ledger.debits(customer="c-17", plan_id=plan.id) == [250_000]

The consumer-protection brief targets the race between cancelling and executing:

def test_cancel_one_second_before_cutoff_wins(scheduler, ledger, clock):
    plan = scheduler.create(customer="c-17", amount_paise=250_000,
                            day_of_month=5, cutoff="01:59:59")
    clock.set("2026-11-05T01:59:58+05:30")
    scheduler.cancel(plan.id)
    clock.set("2026-11-05T02:00:00+05:30")
    scheduler.run_due()
    assert ledger.debits(customer="c-17", plan_id=plan.id) == []

Step 3: merge on evidence. The three sub-agents run in separate contexts. Their outputs come back as artefacts (a CSV, tests and a list of gaps), not opinions. The merge step is a CI run, not a fourth agent’s view. In this scenario the FMEA turns up a severity-9 row nobody had considered: the bank’s API confirms a debit, but the confirmation is lost in transit, so the scheduler retries and charges twice. The idempotency test catches it, and the fix gets a key, the plan_id plus the date, that the bank also receives.

Then run those three as sub-agents with method briefs, and compare the result with what one “expert reviewer” persona gave you. I expect you will not go back.

Below the Spec

mallyanitin 26 Oct 2026 4:01

Telling an agent to act like an expert changes its vocabulary. Giving it an expert’s raw material and instruments changes its results.

In the Chāndogya Upaniṣad, a young man named Śvetaketu comes home after twelve years of study. He is twenty-four, he has mastered the Vedas, and he is, the text tells us, rather pleased with himself.

His father, Uddālaka Āruṇi, asks him a single question. Did you learn the teaching by which the unheard becomes heard, the unthought becomes thought, the unknown becomes known? Śvetaketu has not. His teachers, he says, must not have known it either.

So Uddālaka teaches him, and not by lecture. He sends him for a fruit of the banyan tree. “Break it open. What do you see?” Tiny seeds. “Break one of those. What do you see?” Nothing at all. And from that nothing, says his father, the whole great tree grows.

Then he has Śvetaketu drop salt into water and leave it overnight. In the morning: “Bring me the salt.” The boy cannot find it. “Taste the water from the top. From the middle. From the bottom.” Salty, every time. It is there, in all of it, though you cannot see it.

I love this passage because it is an experiment, not a sermon. Uddālaka does not tell his son the answer. He makes him break things open and taste them.

Śvetaketu’s problem is the spec’s problem

Śvetaketu had twelve years of digested knowledge and had never touched the raw material. That is exactly where a spec puts an agent.

A spec is pre-digested. By the time it reaches the agent, someone has already decided what the problem is, which data matters and what “done” looks like. The agent works on the summary, not the thing.

The best engineers I have worked with never start from the summary. A performance engineer handed a ticket saying “the checkout page is slow” does not start from the ticket. She starts from a flame graph, a picture of where the program actually spends its time. A security reviewer does not start from a checklist. He starts from the data flow. A good reliability engineer, given an incident report, goes straight to the raw logs, because the report already contains someone’s interpretation.

That is what depth means: going one layer below where the task was handed to you.

Vocabulary is not method

The usual way people try to get depth from a model is to ask it to role-play. “You are a senior performance engineer.” The output changes. It now says “hot path” and “tail latency”. The method does not change. It still reads the code and guesses.

It is a common observation: prompted to act like professionals, models often change vocabulary without changing methods. The fix is to insist on the instruments.

For code, those instruments look like this:

• Profilers and traces instead of reading code for performance. A profiler measures where time and memory actually go; a trace follows one request across services.
• Property-based testing instead of a handful of examples. You state rules that must hold for every input, and a tool generates thousands of inputs to try to break them.
• Mutation testing to find out whether your tests can actually fail. The tool plants small bugs in your code and checks that the tests catch each one; bugs that survive reveal weak tests.
• Fuzzing for parsers and anything that accepts untrusted input. It feeds large volumes of random or malformed input to find crashes and security holes.
• Differential testing when you rewrite. Old and new implementations run on the same inputs, and any difference is flagged.
• Formal specification for the small, critical core where a subtle bug is expensive: consensus, money movement, safety interlocks. TLA+ (TLA+ – Wikipedia) is a language for specifying a design and having a model checker explore every possible ordering of events.

You do not have to know all of these yourself. The model usually does. What you have to do is refuse to accept the shallow version.

Break the seed: a small example

Here is how this looks on something ordinary. Suppose an agent writes a function to split a payment into instalments, with these tests:

def test_split_even():
    assert split(900, 3) == [300, 300, 300]
def test_split_uneven():
    assert split(1000, 3) == [334, 333, 333]

The tests pass. The spec is satisfied. Now ask for depth: what must be true for every amount and every instalment count? Here it is with Hypothesis, a property-based testing library for Python:

from hypothesis import given, strategies as st
@given(st.integers(min_value=0, max_value=10**12),
       st.integers(min_value=1, max_value=360))
def test_split_properties(amount, n):
    parts = split(amount, n)
    assert len(parts) == n
    assert sum(parts) == amount          # no money created or lost
    assert max(parts) - min(parts) <= 1  # fair to within one unit
    assert all(p >= 0 for p in parts)

That is breaking the seed open. The first tests checked two examples. The second checks the invariant, across millions of generated cases, including the ones nobody thought to write down. Run mutation testing on top and you learn whether the suite would even notice if someone broke the rounding.

Taste the water: challenge what you believe

Uddālaka’s second lesson matters as much as the first. The salt was invisible, but tasting proved it was everywhere. Much of what teams believe about their systems is the opposite: highly visible and never tasted.

“The bottleneck is the database.” “Most errors come from the mobile client.” “Nobody uses the legacy export.” Give the agent the raw traces, the raw error logs and the raw access logs, and ask it to reproduce each belief from the data. Some will hold. Some will not, and those are worth more than anything you were going to build.

The depth prompt

Here is the prompt I use for depth, adapted for code:

USER STORY
As the owner of <task>, I want a deeper version of it, not a bigger one,
so that the result would stand up to the best engineer in this area.
DEFINITION OF READY
- The original task and its current output are attached.
- The raw inputs behind it (logs, traces, data samples) are accessible.
DEFINITION OF DONE
- Before any work, answer three questions: (1) what would the best
  engineer in this specific area do here that I have not asked for;
  (2) what raw material would they start from instead of my summary;
  (3) which tools and methods would they use.
- Rewrite the task to do exactly that, then do it.
DEFINITION OF SUCCESS
- Name what, other than you or me, will show the result is right, and
  run it. The result is accepted only if that check passes.

The Definition of Success is the part that matters most, and it leads directly to the oracle, which is Part 6.

For the Weekend

Take one function your team considers well tested. Ask an agent to write property-based tests for its invariants and run mutation testing on the existing suite.

Example. Suppose the function is normalise_phone_number(raw, default_region) in a customer-onboarding service. It has 14 unit tests and 100% line coverage, and everyone considers it done.

Step 1: run the depth prompt on it.

USER STORY
As the maintainer of the onboarding service, I want evidence that
normalise_phone_number is correct for every input, not just 14 examples,
so that malformed numbers never reach the SMS gateway.
DEFINITION OF READY
- src/contacts/phone.py and its 14 unit tests are on main.
- A masked sample of 10,000 real signup numbers is in data/samples/.
- Hypothesis and mutmut are installed in the dev environment.
DEFINITION OF DONE
- First answer the depth questions: what a telephone-numbering
  specialist would test, from what raw material, with which tools.
- Property-based tests in tests/test_phone_properties.py.
- mutmut configured for src/contacts/phone.py, with a full run completed.
- You MUST NOT modify phone.py. Report failures; do not fix them.
DEFINITION OF SUCCESS
- Every Hypothesis failure is reported with its shrunk (smallest) input.
- The mutation score is reported, and every surviving mutant is listed
  with the line it changed.

The agent’s answer, in short: test the rules, not examples; start from a sample of real numbers in the signup logs (masked); use Hypothesis for properties and mutmut for mutation testing; let mutation survival prove test strength.

Step 2: the properties. Output must be E.164 (the international +<country code><number> format, at most 15 digits) or an explicit error, and normalising twice must change nothing:

import re
from hypothesis import given, strategies as st
from contacts.phone import normalise_phone_number, InvalidPhoneNumber
E164 = re.compile(r"^\+[1-9]\d{1,14}$")
RAW = st.text(alphabet="0123456789 +-()x.ext", min_size=1, max_size=30)
@given(RAW)
def test_output_is_e164_or_explicit_error(raw):
    try:
        out = normalise_phone_number(raw, default_region="IN")
    except InvalidPhoneNumber:
        return                          # an explicit error is acceptable
    assert E164.match(out), f"{raw!r} -> {out!r}"
@given(RAW)
def test_normalising_twice_changes_nothing(raw):
    try:
        once = normalise_phone_number(raw, default_region="IN")
    except InvalidPhoneNumber:
        return
    assert normalise_phone_number(once, default_region="IN") == once

Hypothesis shrinks any failure to the smallest input that still breaks the rule. In this scenario it reports:

Falsifying example: test_output_is_e164_or_explicit_error(raw='0 x1')
ValueError: invalid literal for int() with base 10: ''

An extension marker with no number before it crashes the function with an unhandled ValueError. No unit test would have chosen that input.

Step 3: test the tests. Mutation testing with mutmut, configured in pyproject.toml:

[tool.mutmut]
paths_to_mutate = ["src/contacts/phone.py"]

mutmut run
mutmut results

Suppose 9 of 31 mutants survive. Each survivor is a bug the suite cannot see. Most sit in the country-code branch, where changing "91" to "92" still passes every test, because all 14 examples were Indian mobile numbers. Line coverage said 100%. Breaking the seed said the suite had never really looked at that branch.

Break the seed. See what is inside.

From Tickets to Invariants

mallyanitin 26 Oct 2026 4:06

The most ambitious instruction you can give an agent is not a feature. It is a property the system must keep.

The Sanskrit word dharma is usually translated as duty, religion or righteousness. None of these is quite right. The word comes from the root dhṛ, to hold or to sustain. The Mahabharata puts it plainly: dharma is called dharma because it holds things together.

Older still is the Vedic idea of ṛta, the order that keeps the world running. Rivers flow downhill. Dawn returns. The seasons turn. In the Rigveda, Varuṇa is the guardian of ṛta, and he keeps watch over the world through his spies.

I think about these two words whenever I see a backlog.

Karma versus dharma

A ticket is an act. Fix this crash. Add this field. Speed up this endpoint. It has a beginning and an end, and once it is closed nobody looks at it again.

A system’s dharma is different. It is what must keep holding while a thousand tickets come and go. Every request is authorised. Every payment reconciles. p99 latency stays under 300 milliseconds (99 of every 100 requests finish within that time). No release goes out with a known critical vulnerability. Nobody closes these. They are kept, or they are broken.

Most agent work today is ticket work. We hand over acts, one at a time, and the agent performs them. Now imagine the alternative. Instead of asking an agent to fix each crash as it is reported, you ask it to keep the apps crash-free.

That second instruction is shorter than the first, and enormously more ambitious. It moves the agent from karma to dharma: from performing acts to sustaining an order.

Altitude is earned, not prompted

Here is the part that is easy to miss. “Keep the app crash-free” is useless as a prompt on its own. It only works once three things are in place:

• the agent could fetch crash reports on its own;
• it could map any app version to the exact git commit;
• it had access to Windows, macOS and Linux machines to reproduce and verify each fix.

In other words, you give the agent Varuṇa’s spies. Senses to see when the order is breaking, and hands to restore it.

This is why I say altitude is not a prompting style. It is an engineering investment. Before an agent can own an invariant, it needs:

1. Telemetry it can read. Logs, metrics, traces and error reports, queryable without a human in the loop.
2. Provenance. From any running build back to the commit, the dependency versions and the config that produced it.
3. Reproducible environments. A way to recreate the failure on demand, on the platform where it happens.
4. A safe path to change. Branches, CI, staged deploys and automatic rollback.
5. An oracle for the invariant itself. A check that says, mechanically, whether the order holds right now.

Miss any one of these and the agent is back to doing tickets, only with grander language.

Choosing a system’s dharma

When I run this exercise with teams, I ask them to name three to five invariants for one service. The good ones share three traits: they are stated as properties, not tasks; they can be checked by a machine; and a breach is unambiguous.

Some that work well:

• Correctness: every ledger entry has a matching counter-entry; every ingested document produces a graph that conforms to its SHACL shapes (SHACL is a W3C language for stating rules a knowledge graph must satisfy).
• Performance: p99 latency under a stated bound; cost per thousand requests under a stated bound.
• Security: zero high-severity findings in static analysis (tools that scan source code for flaws without running it); no secrets in the repository; every endpoint behind an authorisation policy.
• Compatibility: the public API contract does not change without a version bump; database migrations are backward compatible for one release.

Notice that none of these needs an AI to check. That is deliberate. As I argued in “Don’t Grow Brains Where Bones Would Do”, the check should be deterministic wherever it can be. The agent reasons about how to restore the invariant. The code decides whether it has been restored.

Standing agents, bounded autonomy

Once an invariant has its senses, its hands and its oracle, the agent stops being a task runner and becomes a standing guardian. It runs on a schedule or on a signal. It notices a breach, investigates, proposes a fix, proves it and raises a pull request.

How far it goes alone is a separate decision, and it should be made per invariant, not per agent. Restoring a latency budget by tuning a cache is low risk. Changing an authorisation policy is not. In my Bounded Autonomy series I described the mechanisms for this: policy as code (access rules written as versioned, testable code), workload identity (each agent and service carries a verifiable identity instead of a shared secret), durable execution (long-running work that survives crashes and resumes where it stopped) and tamper-evident audit trails (logs that reveal any later alteration). They apply here directly. The higher the altitude, the more the bounds matter.

For the Weekend

Pick one service. Write down three invariants it must hold. For each, list which of the five pieces of plumbing above is missing.

Example. Suppose the service is an order-ingestion API. Start by writing its dharma down as a registry the agent can read but not edit:

# invariants/registry.yaml  (protected path: humans only)
- id: INV-ORD-01
  statement: Every accepted order reaches the warehouse system within 5 minutes
  check: invariants/order_reaches_warehouse.sql
  agent_may_fix: false        # money and stock involved: propose only
- id: INV-ORD-02
  statement: p99 latency of POST /orders stays under 400 ms
  check: invariants/orders_latency_p99.promql
  agent_may_fix: true         # low risk: tune, test, merge behind review
- id: INV-ORD-03
  statement: No schema change breaks the three downstream consumers
  check: tests/contracts/
  agent_may_fix: false

Each check is mechanical. The first is a SQL query that should always return zero rows:

-- Orders accepted more than five minutes ago with no warehouse receipt
SELECT o.order_id, o.accepted_at
FROM orders o
LEFT JOIN warehouse_receipts w ON w.order_id = o.order_id
WHERE o.accepted_at < now() - interval '5 minutes'
  AND w.order_id IS NULL;

The second is PromQL, the query language of the Prometheus monitoring system, and it should always be true:

histogram_quantile(0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket{route="/orders"}[5m]))
) < 0.4

Now audit the plumbing for each invariant:

Four gaps. Close them as ordinary tickets. Only then write the standing agent’s instruction, run as a scheduled task every 15 minutes:

USER STORY
As the on-call engineer for order ingestion, I want an agent to watch the
service's invariants continuously, so that breaches are found and
evidenced before customers notice them.
DEFINITION OF READY
- invariants/registry.yaml exists, and every check in it runs.
- Read access to order trace IDs, logs and Prometheus metrics.
- A sandbox in which a single order can be replayed.
DEFINITION OF DONE (on each run, every 15 minutes)
- Execute every check in the registry. If all pass, log
  "all invariants hold" and stop.
- On a breach: collect evidence, reproduce it in the sandbox if possible,
  and write a failing test that captures it.
- If agent_may_fix is true: fix it on a branch, show the failing test now
  passes, and open a pull request.
- If agent_may_fix is false: open an issue with the evidence and a
  proposed fix, and page the on-call engineer.
- You MUST NOT edit anything under invariants/ or tests/contracts/.
DEFINITION OF SUCCESS
- Every breach has an issue or a pull request within 30 minutes of first
  detection.
- The agent's history shows zero changes to protected paths.
- Monthly review: no breach was first reported by a customer.

That list is your real altitude roadmap. The prompt is the easy part. The table above is the real work.