Tag: spec-driven-development

Spec-driven development fixed vibe coding. It has also become the ceiling. An old Indian question about knowledge explains why.

In the Nyaya tradition, a student who made a claim was asked one question before anything else: kena pramaṇena? By what means do you know this? Where is the proof?

The question was not rude. It was the start of every serious conversation. Gautama’s Nyāya Sūtra, written nearly two thousand years ago, lists four valid means of knowing, the pramāṇas:

• Pratyakṣa, perception. You saw it.
• Anumāna, inference. You reasoned to it from a reliable rule, the way smoke on a hill tells you there is fire.
• Upamāna, comparison. You recognised it by its likeness to something you already knew.
• Śabda, testimony. A trustworthy person told you.

I have been thinking about this list a great deal, because I now spend much of my week reviewing artefacts that no human wrote. And I have noticed that most of our agentic coding practice runs on only one of the four.

What spec-driven development got right

Let me be fair to where we are. A year ago, much AI-assisted coding was a slot machine. You typed a wish, pulled the lever and hoped. Reviews were archaeology.

Spec-driven development changed that. Plan mode before code, where the agent proposes a plan for approval before it writes anything. A story small enough to fit in one branch. A CLAUDE.md or Copilot instructions file that tells the agent the house rules. A written specification the agent must satisfy and the reviewer can check against. I have pushed teams hard towards this. It made agent work legible.

But look at it through Gautama’s eyes. A spec is śabda. It is testimony: a trusted person telling the agent what is wanted. And when the agent finishes and reports “all tests pass, the feature is complete”, that is śabda too, from a source whose reliability we have not established.

Nyāya was strict about testimony. Śabda is valid only when it comes from an āpta, a reliable person who knows the matter and has no reason to mislead. An agent under pressure to finish is not an āpta. It is a very fast, very capable colleague who will sometimes tell you what you wanted to hear.

So we have built an engineering practice where the input is testimony and the output is testimony, and the only perception in the loop is a tired reviewer reading a diff at six in the evening.

The spec is a ticket with manners

Here is the limit in plain terms. A spec tells the agent what to build. It does not say:

• what must remain true after the change, and keep remaining true;
• what the best practitioner in this area would measure before believing the result;
• how anyone, human or machine, would know the work is right without reading every line.

The agent can satisfy the spec and miss the point. It can pass the tests it was shown and break the property nobody wrote down. And because generation is now so cheap, it can do this at a volume no review queue can absorb.

A useful way to think about ambition with AI models splits it four ways: speed (same work, less time), altitude (aim at the outcome, not the task), depth (work the way the best expert in the field would) and range (bring in methods from other fields). It is a good map, and this series borrows it.

But for code, I think it needs a fifth axis, and that axis sets the ceiling on the other four.

The fifth axis: the oracle

In testing, an oracle is whatever tells you whether an output is correct. A unit test is an oracle. So is a type checker, a property-based test, a production SLO (service level objective, a target such as 99.9% of requests succeeding), a second reviewer who never saw the implementation.

My thesis for this series is simple: in agentic coding, your ambition is capped by your oracle.

If you cannot verify it, you cannot safely ask for it. Ask an agent to rewrite a payment reconciliation engine and the only thing standing between you and an incident is the quality of whatever checks the result. Improve the checks, and the size of what you can safely ask for grows. Leave them as they are, and better prompts will not help you. You will simply generate unverifiable work faster.

This is where the pramāṇas earn their place. Each one maps to a family of oracles, and a mature team uses all four:

In plain terms: traces follow a single request across every service it touches; integration tests exercise several real components together; property-based tests state a rule that must hold for every input and let a tool generate thousands of cases to break it; contracts fix the agreed shape and behaviour at a boundary between services; formal proofs are machine-checked arguments that code meets its specification; differential tests run old and new code on the same inputs and flag any difference; golden files are saved, approved outputs that new results must match.

The four pramāṇas appear in turn, then everything except śabda dims.

The Nyāya example for inference is worth dwelling on. Smoke tells you there is fire because of vyāpti, an invariable concomitance: wherever there is smoke, there is fire. Much of good engineering is the search for vyāptis. Wherever a request is authorised, an audit record exists. Wherever a document is ingested, the resulting graph satisfies its shapes. State the vyāpti, check it mechanically, and you no longer need to trust anyone’s testimony about that property.

Why this matters now

Terence Tao, writing about AI-assisted mathematics, has described a widening gap between how fast proofs can now be generated, how fast they can be verified and how fast humans can understand them. Swap “proofs” for “pull requests” and you have described every engineering team I talk to.

The teams I see pulling ahead are not the ones with the cleverest prompts or the longest specs. They are the ones who have quietly moved their best engineers from writing code to building the instruments that judge code. They treat telemetry, test design and invariants as the product, and the agent as the labour.

For the Weekend

Take the last five tasks you gave an agent. For each, write two lines: what was the spec, and what was the oracle? Then ask kena pramāṇena? How did you actually know the result was right?

A note on how prompts are written in this series. Every agent prompt uses one structure, borrowed from agile practice. The user story says who wants what, and why. The Definition of Ready (DoR) lists what must exist before the agent starts. The Definition of Done (DoD) lists what it must deliver and what it must not do, written with RFC 2119 keywords (MUST, MUST NOT, SHOULD) so that obligations are unambiguous. The Definition of Success (DoS) says how we will know it worked, measured by something other than the agent’s own report.

That last section is the oracle. If you can fill in only the first three, you have written a spec.

Example. Suppose last week’s task was a retry fix on a payment-status poller. Here is the prompt as it was given:

USER STORY
As an operations analyst, I want the payment-status poller to retry
transient failures, so that payments are not marked "unknown" after a
single network blip.
DEFINITION OF READY
- src/payments/poller.py and its unit tests are on main and passing.
- Retry policy agreed: at most 3 attempts; back-off 200 ms, then 400 ms;
  retry on 5xx responses and timeouts only.
DEFINITION OF DONE
- Plan mode: present the plan before writing any code.
- Retries implemented in StatusPoller as per the policy above.
- The public interface of StatusPoller MUST NOT change.
- Unit tests added for the retry behaviour.
DEFINITION OF SUCCESS
- All tests pass.

The agent’s closing report said: “Implemented retries with exponential back-off. Added 2 unit tests. Timeouts and connection resets are handled. All tests pass.”

Now the two-line audit:

Two of three claims rest on testimony. Now look back at the story. It was well formed, but its Definition of Success was “all tests pass”, and the agent wrote the tests. The next story fixes the DoS, not the code:

USER STORY
As the reviewer of the poller change, I want independent evidence for
each retry claim, so that I approve it on proof rather than on the
agent's report.
DEFINITION OF READY
- The retry change is on branch fix/poller-retries.
- tests/fakes.py provides ScriptedTransport and FakeClock.
- Hypothesis is installed in the dev environment.
DEFINITION OF DONE
- Property-based tests in tests/test_poller_properties.py, generating
  outcome sequences drawn from {200, 500, 503, timeout, connection reset}.
- The tests MUST assert: (1) at most 3 calls; (2) if any of the first
  three outcomes is 200, the final status is "ok"; (3) the waits are
  200 ms, then 400 ms.
- Use FakeClock so the tests run instantly.
- You MUST NOT modify anything under src/.
- If a property fails, report the smallest failing sequence. You MUST NOT
  fix it.
DEFINITION OF SUCCESS
- The properties run in CI on every pull request that touches
  src/payments/.
- A Toxiproxy fault-injection run against the payments sandbox shows
  correct retries on connection resets.

The resulting test looks like this:

from hypothesis import given, strategies as st
from payments.poller import StatusPoller
from tests.fakes import ScriptedTransport, FakeClock
OUTCOMES = st.sampled_from(["200", "500", "503", "timeout", "reset"])
@given(st.lists(OUTCOMES, min_size=1, max_size=8))
def test_retry_properties(sequence):
    transport = ScriptedTransport(sequence)   # one outcome per call; repeats the last when exhausted
    clock = FakeClock()
    poller = StatusPoller(transport, sleep=clock.sleep)
    poller.poll("pay_123")
    assert transport.calls <= 3
    if "200" in sequence[:3]:
        assert poller.last_status == "ok"
    assert clock.waits == [0.2, 0.4][: transport.calls - 1]

Then pratyakṣa against something real. Toxiproxy is a small proxy that injects network faults between your service and a dependency:

# Drop the connection 500 ms into every response from the payments sandbox
toxiproxy-cli toxic add -t reset_peer -a timeout=500 payments_sandbox
pytest tests/integration/test_poller_faults.py -v

In this scenario, Hypothesis finds the gap within seconds: on a connection reset, the poller raises instead of retrying, because the agent caught TimeoutError but not ConnectionResetError. The report said resets were handled. Kena pramāṇena? Now we know.

If most of your answers are śabda, you have found the ceiling. The rest of this series is about raising it.