The Oracle Problem

The Oracle Problem

mallyanitin 27 Oct 2026 4:23

Generation has outrun verification. The team with the better oracle gets to be more ambitious, because everyone else is limited by their review queue.

Consider Sītā’s position in the Aśoka grove.

She is a prisoner in Lanka, surrounded by rākṣasīs, beings who can take any form they choose. She was abducted in the first place by Rāvaṇa disguised as a wandering holy man. Then, one day, a monkey appears in the tree above her and says, in perfect Sanskrit, that he is a messenger from her husband.

Why should she believe him? Everything about her situation tells her that appearances lie and words cost nothing.

Rāma had foreseen this. Before the search parties left, he gave Hanumān his signet ring. Hanumān’s words were testimony. The ring was evidence: something the other side could not easily produce.

Sītā, in turn, sent Hanumān back with her cūḍāmaṇi, the jewel from her hair, and something more: a private memory, the story of a crow that once attacked her, known only to her and Rāma. A token and a secret. Mutual verification, long before anyone wrote the word “handshake”.

I tell this story to engineering teams because every agent’s pull request arrives like that monkey in the tree. Fluent, confident and claiming to come from the right place. The question is what ring you asked it to carry.

The three-speed problem

An agent can now write a thousand lines before lunch. A senior engineer can genuinely review, not skim, perhaps a couple of hundred lines an hour on unfamiliar code. Terence Tao has described the same imbalance in mathematics: proofs are now produced faster than they can be checked, and checked faster than anyone can understand them.

Three activities, three speeds: generation, verification, comprehension. The gap between the first and the other two is where defects slip through, and where teams quietly lower their standards because the queue is too long.

You cannot close this gap by reviewing faster. You close it by moving judgement out of the reviewer’s head and into mechanisms. As I like to put it: the oracle reasons; the code decides.

Hetvābhāsa: reasons that only look valid

Nyāya logicians catalogued hetvābhāsa, reasons that appear sound but are not. A reason that proves too much. A reason that contradicts the conclusion. A reason that is itself unproven. A reason that would prove the opposite equally well.

Agent output is full of hetvābhāsa, and not because agents are dishonest. They are fluent, and fluency is excellent camouflage. Tao makes a related point about AI-written proofs: polished arguments can hide the one weak step. A pull-request description that says “refactored for clarity, all tests pass, no behaviour change” is three claims, each of which needs its own pramāṇa.

A hierarchy of oracles

Not all checks are equal. From strongest to weakest:

1. Types and formal proofs. The compiler, or a proof assistant (software that checks every step of a proof), either accepts it or does not. No persuasion possible.
2. Property and contract tests. Invariants checked across thousands of generated inputs, and agreed data shapes and behaviour checked at every service boundary.
3. Differential and golden tests. Does the new code behave like the old code (a differential test), or match a saved, approved output (a golden test)?
4. Integration and production signals. Does it hold up against real traffic, real data and real infrastructure?
5. Independent agent review. A second agent, with no access to the first one’s reasoning, looking for flaws.
6. Human review. Essential, and the most expensive and least scalable, so it should be spent where the layers above cannot reach: design, intent and judgement.

Most teams I see lean heavily on layer 6 and lightly on layers 1 to 3. Ambition grows as you move weight up the list.

The generation, verification and comprehension bars grow at different rates, then the six oracle layers stack from strongest to most costly.

Pūrvapakṣa: the reviewer who argues the other side

Indian philosophical debate, śāstrārtha, had a rule I wish code review shared. Before stating your conclusion, the siddhānta, you had to set out the opposing view, the pūrvapakṣa, in its strongest form. A conclusion that had not survived the best opposing case was not yet a conclusion.

This is the right design for agent review. The reviewing agent’s job is not to approve. It is to construct the strongest case that the change is wrong. And it must work independently: no access to the author agent’s plan or reasoning, only the diff, the tests and the spec. An agent that has read the author’s justification is already half-persuaded. One that has only seen the evidence must reason from it.

Separation of duties

The single most effective rule I know is also the simplest: the agent that writes the code never writes or edits the checks that judge it.

In practice:

• Tests, thresholds, benchmark configurations and policy files live in protected paths. The coding agent cannot write to them; hooks (scripts that run automatically before an agent’s action or a commit) and CI (the pipeline that builds and tests every change) enforce it.
• Any diff that touches a test, loosens an assertion, adds a mock or changes a threshold is flagged automatically for human review.
• Test-writing and code-writing agents are separate, run in separate contexts and, ideally, see different information. The test writer sees the spec and the invariants. The code writer sees the spec and the failing tests.

This is Rāma’s ring. Sītā never asked Hanumān to prove himself with evidence he had made himself.

Measuring the gap

If the oracle is the ceiling, measure it. Three numbers are enough to start:

• Review latency: how long agent pull requests wait for a human.
• Escaped defects: bugs found after merge, per hundred agent pull requests.
• Verified share: the proportion of changed lines covered by layers 1 to 4, not just by human eyes.

For the Weekend

Run an oracle audit on one repository. List which of the six layers exist. Then list which of them the coding agent can currently modify.

Example. Suppose the repository is a claims-processing service.

Step 1: let an agent run the audit.

USER STORY
As the claims team lead, I want to know which of our checks a coding agent
could quietly weaken, so that I can protect the strongest ones first.
DEFINITION OF READY
- Read access to the repository, .claude/settings.json, CODEOWNERS, the
  CI workflows and the branch-protection notes in docs/.
DEFINITION OF DONE
- A table with one row per oracle layer: (1) types and proofs,
  (2) property and contract tests, (3) differential and golden tests,
  (4) integration and production signals, (5) independent agent review,
  (6) human review.
- For each layer: the files that implement it, and whether a coding agent
  can modify each file without a human noticing, with the reason.
- You MUST NOT change anything.
DEFINITION OF SUCCESS
- Every "can modify" finding names the missing control (hook, CODEOWNERS
  entry or branch rule), so that a human can verify it in under a minute.

The result:

The two strongest checks the team has are the two the agent can quietly loosen.

Step 2: protect the ring. In Claude Code, a hook is a script that runs before a tool call and can block it. This one, in the project’s .claude/settings.json file, stops the coding agent from writing to oracle files:

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Edit|Write|MultiEdit",
        "hooks": [{ "type": "command", "command": "python3 .claude/hooks/protect_oracles.py" }]
      }
    ]
  }
}

# .claude/hooks/protect_oracles.py
import fnmatch, json, os, sys
PROTECTED = ["mypy.ini", "tests/golden/*", "tests/contracts/*",
             ".github/workflows/*", ".claude/*"]
event = json.load(sys.stdin)
path = event.get("tool_input", {}).get("file_path", "")
rel = os.path.relpath(path, event.get("cwd", "."))
if any(fnmatch.fnmatch(rel, pattern) for pattern in PROTECTED):
    print(f"{rel} is an oracle file. Describe the change you need in your "
          "report instead; a human will make it.", file=sys.stderr)
    sys.exit(2)   # exit code 2 blocks the edit and shows the message to the agent

The hook is the agent-side guard. CODEOWNERS is the repository-side one, so even a human pull request touching these paths needs a lead’s approval:

/mypy.ini           @claims-leads
/tests/golden/      @claims-leads
/tests/contracts/   @claims-leads

Step 3: add layer 5, the pūrvapakṣa reviewer. A second agent, started fresh, sees only the diff, the tests and the ticket, never the author agent’s plan or reasoning:

USER STORY
As the human approver of pull request #412, I want the strongest honest
case against this change, so that I approve it only after it has
survived that case.
DEFINITION OF READY
- You have the diff, the CI results and ticket CLM-881.
- You have NOT seen the author agent's plan or reasoning, and you MUST NOT
  try to infer it.
DEFINITION OF DONE
- A list of concrete objections: a behaviour the change breaks, an input
  it mishandles, or a claim in the description that the evidence does
  not support.
- Each objection cites the line and proposes a test that would
  demonstrate it.
- If you cannot build a case, say so plainly.
- You MUST NOT approve or reject.
DEFINITION OF SUCCESS
- Each proposed test is written and run. An objection counts only if its
  test fails on the branch.

Every check an agent can edit is a ring it can forge.