Below the Spec

Below the Spec

mallyanitin 26 Oct 2026 4:01

Telling an agent to act like an expert changes its vocabulary. Giving it an expert’s raw material and instruments changes its results.

In the Chāndogya Upaniṣad, a young man named Śvetaketu comes home after twelve years of study. He is twenty-four, he has mastered the Vedas, and he is, the text tells us, rather pleased with himself.

His father, Uddālaka Āruṇi, asks him a single question. Did you learn the teaching by which the unheard becomes heard, the unthought becomes thought, the unknown becomes known? Śvetaketu has not. His teachers, he says, must not have known it either.

So Uddālaka teaches him, and not by lecture. He sends him for a fruit of the banyan tree. “Break it open. What do you see?” Tiny seeds. “Break one of those. What do you see?” Nothing at all. And from that nothing, says his father, the whole great tree grows.

Then he has Śvetaketu drop salt into water and leave it overnight. In the morning: “Bring me the salt.” The boy cannot find it. “Taste the water from the top. From the middle. From the bottom.” Salty, every time. It is there, in all of it, though you cannot see it.

I love this passage because it is an experiment, not a sermon. Uddālaka does not tell his son the answer. He makes him break things open and taste them.

Śvetaketu’s problem is the spec’s problem

Śvetaketu had twelve years of digested knowledge and had never touched the raw material. That is exactly where a spec puts an agent.

A spec is pre-digested. By the time it reaches the agent, someone has already decided what the problem is, which data matters and what “done” looks like. The agent works on the summary, not the thing.

The best engineers I have worked with never start from the summary. A performance engineer handed a ticket saying “the checkout page is slow” does not start from the ticket. She starts from a flame graph, a picture of where the program actually spends its time. A security reviewer does not start from a checklist. He starts from the data flow. A good reliability engineer, given an incident report, goes straight to the raw logs, because the report already contains someone’s interpretation.

That is what depth means: going one layer below where the task was handed to you.

Vocabulary is not method

The usual way people try to get depth from a model is to ask it to role-play. “You are a senior performance engineer.” The output changes. It now says “hot path” and “tail latency”. The method does not change. It still reads the code and guesses.

It is a common observation: prompted to act like professionals, models often change vocabulary without changing methods. The fix is to insist on the instruments.

For code, those instruments look like this:

• Profilers and traces instead of reading code for performance. A profiler measures where time and memory actually go; a trace follows one request across services.
• Property-based testing instead of a handful of examples. You state rules that must hold for every input, and a tool generates thousands of inputs to try to break them.
• Mutation testing to find out whether your tests can actually fail. The tool plants small bugs in your code and checks that the tests catch each one; bugs that survive reveal weak tests.
• Fuzzing for parsers and anything that accepts untrusted input. It feeds large volumes of random or malformed input to find crashes and security holes.
• Differential testing when you rewrite. Old and new implementations run on the same inputs, and any difference is flagged.
• Formal specification for the small, critical core where a subtle bug is expensive: consensus, money movement, safety interlocks. TLA+ (TLA+ – Wikipedia) is a language for specifying a design and having a model checker explore every possible ordering of events.

You do not have to know all of these yourself. The model usually does. What you have to do is refuse to accept the shallow version.

Break the seed: a small example

Here is how this looks on something ordinary. Suppose an agent writes a function to split a payment into instalments, with these tests:

def test_split_even():
    assert split(900, 3) == [300, 300, 300]
def test_split_uneven():
    assert split(1000, 3) == [334, 333, 333]

The tests pass. The spec is satisfied. Now ask for depth: what must be true for every amount and every instalment count? Here it is with Hypothesis, a property-based testing library for Python:

from hypothesis import given, strategies as st
@given(st.integers(min_value=0, max_value=10**12),
       st.integers(min_value=1, max_value=360))
def test_split_properties(amount, n):
    parts = split(amount, n)
    assert len(parts) == n
    assert sum(parts) == amount          # no money created or lost
    assert max(parts) - min(parts) <= 1  # fair to within one unit
    assert all(p >= 0 for p in parts)

That is breaking the seed open. The first tests checked two examples. The second checks the invariant, across millions of generated cases, including the ones nobody thought to write down. Run mutation testing on top and you learn whether the suite would even notice if someone broke the rounding.

Taste the water: challenge what you believe

Uddālaka’s second lesson matters as much as the first. The salt was invisible, but tasting proved it was everywhere. Much of what teams believe about their systems is the opposite: highly visible and never tasted.

“The bottleneck is the database.” “Most errors come from the mobile client.” “Nobody uses the legacy export.” Give the agent the raw traces, the raw error logs and the raw access logs, and ask it to reproduce each belief from the data. Some will hold. Some will not, and those are worth more than anything you were going to build.

The depth prompt

Here is the prompt I use for depth, adapted for code:

USER STORY
As the owner of <task>, I want a deeper version of it, not a bigger one,
so that the result would stand up to the best engineer in this area.
DEFINITION OF READY
- The original task and its current output are attached.
- The raw inputs behind it (logs, traces, data samples) are accessible.
DEFINITION OF DONE
- Before any work, answer three questions: (1) what would the best
  engineer in this specific area do here that I have not asked for;
  (2) what raw material would they start from instead of my summary;
  (3) which tools and methods would they use.
- Rewrite the task to do exactly that, then do it.
DEFINITION OF SUCCESS
- Name what, other than you or me, will show the result is right, and
  run it. The result is accepted only if that check passes.

The Definition of Success is the part that matters most, and it leads directly to the oracle, which is Part 6.

For the Weekend

Take one function your team considers well tested. Ask an agent to write property-based tests for its invariants and run mutation testing on the existing suite.

Example. Suppose the function is normalise_phone_number(raw, default_region) in a customer-onboarding service. It has 14 unit tests and 100% line coverage, and everyone considers it done.

Step 1: run the depth prompt on it.

USER STORY
As the maintainer of the onboarding service, I want evidence that
normalise_phone_number is correct for every input, not just 14 examples,
so that malformed numbers never reach the SMS gateway.
DEFINITION OF READY
- src/contacts/phone.py and its 14 unit tests are on main.
- A masked sample of 10,000 real signup numbers is in data/samples/.
- Hypothesis and mutmut are installed in the dev environment.
DEFINITION OF DONE
- First answer the depth questions: what a telephone-numbering
  specialist would test, from what raw material, with which tools.
- Property-based tests in tests/test_phone_properties.py.
- mutmut configured for src/contacts/phone.py, with a full run completed.
- You MUST NOT modify phone.py. Report failures; do not fix them.
DEFINITION OF SUCCESS
- Every Hypothesis failure is reported with its shrunk (smallest) input.
- The mutation score is reported, and every surviving mutant is listed
  with the line it changed.

The agent’s answer, in short: test the rules, not examples; start from a sample of real numbers in the signup logs (masked); use Hypothesis for properties and mutmut for mutation testing; let mutation survival prove test strength.

Step 2: the properties. Output must be E.164 (the international +<country code><number> format, at most 15 digits) or an explicit error, and normalising twice must change nothing:

import re
from hypothesis import given, strategies as st
from contacts.phone import normalise_phone_number, InvalidPhoneNumber
E164 = re.compile(r"^\+[1-9]\d{1,14}$")
RAW = st.text(alphabet="0123456789 +-()x.ext", min_size=1, max_size=30)
@given(RAW)
def test_output_is_e164_or_explicit_error(raw):
    try:
        out = normalise_phone_number(raw, default_region="IN")
    except InvalidPhoneNumber:
        return                          # an explicit error is acceptable
    assert E164.match(out), f"{raw!r} -> {out!r}"
@given(RAW)
def test_normalising_twice_changes_nothing(raw):
    try:
        once = normalise_phone_number(raw, default_region="IN")
    except InvalidPhoneNumber:
        return
    assert normalise_phone_number(once, default_region="IN") == once

Hypothesis shrinks any failure to the smallest input that still breaks the rule. In this scenario it reports:

Falsifying example: test_output_is_e164_or_explicit_error(raw='0 x1')
ValueError: invalid literal for int() with base 10: ''

An extension marker with no number before it crashes the function with an unhandled ValueError. No unit test would have chosen that input.

Step 3: test the tests. Mutation testing with mutmut, configured in pyproject.toml:

[tool.mutmut]
paths_to_mutate = ["src/contacts/phone.py"]

mutmut run
mutmut results

Suppose 9 of 31 mutants survive. Each survivor is a bug the suite cannot see. Most sit in the country-code branch, where changing "91" to "92" still passes every test, because all 14 examples were Indian mobile numbers. Line coverage said 100%. Breaking the seed said the suite had never really looked at that branch.

Break the seed. See what is inside.