I Stopped Prompting Claude Code and Started Engineering the Loop

A private auditMalin review recorded 110 passing tests and named repository checks that were green. The same review finding identified a seam the narrower checks had missed: the FastAPI-to-ADK integration path could call asyncio.run() inside an already active event loop.

This was a review finding, not evidence of harm in a live system. A later attributable regression test exercised that API-to-runner seam directly. The useful lesson is narrower and more durable: green checks were not a complete universal quality proof, especially when none of them crossed the boundary where the failure lived.

I still prompt. I stopped treating the prompt as the reliability mechanism. The prompt starts a run and communicates intent; the engineering work is to put durable context, executable contracts, review criteria, and recoverable state around the stochastic worker.

In this workflow, confidence is not an acceptance term. Three questions direct the evidence search: Which sensor can produce a counterexample? Does it reach the relevant seam? What external release boundary remains after the agent finishes its turn?

A tiny retry function with two surviving defects

Consider a retry delay that doubles until it reaches a cap:

function retryDelay(attempt, cap = 8) {
  return Math.min(2 ** attempt, cap);
}

for (let attempt = 0; attempt <= 10; attempt += 1) {
  console.log(attempt, retryDelay(attempt));
}

Save that as retry-delay.mjs and run it with Node. The outputs for attempts 0, 1, and 2 are 1, 2, and 4. Those are attractive examples: short, familiar, and all below the cap. They are also weak evidence. Two plausible defects agree with every one of them.

The no-cap mutant is 2 ** attempt; the cap-minus-one mutant is Math.min(2 ** attempt, cap - 1). The first forgets the upper bound. The second is an off-by-one error in the bound itself. With cap = 8, both mutants return the expected values for those first three attempts.

Now move one step to the boundary. At attempts 3 and 4, the reference outputs are 8 and 8. The no-cap version returns 8 and 16. The cap-minus-one version returns 7 and 7. Nothing about the implementation became harder to understand; the input selection finally reached the part of the contract where the implementations differ.

Examples still leave a question: what should hold across a broader input domain? For attempts 0 through 10, a useful property says the output remains in [1, 8], doubles while the previous output is below 8, and stays at 8 afterward. This bounded recurrence rejects both named mutants on the frozen domain.

A polished property can also encode the wrong specification. Suppose I write “every adjacent output always doubles.” It looks stronger than three examples. It quantifies across the domain and compares adjacent values. Yet it has deleted the cap from the oracle. After attempt 3, it rejects exactly the behavior the function is supposed to have.

The frozen experiment records four sensors against the reference and the two hand-written mutants. Here, pass means the sensor accepts an implementation and reject means it flags one.

Figure 1. A polished oracle can still encode the wrong contract: this broad but invalid property makes both a false rejection and a false acceptance.

The always-doubles oracle falsely rejects the correct reference yet falsely accepts the uncapped mutant. That pair is the center of the experiment: more inputs and more mathematical-looking machinery do not rescue an invalid oracle.

These 12 outcomes are exact only for the two named mutants and this frozen input domain: cap = 8, attempts 0 through 10. The table is not a Claude Code benchmark and not a general mutation score. It does not estimate an agent’s accuracy or a test suite’s effectiveness on an unseen defect population. It demonstrates, on inspectable inputs, why “the tests passed” is incomplete without asking what the tests could distinguish.

Goodenough and Gerhart’s early theory of test-data selection made a related distinction: familiar structural coverage criteria are not generally sufficient for reliable testing. Barr and colleagues name the deeper difficulty the oracle problem. Inputs make a program produce observations; an oracle decides whether those observations are acceptable. QuickCheck showed how executable properties could be checked over generated inputs, but the property still has to express the intended behavior. Generation scales an oracle. It does not repair one.

Turn a response into a state transition

The event-loop seam and the retry mutants have the same shape. A proposal passes through some checks. A named counterexample reveals that the current sensor set did not cover a relevant behavior. The patch goes back through repair with that counterexample attached.

The acceptance rule is:

accept(p_t) = spec(p_t) ∧ focused_tests(p_t) ∧ integration_tests(p_t) ∧ fresh_review(p_t)

latest_counterexample = api_runner_inside_active_loop
p_(t+1) = repair(p_t, latest_counterexample)

p_t is the proposed patch at turn t. Each term is a predicate over that patch and the applicable repository state. spec checks the stated behavior and exclusions. focused_tests gives fast local evidence around the change. integration_tests crosses the real boundary - in the running example, from the API path into the runner while an event loop is active. fresh_review looks for contradictions, missing cases, and unsupported claims after the patch and test evidence exist.

The conjunction is deliberately unforgiving: one false term returns a counterexample instead of an acceptance. The repair transition is equally important. Feedback such as “try again” carries little state. api_runner_inside_active_loop failed because the synchronous wrapper entered an active loop names a reproduction that can become a regression test, be rerun, and survive the chat session.

This predicate does not prove correctness. Each term is only as good as its scope and oracle. The value is operational: acceptance becomes inspectable, a failed term has a name, and repair starts from evidence rather than from another paraphrase of the original prompt.

Figure 2. Named feedback transitions make rejection actionable, while an external release boundary keeps conversational acceptance from becoming merge authority.

Guidance, feedback, and release are different layers

The path from intent to release is CLAUDE.md → tests and hooks → CI plus protected merge. Each layer has a different job.

Repository guidance consists of CLAUDE.md, imported rules, and skills. It tells the agent how this codebase is organized, which commands are authoritative, what evidence is allowed, and which reusable procedure applies. Guidance shapes choices, but it is context rather than enforcement.

Turn-level feedback consists of tests and hooks. A failing focused test provides a concrete counterexample during the work. A hook can run a deterministic command at a lifecycle event and feed a failure back into the same Claude turn. These controls are close to the worker and useful for fast repair.

The release boundary is CI plus a protected merge policy. It evaluates committed repository state outside the agent’s conversational decision to stop. A local hook can be skipped, misconfigured, or bounded by a liveness rule. Protected merging is where required checks become a condition for integrating the patch.

The layers should reference one another without pretending to be interchangeable. A concise rule can name make test; the test encodes the behavior; CI executes it from a clean environment; protected merge refuses a result that lacks the required status. A reusable skill can describe an evidence-review procedure, while a hook ensures a must-run command actually runs at the selected event. One prompt should not carry all of those jobs.

Make repository guidance discoverable

The frozen auditMalin evidence for this article tracks AGENTS.md but no CLAUDE.md. That is an important disclosure because official Claude Code documentation describes CLAUDE.md as project memory; Claude Code does not automatically load AGENTS.md merely because another agent convention uses that filename.

A minimal proposed bridge is one line:

@AGENTS.md

Place it in the applicable CLAUDE.md, start a fresh session, and inspect /context to verify that the import appears. That final check matters. Naming a file in an essay does not establish what a particular session loaded.

The imported rules should stay concise and navigational: name the source of truth, required commands, privacy boundaries, and paths to deeper procedures. Official guidance distinguishes persistent project instructions from skills, whose full content enters context when invoked. That division keeps always-loaded guidance small and puts longer, task-specific steps behind a discoverable procedure.

Even correctly loaded CLAUDE.md remains context. The official memory documentation explicitly contrasts instructions with enforced configuration. A sentence saying “always run the checks” may influence behavior; a test command or hook produces observable evidence.

A proposed Stop-hook hardening sample

The following is a proposed hardening sample, not currently deployed in the frozen auditMalin evidence. It is intentionally small so its control flow is visible. Claude Code passes hook input on stdin, and the sample runs an existing repository command rather than reproducing the command’s logic inside the hook.

import { readFileSync } from "node:fs";
import { spawnSync } from "node:child_process";

const input = JSON.parse(readFileSync(0, "utf8"));

if (input.stop_hook_active) {
  process.exit(0);
}

const result = spawnSync("make", ["check"], { stdio: "inherit" });
if (result.status === 0) {
  process.exit(0);
}

console.error("make check failed; repair the reported counterexample");
process.exit(2);

For a Stop hook, exit code 2 tells Claude Code to block stopping and use stderr as feedback for continued work. Every other non-2 nonzero exit, including exit code 1, is nonblocking. Accidentally returning 1 after a failed command therefore reports a hook error without continuing the agent turn.

The stop_hook_active field deserves an explicit branch. Claude Code sets it when the Stop hook is already continuing a turn because of a previous block. stop_hook_active does not guard recursion by itself; the hook must inspect it and allow stopping while it is active. Otherwise the same failing gate can re-block its own continuation.

Stop fires after the main agent finishes responding; it is not a signal that the objective task is complete. Claude Code overrides the hook after eight consecutive blocks by default, but CLAUDE_CODE_STOP_HOOK_BLOCK_CAP can raise that cap. This configurable liveness boundary changes how long turn-level feedback may continue; it is not eight guaranteed retries and does not change the separate release boundary.

The sample is a turn-level feedback control. CI plus protected merge remains the release gate. Before enabling a real hook, its command should fit the event’s runtime budget, preserve diagnostic output, and be tested with controlled JSON inputs for inactive, active, passing, and failing cases.

Fresh review is another sensor

A fresh scoped review can catch a contradiction that the author and focused tests share. That makes it another sensor, not a statistically independent oracle and not a vote. “Fresh” means the reviewer receives the current patch, requirements, and test evidence after implementation. It does not mean the reviewer’s errors are uncorrelated with the worker’s.

Knight and Leveson’s journal article is useful only at its recorded scope. The bounded experimental details come from UVA TR-85-11: for one specification, 27 versions were prepared independently at two universities and exercised with one million test cases; coincident failures exceeded the independence model’s expectation. It would be a category error to turn that experiment into a universal theorem about software diversity, and a larger error to declare two language-model passes independent because their prompts or sessions differ.

Kim and colleagues report task- and dataset-specific measurements of correlated errors in language models. The bounded engineering implication is to diversify the evidence surface. A deterministic boundary test, a type checker, a browser assertion, and a scoped semantic review fail for different immediate reasons. An LLM review can still be useful, but two LLM verdicts should not be combined as though they were independent trials.

Scope is part of review quality. “Review this patch” invites taste and summaries. “Check whether the FastAPI request path can invoke the synchronous runner from an active loop; cite the line and supply a reproducer” asks for a falsifiable result. When that result exists, it should be converted into the cheapest durable sensor that preserves the behavior. The later API-to-runner regression test is more valuable than remembering that a reviewer once noticed the seam.

The replay capsule is for diagnosis

Long agent runs are difficult to reconstruct from a transcript. Suppose api_runner_inside_active_loop fails during repair. A useful capsule pins the Git commit and diff, the Claude Code, model, and provider identifiers, the exact regression-test invocation used for that run, the fixture in which a FastAPI request reaches the synchronous runner from an active event loop, and the output hash. The next pass can then distinguish changed code, invocation, fixture, executor, or generated result instead of guessing from the earlier response.

Figure 3. Frozen controllable state makes differences diagnosable, while model sampling and changing services prevent the capsule from promising byte-identical reruns.

A saved conversation can resume through Claude Code’s documented session controls. Claude editing-tool changes can rewind through checkpointing. Git records software state across commits and diffs. Checkpointing is not version control: it does not replace named commits, reviewable diffs, durable tests, or protected history. None of these controls freezes hosted model or provider execution, so the capsule improves diagnosis but does not promise identical stochastic replay.

The output hash answers one concrete reconstruction question. If the same recorded Git state, command, and active-event-loop fixture produce a different digest, the divergence is known before anyone compares explanations. The hash does not identify the cause; it narrows the next check to the recorded inputs and executor.

Limits

• Acceptance remains bounded. The private review’s 110-test gate and event-loop seam establish one missed integration boundary, not user impact or repository-wide correctness. The acceptance conjunction makes its evidence inspectable; an omitted requirement or invalid oracle can still pass through.
• The mutation table is deliberately tiny. Its exact counts cover one reference, two hand-written mutants, cap = 8, and attempts 0 through 10. It explains input and oracle selection on that frozen domain; it is neither a model benchmark nor an estimate over real patches.
• Guidance and turn feedback can still be bypassed. CLAUDE.md must be present in context, and the hook shown here is a proposal rather than evidence of current deployment. Stop runs after a response and has a liveness cap, so CI plus protected merge remains the release boundary.
• Review and reconstruction retain shared uncertainty. A fresh reviewer can share the worker’s assumptions, and review errors can correlate. A replay capsule exposes changes in recorded state but cannot freeze hosted model, provider, or external-service execution.

Operational checklist

A compact setup sequence is:

1. Put concise guidance in CLAUDE.md, import existing rules explicitly, and verify the loaded result with /context; move repeated procedures into a versioned skill.
2. Write the acceptance predicate before implementation; name the counterexample returned by each false term.
3. Give focused tests an observable defect and integration tests the consuming boundary; pressure-test the oracle against a correct implementation and plausible mutants.
4. Parse and test Stop-hook input, including stop_hook_active and exit 2 feedback; keep CI plus protected merge as the release boundary.
5. Give fresh review a scoped, falsifiable question; turn a durable finding into an executable check when practical.
6. Save a replay capsule before a long run; compare its output hash before diagnosing prose or model behavior.

The review finding that opened this article did not call for a more emphatic prompt. It called for a sensor at the FastAPI-to-ADK seam, a named active-loop counterexample, and a release process that could require the resulting regression test. That is what engineering the loop means in practice: make failures legible enough that the next turn can repair them and the final boundary can refuse them.

Primary sources

• Anthropic, “How Claude remembers your project”, especially the distinction between CLAUDE.md instructions and enforced configuration.
• Anthropic, “Automate actions with hooks” and the hooks reference, for stdin input, Stop timing, exit behavior, stop_hook_active, and the default/configurable consecutive-block cap.
• Anthropic, “Best practices for Claude Code”, for concise project instructions, hooks, and reusable skills.
• Anthropic, “Manage sessions”, “Checkpointing”, and “Extend Claude with skills”, for the documented recovery and procedure surfaces used above.
• J. B. Goodenough and S. L. Gerhart, “Toward a theory of test data selection” (1975), on the limits of familiar test-selection criteria.
• E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, “The Oracle Problem in Software Testing: A Survey” (2015), on the distinction between producing and judging observations.
• K. Claessen and J. Hughes, “QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs” (2000), on executable properties and generated test data.
• J. C. Knight and N. G. Leveson, “An Experimental Evaluation of the Assumption of Independence in Multiversion Programming” (1986), retained as the canonical journal citation identity.
• University of Virginia, UVA TR-85-11, whose registered abstract describes the bounded experiment: 27 versions of one specification, prepared at two universities and exercised with one million tests, with coincident failures exceeding the independence model’s expectation. This institutional report record is distinct from the journal citation above.
• E. Kim et al., “Correlated Errors in Large Language Models” (2025), used for its task- and dataset-specific correlated-error scope rather than a universal rate.