A meta-harness has to derive the evidence it acts on

The previous post was a design argument: eight components, drawn, with a case for why each one had to exist. This one is twenty recorded runs of the prototype, most of them driven by a deliberately weak model, and the seven ways coordination between agents actually broke. Two of those runs are places where my design was wrong, and they taught me more than the ones where it held.

An agent was asked to build a small dashboard. It came back and said it had finished, and it reported the commit it had made. That commit did not exist. The model had composed a plausible-looking hash and handed it downstream as a fact.

The run stopped there, and what stopped it wasn’t the model catching itself, and it wasn’t a second model reviewing the first. Code sitting above the agent asked Git whether that commit was real, got an answer, and refused to go further. The agent produced a claim, and the evidence came from code that went and looked.

What I actually built

The prototype is called Bhai, short for Blueprint Harness for AI. A blueprint is the workflow itself, a versioned file the runtime reads and executes rather than a pipeline welded into the runtime.

Three names I lean on throughout. What the design post called the workflow engine turned out to be two pieces of code: the engine walks a blueprint from first step to last, and the runner runs a single step, dispatches to a worker, counts attempts, and writes artifacts to disk. An adapter per harness is the only code that knows what a Pi CLI or a Codex CLI is. Blueprints declare roles, not agents, and an agent package binds to a role. That binding is the only place the two meet, which is why a worker can be swapped later without the workflow noticing.

Writing those layers was not the hard part. Most of the code was written by a coding agent, and something that walks a blueprint, wraps a subprocess, and writes JSON to disk is a weekend with Claude Code. What took time was everything around it: where each check goes, what happens when one fires and another doesn’t, which decisions a model gets to make. Every one of those questions came from something going wrong.

Most of these runs use Gemma 26B as a worker on purpose. A weak model misbehaving is the expected output, and the thing under test is the layer around it.

The failure	What answers it
A worker reports work it never did	A gate that checks the claim against Git
No way to tell which worker produced a result	A registry that stamps identity onto every step
Output the next worker can’t use, or a stall	The output contract and the attempt budget
A worker acts outside its authority	The policy binder and the execution environment
One branch of a fan-out dies	A state store that keeps what already finished
Work that never converges, or self-declared completion	A cap the engine holds
No way to see any of it	The event stream

None of those answers is a better prompt.

A self-report is not evidence

Take the invented commit downstream. The implementer says it committed a fix with a given hash and hands along a diff. The diff is real, so a reviewer can read it, form an opinion, and write a perfectly good review without the commit ever needing to exist. The hash rides along as a label.

The review comes back, a consolidator merges it, a package goes out for approval, and inside it is a commit ID pointing at nothing. Every step did real work, and none was positioned to catch it, because each was told the commit existed by the only party that could know it was wrong. An agent is the worst available detector of its own fabrication, and any architecture where worker A’s self-report becomes worker B’s premise carries this hole.

The gate that catches it is one line of the blueprint, running after the worker has exited. It isn’t a tool the worker can call, skip, or see. When it trips, the failure is recorded and the run stops. Nothing retries: the implementer’s four attempts live inside the step, and by the time the gate speaks the step has already succeeded as far as the step is concerned. Resume doesn’t help, because resume replays a stored result and the stored result is the bad candidate.

The gate checks six things, each one there because a model got it wrong in exactly that way: the commit exists, it descends from the base commit the task named, the changed-file list matches the real diff, the referenced test artifact is there, the tree is clean, and no Git remote is configured so nothing can be pushed.

One failure I nearly missed was quieter than a fabrication. The hash was the repository’s own base commit with a single character dropped from the middle, 39 instead of 40. The model was copying and copied wrong, and reading quickly I’d have accepted it.

Then I changed the design, and this is the change I’d recommend to anyone building the same thing. I stopped asking the model what it did.

The change lives in the adapter, after the worker process has exited and before anything is handed back, and everything downstream is untouched: same output contract, same lineage gate, same events, different author. The worker is never told a candidate contract exists. It bought something back too. If the worker produced no commit and no changes there’s nothing to derive, and that returns a step failure rather than an empty candidate, which means the attempt budget can retry it. Validating a claim and not needing the claim are different levels of trust.

Which worker actually ran

This one never gets logged, because it doesn’t look like a failure. It looks like inconsistency. The reviewer was stricter on Tuesday than on Friday.

With four workers across three harnesses, each with its own prompt, tool set, limits, and model, the question of which configuration produced a specific result stops having an answer you can look up. The configuration never existed as a thing. It gets assembled at runtime out of a manifest, some command-line flags, whatever version of the harness is installed that day, and a model alias that can point at a different snapshot than it did last month. Read the config now and you get today’s answer to a question about last Tuesday.

So the runtime composes the identity before the worker starts, and the worker is never asked. Bhai assembles an effective configuration from the package ID and fingerprint, the role name, the tool profile as the intersection of what the package allows with what the role permits, and the workspace mode, then serialises and hashes it, so an unchanged setup gives the same digest every run and a changed one gives a different digest without anybody having to notice. It’s about three lines in each of the three adapters, which is how a run using the Pi CLI, Carbon, and Codex together still produces one comparable record.

Here’s what it was worth. One run recorded the harness at version 0.2, another the same afternoon at 0.3, a third six days later at 0.4, and by then the harness had a different name, because I’d renamed the coding agent from Gemma to Carbon. Inside one week the thing doing the work changed version three times and name once while I pointed it at the same repository, and the older runs still say Gemma.

The thing being identified is a folder: a manifest, an instruction file, and limits. No code, just data you can read and diff. A registry doesn’t make a run reproducible on its own. It makes the run answerable, which has to come first.

Shapes and budgets

A worker finishes and hands back something the next worker can’t use. In a pipeline, A’s output is B’s input, so a worker that returns three paragraphs of prose where the next step expected a structured result doesn’t fail, it propagates. If B were a function the wrong shape is a type error, caught before B does any work. B is a model, its input is a prompt, and a prompt accepts any string, so nothing in that handoff can raise. B interprets the nonsense and comes back with something confident and well-formed, built on nothing. The stall is the quieter version: nothing errors, nothing returns, the run sits, and with five workers running unattended nobody is watching.

The output contract belongs to the runner and fires inside the loop that counts attempts. The adapter returns a result, the runtime writes it and the trace to disk whether it passed or failed, and then it pulls exactly one fenced JSON block out of the worker’s final message and validates it against the shape the role declared. A failed validation is a failed attempt, so the contract is the thing the budget is counting: the contract decides, and the budget bounds how many times it gets to decide. In the stall trace, attempts two and three each came back after eight and a half minutes with the same reason, attempt four failed differently, and then it stopped at four, because the code says four.

Two limits. A harness that throws is never retried at any attempt count, because failing to start and failing to finish are different problems. And Bhai has no loop detection: a worker calling the same tool with the same arguments forever is invisible to the outer system, which can only refuse to wait past a budget declared in the package manifest before the run started.

Authority, in both directions

I don’t want a reviewer that’s supposed to be read-only quietly editing the code it was asked to inspect. When several agents share a machine they inherit the machine’s authority by default: the reviewer can write because the shell it runs in can write, the planner can push because I can push, the security reviewer can see production credentials because they’re sitting in the environment. Nobody granted any of that. Read-only written into a system prompt is a request.

There’s a run where the gate works and isn’t enough. The implementer tries to chain git add and git commit with an ampersand and gets denied three times, because the command gate refuses shell metacharacters outright rather than trying to parse them. Then it runs the two commands separately, both allowed, because commit is on the allow list. The gate did what it was written to do and the model still owned the commit that became the candidate. That’s the run that pushed me into deriving candidates from Git.

The placement matters, and it’s the opposite of the candidate gate.

I went looking for the same thing on the reviewer steps and found no log of a reviewer being denied a write across all twenty runs. The reviewers didn’t earn that by behaving well: Codex runs with a read-only sandbox pointed at a read-only clone, so the capability was never there. Denying an action is a rule you write; never granting it is infrastructure you build, and the second one is stronger. It’s the same argument as agent sandboxing, applied to workers instead of a single loop.

The mirror took a second workflow to find. Authority can be too little. A triage run fans out to three evidence gatherers, and the one told to read the repository’s history came back empty. It couldn’t have done anything else: the evidence roles are granted exactly one tool, file read, so no Git and no shell. It read five source files and gave up, and it was the slowest branch, so it set the critical path while contributing nothing. The join gate passed it, because the gate checks that each lens returned the label it was dispatched with rather than that it found anything. A well-formed empty result from a worker that never had the capability is indistinguishable from a genuine nothing to report.

What survives when one branch dies

The design post put this up as a thought experiment and said the build would settle it.

Work runs in parallel, one branch dies, the other is fine, and keeping the good one is not the default. The review that succeeded finished inside a process, and its result is in that process’s memory, or a pipe, or a scratch directory the run is about to tear down. What failed is the parallel block, and when the block fails the run fails and everything it was holding goes with it.

There are two wrong answers. Treating one completed review as good enough quietly turns a two-reviewer design into a one-reviewer design without telling anybody. Throwing the whole run away makes the reviewer that already finished correctly do it again against a commit that hasn’t changed.

Part of that is manufactured, and I should say which. I froze the security reviewer’s process by sending it a stop signal at spawn so it could never finish. Everything after is the real system: the real timeout firing at ten minutes, the real adapter error, the real failed-close path, no edits to the manifest. I forced it because setting a tiny timeout is a coin flip on which reviewer finishes first, and in half my earlier runs the security review came back first.

Restarting is cheap to build and expensive to run, since every step runs again and you need no infrastructure. Resuming is cheaper to run and needs somewhere durable to have put the finished work. So I ran resume, changing nothing except that this time I didn’t freeze the reviewer.

Served from disk isn’t a flag, and it isn’t the engine remembering anything. Before the runner spawns anything for a step, it asks six questions of what’s already on disk. The fifth is the same output-contract check from earlier: a stored result has to clear the bar it cleared when it was live, every time it’s reused, which is what stops somebody editing a file on disk and smuggling a result through. If any one misses, the step re-runs. It fails closed, so doubt is never a reason to skip paid work. What convinced me is a file that isn’t there: in the snapshot taken before the resume, the security step has its input and its resolved config and no result file.

The design post drew a line between keeping a file and safely resuming from a checkpoint, so I owe some precision on which this is. There is no mid-worker resume; the security reviewer started from zero. Resume needs the run’s workspace clones still on disk, and deleting them fails loudly. And resume isn’t a repair: if the condition that broke the step is still there, the step breaks again. What it buys is that work which already finished correctly survives a failure next to it, which here meant paying for one step instead of four. When I recorded the design video, the lifecycle box said no general pause, resume, or recovery; resume landed between the two videos.

The cap that cost me

This one has two halves: the guard working, and the price it charges. The second is where I got it wrong.

The failure is work that never converges, or a worker that decides on its own that it’s done. A reviewer finds a problem, the implementer revises, the reviewer finds a problem with the revision, and that loop has no natural end while every individual step behaves correctly and the budget drains. The other direction is worse: a worker concludes the work is finished and takes the terminal action, merges, deploys, closes the ticket. Both come from leaving “are we done” to a participant.

So none of it is left to a model. The revision cap is one, and it’s structural: ask for a second revision and the code raises. There is no prompt anywhere saying please only revise once. The handoff forces approval status to pending regardless of what the caller passes, and refuses to write anything if any workspace the run created has a Git remote configured.

Now the cost. If the loop is capped at one revision, the revision is always the last thing that happens. Review, change the code, end the run. Nothing looks at the last thing that touched the code.

The run where this bit me is the one that built the live dashboard I use in the video. Bhai ran it with the Pi CLI as implementer and two reviewers on different harnesses, and the reviewers did good work: five findings, four of them blocking. Then the implementer revised to address them and introduced two syntax errors, an unterminated f-string in the server and an invalid assignment in the front-end script. The server file couldn’t be imported. The code the reviewers were given parses fine, so both errors were created by the revision, and nothing reviewed the revision.

The gate passed it anyway, because nothing it checks has an opinion about whether the code runs.

The lineage gate checks that the commit exists, descends from a base, that the declared diff matches the real one, that the tree is clean. All true. The test suite passed too: the task allowed one command, pytest, and pytest exited zero, also true and also not a check on this, because the test file imports the reducer module and never imports the server. The broken file was never loaded by anything in the entire workflow, and I didn’t notice for four days, until I tried to run it.

Two things come out of that, and neither is specific to my prototype. A review that never executes anything is a reading rather than a check, and the correctness reviewer made twelve tool calls, all of them file reads. And a passing suite told me nothing because the suite never touched the file that broke, which is the ordinary coverage argument arriving somewhere I didn’t expect it, and it connects back to what evaluation is for.

What the system got right is that the candidate package records that the final revision wasn’t independently reviewed. It wrote down its own blind spot in the artifact it hands to a human, approval pending, nothing merged or deployed or pushed. The design post argued that provenance earns its keep partly by exposing gaps a polished final answer would smooth over, and this is the run where that paid off. What it got wrong is that the known-limitations list came back empty: it flagged the structural gap and missed the specific one.

My mistake in the design was collapsing two properties into one knob. Bounded and reviewed aren’t the same thing, and I treated them as the same because at a cap of one the revision is terminal anyway. Separate them and the blind spot goes: raise the cap to something like five and exit when a review comes back with nothing blocking. That would have caught the syntax errors on the next review, at a cost of up to five extra review rounds, which is real money and real wall clock. I picked the cheap version to keep the prototype cheap. The cheaper guard is to make the gate require that verification actually touch the files that changed instead of accepting any exit code zero.

Seeing any of it

With one agent you read the chat log. With five workers across three harnesses, some in-process and some subprocesses, there’s no single log, and the questions I care about aren’t in any of them. Which commit did the second reviewer inspect? Did the failed branch get cancelled, or never start?

Every one of those is about the space between two workers, and no worker was there for it. A branch cancelled before it started wrote nothing, and writing nothing looks exactly like a branch that was never scheduled. What needs recording is the decision, and decisions don’t happen inside workers, so putting all the log files in one place doesn’t fix it.

So there’s one append-only stream that the engine, the runner, and each adapter all write into. None of them chose where to write: the runtime builds one sink and hands the same one to all of them, which is why the order in that file can be trusted instead of reconstructed afterwards by merging separate logs and hoping their clocks agreed. It takes a threading lock rather than an async one, because the Carbon adapter runs its worker on a separate OS thread and a parallel block has two branches writing at once.

The same run reads three ways. Raw, it’s 253 events in a JSONL file, complete and unreadable. Rendered, it folds 243 worker-event lines away and shows the three that carry a reason. Pushed into OpenTelemetry, it becomes a trace in standard tooling I didn’t have to build, with the two reviewer bars starting and ending together and consolidate beginning only after both close. None of the six failures above would have been diagnosable without this layer.

One note for anyone wiring this up: standard observability tooling assumes a long-lived server, and an agent run is a process that starts, works, and exits. Almost every default I reached for measures change over time, and nothing here does. My dashboard rendered nine empty panels twice before I accepted that a short-lived run produces a value rather than a time series.

The limit I named in the design post held. Observation is built and control isn’t. Bhai can watch a worker call the wrong tool and can’t pause it.

What generalized

The fair question is whether this is an orchestration engine or one hard-coded pipeline with good error handling. I only found the boundary by writing a second workflow.

The machinery generalized. Triage reused the engine, the adapters, the permission model, the event stream, and resume without me touching any of them. The artifact types didn’t: triage needed eight new ones plus its own join gate, and those live inside Bhai rather than in the workflow file, because a gate has to know what it’s looking at. So a new workflow in a domain Bhai already knows is cheap and a new domain is a change to the runtime. I don’t think that’s fixable, because the alternative is letting a workflow ship its own gates, and then the guarantee is only as strong as whoever wrote the workflow.

The same triage blueprint run against three different issues produced three shapes: a question took one step and went back to a human, a feature request took two, a bug took five with a three-way fan-out. Same registry, same adapters, same permission model, same event stream, and the thing choosing between them is a Python dictionary. The model decides what kind of issue it’s looking at and the code decides what happens next. On the other axis, swapping the implementer between harnesses is one package reference, with the workflow, the roles, and the gates unchanged.

Against the six questions the design post left open: external orchestration did contain a weak worker’s mistakes without the worker noticing. Cross-harness review produced two security findings on the dashboard run that the correctness reviewer never touched, which is one data point rather than a study. What survives a dead branch is the completed branch, its validated result, and the workspaces. A thin local runtime replays completed steps, can’t resume a worker mid-flight, and needs the workspaces to still exist. Folder-defined packages do make workers interchangeable at the granularity of a whole step. And caps, gates, permissions, and terminal state belong in code, with judgment left inside the steps.

The question I’d ask of any of these systems

When you’re looking at a setup that runs several agents, the useful move is to skip the agents and look at what happens between them. Who decides the next step runs. Who remembers that the first reviewer already finished. Who holds the revision limit. Who is allowed to say the work is done.

If the answer to any of those is a model holding it in context, then the guarantee is that the model still remembers on a bad day.

The system I built didn’t produce correct code every time. It shipped a dashboard that won’t import. What it did was contain that, write down that the last change hadn’t been reviewed, refuse to merge anything, and stop for me. The next thing I want to fix is the one this run exposed, which is separating the revision cap from the review requirement so nothing ships unread, and then finding out what a second domain costs when the gates have to be built for it.