A domain agent is narrow enough to measure

Over this series we built a coding agent from scratch, and last episode a loop improved it. This time I gave it one real job: answering questions about my amateur cricket league from the league's database. It answered its first question correctly and delivered the second one confidently wrong, and the eval harness I built to tell those two answers apart is what this post walks through.

I play in an amateur cricket league called NECA, and the league’s stats live in a MongoDB database, the same cricket domain that supplied the running example in the memory post. So I stood up CrikIT, sharing its name with the platform I run NECA’s league and tournament management on, this time as a query agent: the same model and harness as the agent we built from scratch, plus a cricket-commentator persona in the spirit of Harsha Bhogle, plus read-only query tools over the league data. Ask it who has scored the most runs and it names the league’s top run-scorer, 2,075 runs, straight out of the database. Ask it for the highest individual score and it says 124, with full commentary-box confidence. The real answer is 152.

Both answers came out of the same agent, and the second one is the reason this post exists.

I think this narrow shape is where agents are going: small agents that know one domain, one database, one workflow, built with a general-purpose agent. That’s exactly what happened here. Gemma, the from-scratch agent from this series, is what CrikIT is made of, and CrikIT forks none of it: a system prompt, fifteen deterministic query tools plus a guarded fallback, and a read-only policy, handed to the same Agent class in about ten lines of configuration. For the rest of this post it’s the thing under test. Last episode was about improving an agent; this one is about measuring one.

A general coding agent is hard to grade: “write me a script” has a thousand acceptable outputs. A domain agent is different. “Who has the most runs” has one correct answer, and it’s sitting in a database I can query myself. Correct behavior is definable, which makes it measurable.

The vocabulary

The eval world’s mental model is five words. A dataset is a collection of items, and each item is a question I’d genuinely ask CrikIT. Each item gets metrics: did the answer match the database, did it call the right tool, how long did it take. Every run produces a score per metric, and tracking scores over time is how I tell whether a change actually improved the agent or just felt like it did.

My baseline dataset is 25 items, each run three times, on the same local Gemma 4 26B-A4B that powers the agent. The repeats matter because a single right answer can be a fluke; the second and third runs are what tell me whether the first was skill or luck.

One item is five lines of Python: an id, the question (what’s the highest individual score in a single innings), its category, the expected answer kind, the tool it should reach for, and a little function that computes the correct answer, the gold answer, from the database itself, so I never hand-typed an expected value that could drift out of sync with the data. Next to that spec sits what the harness recorded: 124 on all three tries, zero tool calls, 2,162 tokens per attempt.

Six kinds of good

Correct answers are only one slice of good. That 124 was wrong in a specific, dangerous way, and an agent can also be right but slow, right but expensive, right for the wrong reason, or wrong in a way that’s actually fine. What good means for CrikIT came out to six things.

Outcome first: did the answer match the database. Across the baseline CrikIT lands at 0.787, roughly 79 percent correct. That’s the headline number, and it hides more than it shows. Split by category: happy-path questions 0.92, adversarial and edge cases a clean 1.0, questions I already knew it struggled with 0.67, ambiguous questions 0.17, questions with no matching tool 0.33. Same agent, wildly different behavior depending on what you ask.

Trajectory second: which path did it take. When CrikIT reached for a specific tool, it picked the right one 93 percent of the time, and the trajectory data is what explains the confident 124: on that question, CrikIT made zero tool calls, on every attempt. Nothing was queried; the model guessed. The outcome metric said the answer was wrong, the trajectory said why, and the why is the part I can act on.

Cost third: every answer averages about 5,262 tokens and 6.4 seconds. Fine for a hobby league bot, a number you watch at volume.

Reliability fourth: CrikIT got the right answer at least once out of three tries for 0.84 of the questions, which the eval literature writes as pass@k, and every single time for 0.76, written pass^k. The gap between the two is flakiness, questions it can answer and sometimes doesn’t, and it’s the difference between an agent that knows something and an agent you can rely on to say it. Run each item once and you never see it.

Safety fifth: the dataset includes prompt injections, destructive requests, attempts to fish out personal data, and out-of-domain questions, and CrikIT declined every one, a clean 1.0. Part of that is architecture rather than model behavior: the tools are read-only against a frozen, redacted local snapshot, so no code path can modify anything, whatever the model decides.

And experience sixth: does it stay in character, is it clear, does it admit when it can’t find something, does it ask a clarifying question when the question was ambiguous. That last one produced the worst score in the entire eval, and I’ll get to it.

The dangerous kind of wrong and the safe kind

Accuracy hides the failure mix, so I broke every graded answer down by type: 59 correct, 11 confidently wrong, and 5 where it declined, telling me it couldn’t find the answer. The confidently-wrong fraction works out to a hallucination rate of about 0.15.

The 11 are the ones that worry me. The highest-score question is one of them: 124, delivered like a fact. Another: how many centuries have been scored in the league? CrikIT says zero, and there are plenty. A wrong answer delivered confidently is dangerous precisely because the user believes it; nobody double-checks an answer that sounds sure of itself.

The 5 declines are wrong in the safe way. I asked how many fifties had been scored, its query came back with nothing, and it said so instead of manufacturing a number; the eval literature calls that an abstention. On a scoreboard a decline and a confident wrong answer look the same. For a user they are completely different, and once you measure them separately you can start engineering toward the safe one.

Grading with code, judging small

Wherever plain code can decide, I let plain code decide: did the answer contain 2,075, did the agent call top_run_scorers, how many tokens did the run burn. Code checks are cheap, deterministic, and never change their mind between runs. One agent-side decision, made before the eval existed, makes them possible: every CrikIT reply ends with a small machine-readable JSON block carrying the answer’s value and kind. The prose above it is the persona, the JSON is for grading, and whether the block shows up correctly is itself a metric (it held at 0.973).

Some things code can’t reach: did it stay in persona, was the answer clear, did it ask which season I meant. For those I use a model as judge, with a deliberately small remit: four yes-or-no criteria against a rubric, with an allowed “unknown” so it never has to force a verdict.

Two caveats on the judge. The first: my judge is the same local model that powers the agent, grading its own homework, and a model will happily flatter itself. So before trusting any judge number, I calibrated it against 24 answers I hand-labeled myself. It agreed on all of them, across every criterion; the stats shorthand for that agreement is kappa, and mine came out at 1.0, which is the only reason judge scores appear in this post at all.

The second caveat turned into the finding that surprised me most. Persona, clarity, and admitting gaps all scored high. Asking a clarifying question when the question was ambiguous scored 0.03. Ask CrikIT something vague like “who’s the best player” and instead of asking best at what, it commits to an interpretation and answers with full confidence, roughly 97 percent of the time. It’s the same disease as the confident wrong answers, showing up in the experience layer: this agent would rather guess than ask.

The fallback writes a shopping list

The fifteen tools are deterministic on purpose: each owns exactly one correct query, the model only picks the tool and fills in parameters, so the query is right every time by construction. The guarded fallback is the exception, a find-and-aggregate capability where the model does author its own query, allowed only when no specific tool fits.

The eval harness knows every time the fallback fires. Tracing came free from the series, since observability is its own primitive in Gemma, but a raw trace only tells you a tool ran. So every CrikIT tool also carries labels the eval will need: at registration I declare its tier, domain or fallback, and at run time CrikIT stamps whether the result came back ok, errored, or empty. The eval harness lives entirely outside the agent and just reads those attributes. That’s the template I’d suggest for instrumenting your own product: the recording is free, and deciding which labels the evaluation will need is work you do while building the agent, much cheaper there than as a retrofit.

In the baseline the fallback fired on about 15 percent of answers, and the routing itself is clean: every time I expected the fallback, CrikIT used it, and it never fell back when a specific tool existed. The problem is what happens after it fires. Counting centuries and counting fifties have no dedicated tool, CrikIT drops to writing its own aggregation, and it gets them wrong; the uncovered category at 0.33 is this exact failure. So the eval is doing more than grading the agent: the fallback log is a shopping list. Every question that keeps hitting the fallback and failing is a candidate for the next deterministic tool, count_centuries, count_fifties, ranked by how often real questions fall into the hole.

The eval also itemized the fallback’s price. The centuries and fifties attempts each burned around 6,500 tokens on constructed queries and still came back wrong. The worst case was a question about the man of the match, where the league’s data is sparse enough that a decline or a clarifying question is the defensible answer. On two of three attempts the fallback turned that into a thrash instead: five constructed queries, six tool calls, nearly 29,000 tokens, more than five times the average answer, all to fail anyway. The third attempt declined cleanly in a single call. An escape hatch invites over-trying. I still want the fallback, since it’s the only reason some questions are answerable at all, but it costs determinism, tokens, and some clean refusals, and I only know the size of that bill because the eval itemizes it.

Chains, and breaking one on purpose

The dataset also has one multi-step question: who holds the highest-individual-score record, and across that player’s career, how many runs and how many wickets? Three dependent tool calls, and no planner module underneath, which I checked in the code before claiming: CrikIT runs on the plain drive loop, so the chaining is the model’s own behavior, and nothing structural guarantees it. Grading a chain pass-or-fail throws away where it breaks, so I grade with weighted checkpoints, partial credit per step. CrikIT chained the three tools and nailed it on all three runs, a mean checkpoint reward of 1.0, and when I injected a fault that forced the first call to return empty, it noticed, retried, and still landed the answer; the run log reads faulted true, retried true, recovered true. Put that next to the centuries failure and you get a real capability profile: strong at sequencing tools, weak at authoring queries. That is a far more useful sentence than “79 percent accurate.”

The numbers moved while I was editing

Then the scores-over-time story stopped being theoretical. While I was producing the episode, Gemma shipped an interface release, mechanism-only changes to the harness, with CrikIT’s domain code untouched. I re-ran the same 25 items, three times each.

The headline went up: 0.787 to 0.813. The mix underneath is messier. The abstentions disappeared, five to zero: the fifties question, my example of the safe kind of wrong, stopped declining and started answering 382, correctly, on two of three attempts, while the man-of-the-match question went the other way, all three attempts now thrashing through constructed queries without ever emitting the answer block. And the average answer got about 14 percent more expensive.

The cause was one plumbing change. CrikIT used to truncate every tool result itself, at 6,000 characters; the new release moves result budgets into the harness, so CrikIT deleted its own truncation. Same model, same tools, same questions. What changed is the shape of what the model sees when a query result comes back, and that was enough to flip a polite “I couldn’t find it” into four exploratory queries and a correct count, and a clean refusal into a 29,000-token thrash. The abstention behavior I’d measured at the baseline turned out to live in that plumbing: change how much of a result the model sees, and the politeness changes with it.

I would not have caught any of this by feel. The eval caught all of it because every results file is stamped with a fingerprint, the harness commit, the exact model, a hash of the dataset, so the same 25 questions ran on both versions and every score is comparable. Tracking scores over time is how you grade your own changes, and it turns out to also be how you catch behavior drift you didn’t ask for, from layers you don’t control. One caveat for the record: this release declared itself compatible, so the eval treated my old baseline as resumable, and behavior moved anyway. A compatibility promise is a claim about behavior; the re-run is the measurement of it. I happen to author both layers here and the drift still surprised me; most builders ride an agent SDK someone else ships, and the same rearrangement can happen under their agent without a line of their own code changing.

The fix, measured

The failing fallback questions had one thing in common: the model was authoring queries against a schema it had never seen. It guessed collection names, it filtered scoreCards for per-batter stats that actually live in a collection called battingInnings, and it burned tokens discovering all of that the hard way, on every attempt.

The fix used a primitive that has been sitting in the harness since the original series: Gemma auto-loads an AGENTS.md file from the agent’s root into the system prompt. CrikIT shipped with that loader deliberately aimed at a directory with no AGENTS.md in it, to keep repo instructions out of the persona. So I wrote one, a data map: which collection holds what, references are ids rather than names, a century is a hundred runs or more in a single innings, the exact field names. Then I pointed the loader at it. That is the entire change: no new tools, no model change, one existing primitive switched on with the right content in it.

Centuries went from zero out of three to a stable three out of three, fifties too, and both got cheaper: one constructed query and about 5,900 tokens per attempt, instead of the multi-query hunt that used to fail. On the full suite, outcome went 0.813 to 0.960, and three more questions came along for the ride, all downstream of the same blindness. The highest-score question, the 124 from the top of this post, answers 152 on every attempt now. The man-of-the-match thrash collapsed to a clean one-call decline.

The eval also flagged the bill. One question regressed: “How did we do this year?”, ambiguous on purpose, because nobody said which team “we” is. Before the schema map it held back on two of three attempts; after it, all three attempts confidently dumped the standings table. My read: the richer prompt, plus the instruction to prefer a specific tool for every question, made the agent more answer-eager. A fix shifts the behavior around it as well, and the eval is the only thing watching the whole board.

So the loop ran a second time. I added one paragraph to the same AGENTS.md: if the question’s subject is unspecified, don’t assume it and don’t answer with all of them, ask. Then the full suite again. It landed at 0.973, every one of the 25 questions right at least once, the ambiguous questions now ask, and the count I care most about, confidently wrong answers, went from eleven at the baseline to zero.

What’s still imperfect is one flaky question: man of the match over-tried on two of three attempts this run, the whole remaining gap between right-at-least-once at 1.0 and right-every-time at 0.96. Even that failure is the safe kind, tokens burned and nothing produced rather than something false. The judged pass to re-score the experience criteria on the new baseline hasn’t run yet, and the shopping list stands, since a dedicated count_centuries tool would still beat a well-informed fallback on determinism. But the loop ran twice, end to end: watch it fail, read the trace, change one primitive of the harness, re-run the same questions, and watch every number move, including the ones you didn’t mean to move.

Evaluation has a bill

One thing I didn’t appreciate going in. My baseline is small, 25 items times 3 runs is 75 answers at roughly 5,262 tokens each, call it about 400,000 tokens per full evaluation pass, and that’s a toy dataset on a local model. The EvalEval Coalition, writing on the Hugging Face blog, argues that evals are becoming the new compute bottleneck. The numbers they cite: about 40,000 dollars to run one agent leaderboard’s 21,730 rollouts, around 2,800 dollars for a single run of one agent benchmark, and a corner of scientific ML where evaluation compute now exceeds training compute by roughly two orders of magnitude. Their sharpest point is about who gets priced out: academic groups now hit the budget constraint before the technical one when they evaluate frontier agents independently.

So you have to be selective, and the levers are the same vocabulary from the top of this post: which agents deserve a real eval, how many items, how many repeats per item (reliability needs repeats, and repeats multiply cost), how often you re-run. I’d rather have 25 sharp items I run on every change than 500 items I can only afford to run quarterly.

Before you trust an agent

I spent a series building a general agent, and the most useful thing it has done so far is become a narrow one. I think that pattern generalizes: domain agents built by general agents, one per job, each with an eval harness so the person who built it can trust it and keep improving it without guessing. The proof layer was the tenth primitive on the original map; this post is what it looks like when it’s the whole job.

The habit I’m keeping from this exercise: before I trust an agent, I ask what kind of wrong it is when it’s wrong. An agent that says it couldn’t find the answer is a tool I can work with. An agent that says 124 with a straight face when the answer is 152 needs a harness watching it.

Still open: the man-of-the-match over-trying, the judged re-score of the experience criteria, and the tools on the shopping list. Every one of those changes gets the same 25 questions, and the results file each run leaves behind will carry the same fingerprint the baseline does.