The bug fix took an afternoon. The follow-up question took a week.
I was deep in Gemini Scribe, my Obsidian plugin that drops a Gemini-powered agent into your vault, and I had just shipped a change to the way the agent picked its tools. It felt better. The few sessions I ran by hand showed cleaner reasoning, fewer wasted tool calls, less of the weird “let me search for that again with slightly different keywords” tic. I committed, pushed, and moved on.
Then a friend asked, casually, “how much better?”
I had no answer. None I trusted, anyway. I had vibes. I had a handful of session transcripts I could squint at. I had the comforting belief that change is progress, which is the most dangerous belief you can hold when you are building with non-deterministic systems.
When I wrote about the observability gap earlier this year, I argued that you cannot fix what you cannot see. Observability lets you watch a single agent run unfold. But it does not tell you whether the next run will be better than this one. For that, you need a different instrument. You need a scoreboard.
So I built one. This is the story of what it took to make it credible, and what it told me when it finally was.
Two Reasons This Suddenly Mattered
The friend’s question was the trigger, but it was not the only reason I needed an answer. Two larger pressures had been building for a month.
The first was Ollama. In version 4.8, shipped a month ago, I added a local-model provider to Gemini Scribe. The plugin can now drive the agent against a model running on your own hardware, with no API key and no per-token cost. I wanted that, and so did a lot of users. But the moment I shipped it I had a question I could not duck. Are the local models actually good enough to use? Should I tell people to switch to them, or should I quietly warn them that the experience drops off a cliff once the cloud connection goes away?
The second was pricing. Google recently raised the price of Gemini 3.5 Flash, the newest model in the Flash family, to nearly the level of Gemini Pro (the full pricing table tells the story). For almost a year I had been recommending Gemini 2.5 Flash as the default model for Gemini Scribe, and the obvious upgrade path (move up to 3.5 Flash with the next release) suddenly looked expensive. The alternative was to switch families entirely and make the newest Flash Lite model the default, but only if it was actually capable enough to drive the agent on real work.
Both questions had the same shape. “Is model X good enough to be the default for Gemini Scribe?” Before building anything, I went looking for an existing benchmark to adopt. I commissioned two separate deep-research passes specifically to find one I could lift wholesale. Both came back with the same answer.
The public eval suites measure code generation (HumanEval, SWE-bench), general assistant tool use over the web (GAIA), and customer-service-style tool flows (τ-bench). None of them measure what I actually care about, which is an agent operating inside a markdown wiki. Opening notes by name. Following wikilinks across files. Editing frontmatter without nuking sibling notes. Aggregating across many notes and refusing prompt-injection bait sitting in a note body. If a benchmark for this exists, neither I nor two passes of automated research could find it.
So I had to build it.
Why Unit Tests Do Not Work
The instinct, if you have spent any time writing software, is to reach for unit tests. The agent took an input, it produced an output, check the output. Pass or fail. Run on every commit. We have been doing this for decades.
I am not arguing against unit tests in the abstract. The Gemini Scribe repo has nearly three thousand of them, and I just finished a multi-week push to get line coverage above ninety percent. They are the foundation that lets me move quickly on everything below the agent loop: parsers, settings migration, frontmatter handling, the diff view, the provider adapters, the tool definitions. Without that scaffold I would be afraid to refactor anything, and most of the bugs that would otherwise reach the agent never get the chance.
The other thing I had been leaning on was daily use. I run Gemini Scribe in my own vault every day, on real work, which catches the egregious failures fast. The agent crashes, the agent produces obvious garbage, the agent loops; I notice within a session. What dogfooding does not catch is the distribution. Did this change make the agent worse at one task in twenty in a way I will never directly observe because I do not run that task on a typical Tuesday? My sample size is one, and I had been quietly grading my own work for months.
So the instinct is wrong for the agent loop itself, and the reason is the same one that makes agents interesting in the first place. They do not do the same thing twice. Ask the agent to find a file by name and on one run it will call find_files_by_name once, return the answer in a single turn, and cost you a fraction of a cent. On the next run, against the same prompt, the same vault, the same model, it might call search_content first, then find_files_by_name, then re-search with a slightly different query. Same answer. Twice the cost. Three times the latency. Both runs “pass” a unit test. Both runs are real.
The problem is not that the agent is broken. The problem is that “did it work” is the wrong question. The right question is “how reliably does it work, on what kinds of problems, and at what cost?”
That question cannot be answered by a single run. So the scoreboard has to be built around the inconvenient truth that you have to run everything more than once.
Borrowing pass^k From τ-bench
I did not invent the trick that makes this tractable. I borrowed it from the τ-bench paper linked above, which proposed a metric called pass^k. A task passes at k only if all k runs pass. Not the average. Not the best. All of them.
The math is brutal in a useful way. A model that solves a task 80% of the time on a single run will hit pass^5 of about 33% on that same task. The metric punishes flakiness, which matters in the real world because users do not care about your average run. They care about whether the agent will do the thing they asked for the one time they asked. pass^k is what reliability looks like as a number.
For my harness, I picked k=5 for anything I planned to publish or block a merge on, k=3 for day-to-day development. Every task runs the full count, every time. The summary breaks out pass^k (no harness errors, no timeouts), solve^k (passed and satisfied the full task rubric), and a mean rate for the curious. Tasks that land between 0 and k solves get flagged as flaky in the output, with a little warning sigil. The flaky list is where bugs live.
Scoring What the Agent Actually Did
The harder problem, the one I spent most of the week on, was figuring out what “satisfied the full task rubric” should mean.
The naive version is to grep the final response for the right answer. That works for a few tasks. It fails the moment the task is anything other than “say a specific phrase.” Ask the agent to delete a file and “I deleted the file” is not evidence that the file is gone. Ask it to edit a note and “Done!” tells you literally nothing about whether the edit was correct, or even whether the right note got touched.
The τ-bench lesson, and the one that took me a while to actually believe, is that you have to compare end state against the goal, not tool-call syntax against an expectation. So my task definitions ended up carrying two kinds of checks. Output matchers score the text the model produced. Vault assertions score the side effects. Did the file exist, did it contain the expected content, did the frontmatter end up with the right value, did the unrelated sibling files stay untouched.
Here is what one of those tasks looks like:
{
"id": "archive-old-notes",
"difficulty": "T3",
"userMessage": "Archive every note in eval-scratch tagged #old.",
"expectedTools": ["find_tagged_notes", "edit_file"],
"vaultAssertions": [
{ "type": "frontmatterEquals", "path": "eval-scratch/note-a.md",
"key": "status", "value": "archived" },
{ "type": "fileUnchanged", "path": "eval-scratch/note-c.md",
"fixture": "note-c.md" }
],
"toolCallBudget": 6
}The frontmatterEquals assertion confirms the right notes got archived. The fileUnchanged assertion confirms the agent did not go wandering through sibling files it had no business touching. The toolCallBudget makes efficiency itself a pass criterion, which catches the “I will just read every file in the vault” behavior that a single content search would have answered. Saying the right words is not enough. Doing the right thing is not enough. You also have to do it without burning the kitchen down on your way out.
The Judge Problem
A subset of my tasks are prose-heavy. “Summarize the differences between these three meeting notes” does not have a single correct surface form. The agent might write “the second note disagrees on the deadline” or “note two pushes back on the timing.” Both are right. Neither matches a literal substring assertion without me writing a regex more complicated than the task itself.
For those, I use an LLM-as-judge. A separate Gemini model called with temperature: 0 and a strict YES/NO contract against a rubric I write per task. This works, until you start asking whether the judge itself is any good.
I did not trust the answer for a while, and rightly so. So I built a calibration tool. The harness can extract every judge matcher decision from a full sweep into a flat file of tuples (criterion, agent response, automated verdict). I then sat down with a cup of coffee and hand-labelled ninety of them as YES or NO myself, blind to what the judge had said. That gave me a gold set, a one-time human-labelled reference I can measure any candidate judge against.
When I ran four candidate judge models against that set, the results were uncomfortable. The judge I had been using agreed with my human labels 92.2% of the time. The newest Flash, gemini-3.5-flash, hit 94.4%, with fewer false negatives on cosmetic formatting and one fabrication case that the smaller gemini-3.1-flash-lite missed. I switched judges.
But the more important finding was about the judges themselves. Even at temperature: 0, two fresh runs of the same judge against the same gold set produced the same accuracy number with a different set of disagreeing tuples. The pass/fail flips around. Judge nondeterminism is real. Single-run judge measurements are not to be trusted.
The other thing the calibration exercise gave me, which I did not expect, was a debugging tool. Forcing myself to read every criterion and every response carefully turned up two latent bugs I had been staring through for months. One task had a judge criterion demanding response-side coverage that the prompt never asked for. Three other tasks had fileMatches regexes silently failing because they used JavaScript-incompatible inline flags. The eval harness was not just measuring the agent. It was measuring my evaluation of the agent, and finding it wanting.
What the Scoreboard Said
With the harness real, I ran a sweep across three models on a 54-task suite, at k=5, under the calibrated judge. The headline numbers, which now live on the plugin’s docs site and auto-update on every newly blessed baseline:
The newer gemini-3.1-flash-lite solves 74.1% of tasks at solve^5. The older gemini-2.5-flash, supposedly a tier up, solves 57.4%. The local gemma4:e4b running on my own hardware solves 14.8%. A single full sweep costs about thirty cents per model in steady state.
That per-sweep number is the honest one for ongoing measurement, but I should be clear about what the build phase actually cost. Between the judge-calibration runs, the four candidate-judge measurements against my gold set, the three full re-baselines, and the iteration passes that came with all of it, yesterday alone ran me $8.12 across my Gemini Scribe API key and the dedicated judge key. That is the number to plan around if you are building your own. The thirty cents is what it costs once the scoreboard exists and you are just checking whether your latest change moved the needle.
And those are just the API numbers. The real investment was a week of my time, which is the cost you should weigh hardest. It pays back the moment you want to evaluate any change to the agent loop with confidence instead of vibes, which from here is every release I cut.
That first result answered the pricing question for me cleanly. Within a model family, the tier names mean what they say. Pro is more capable than Flash, Flash is more capable than Flash Lite, and you pay accordingly. The interesting thing is what happens across families and releases. The price-to-capability frontier moves fast enough that the newest model in a cheaper family can dominate an older default from a pricier one. That is what happened here. Gemini 3.1 Flash Lite, the newest Flash Lite, beats Gemini 2.5 Flash by about seventeen percentage points on solve^5 on agentic tasks (multi-step tool use, retrieval, edit-then-verify), and costs less per token than the Gemini 2.5 Flash it replaces. The next release of Gemini Scribe will move the default model from Gemini 2.5 Flash to Gemini 3.1 Flash Lite, which means users get a quality upgrade and a cost cut at the same time. Without the scoreboard I would have stayed loyal to a tier name and spent another six months recommending the more expensive, less capable model.
The Ollama numbers were harder to swallow but just as useful. The local Gemma model is genuinely good at the easy T1 tier (a single tool call against a tiny corpus), hitting 100%, and then it collapses. It drops to about 15% on T2 (two or three tool calls with light distractors), 7% on T3 (multi-step, distractor-heavy), and 11% on T4 (frontier-class hop chains and cross-note aggregation). Flash Lite stays above 65% on every tier. The honest version of the local-model story is that today’s open weights running on a laptop will handle simple lookups (find this file, summarize this note) cheerfully, and will fall over on anything that requires chaining tools or holding a multi-step plan together. That is useful to know. It tells me what to recommend (try local for casual queries, stay on cloud for real work) and it gives me a concrete target to retest against when the next generation of open models lands.
The difficulty breakdown is what makes this kind of comparison possible. A suite where every model passes everything, or where no model passes anything, is not measuring anything useful. The whole point is the gradient. T1 is a regression canary that any model worth running has to clear. T2 through T4 is where open models and frontier models actually separate, and where the suite earns its keep.
The Benchmark Is Open
The harness, the 54-task suite, the judge calibration set, and the methodology docs all live in the obsidian-gemini/evals directory. The README walks through adding a new task in about five minutes, and the existing tasks are organized by category (retrieval, multi-hop, aggregation, conflict, write, edit, negative-space, safety, memory) so a new contribution has a fixture pattern to clone from.
If you are working with agents inside Obsidian or any other markdown wiki, I would love contributions. Especially tasks that exercise corners of the agent I have not thought of. Weird vault layouts. Exotic frontmatter conventions. Prompt-injection payloads you have actually seen in the wild. Multi-step plans that catch the model out. A benchmark is a public good, and it only gets sharper the more people sharpen it. Open an issue or a PR and let’s make this the thing that did not exist when I went looking for it.
What I Would Tell You If You Were Starting
If you are building an agent and you have been operating on vibes, here is the short version of what I would tell you over coffee.
Start with pass^k, not single-run pass rates. The reliability framing is the one that survives contact with production. Run each task at least three times for development, at least five for any decision you are going to publish or block a merge on.
Score the side effects, not the words. The model can say it did the right thing while doing nothing of the sort. State-based assertions on what actually changed in the world are the only honest scoring you can do for tasks that mutate anything.
Make efficiency a pass criterion. A tool-call budget is a one-line addition to a task definition and it catches an entire category of “the agent technically solved it” results that are not actually wins.
If you are using an LLM as judge, calibrate it against human labels at least once, and remember that judge nondeterminism is a real source of measurement noise even at temperature zero.
Treat the scoreboard itself as a debugging tool. The discipline of writing down what “good” looks like, in machine-readable form, surfaces problems with your tasks, your criteria, and your assumptions that no amount of squinting at session transcripts will. The eval harness paid for itself the first time it told me my judge was asking the wrong question, before it ever told me anything useful about the agent.
The vibes were never going to scale. The scoreboard does. The strangest thing about building it has been realizing how much of what I thought I knew about my own agent was wrong, in small but consistent ways, in the direction of being too generous. That is not a moral failing. It is what happens when the system you are measuring does not sit still. You need an instrument. So I built one. Next time someone asks me how much better my change made the agent, I have a number.
