The model is not the moat: how Pavo captures a team's tribal knowledge

Ask why a production number looks wrong and the answer may cross 4 places: a branch in code, a filter in the analytics warehouse, a decision from an old experiment, and one caveat remembered by the person who built the system. No single source tells the whole story.

That connective tissue is tribal knowledge: the unwritten detail a team relies on to operate software correctly. General coding agents can investigate it for one task. Pavo is designed to preserve it so the next teammate—or the next AI agent—does not have to start again.

We call the result a book: a structured, searchable guide to the system's architecture, data, metrics, experiments, failure modes, and history. Every factual section keeps a route back to the code, query, table, dashboard, or document that supports it.

We tested that approach against Cursor and Claude Code on real project scopes. Because all 3 used the Sonnet 4.6 model family, the comparison focuses on the system around the model: how it explores, writes, saves evidence, and checks its work.

The result across 129 expert-checked facts

We evaluated 4 real project scopes spanning recommendations, content analytics, and notification measurement. Domain experts wrote a hidden checklist of 129 facts that a useful guide should contain. Each fact was backed by code, a data warehouse, or a document.

Pavo fully covered 24 facts, compared with 17 for Cursor and 15 for Claude Code. That is 41% more than Cursor and 60% more than Claude Code in this evaluation set. Pavo led on 3 scopes and tied Claude Code on the fourth.

The source trail moved in the same direction. Pavo named a specific source for 50 of the 129 facts, compared with 38 for Cursor and 34 for Claude Code. A specific source means a concrete file path, table or query, dashboard, experiment, or document—not a generic statement that the answer came from the codebase.

For the cleanest comparison, the Pavo result uses the same V8.1 generation workflow across all 4 scopes. Pavo has since kept that continuous-writer core and added stronger citation and repair checks. We do not credit the V8.1 score to guardrails added later.

Why the percentages are low

The headline is easy to misread. 18.6% does not mean that Pavo stated the other 81.4% incorrectly. This benchmark measures how much of a hidden checklist appears completely in one generated book. Most failures are omissions or incomplete coverage, not false statements.

Three properties make this a demanding recall test:

1. The checklist is hidden. The systems are asked to produce a comprehensive guide from a broad project brief. They are not given 129 questions to answer one by one.
2. Full credit requires every material detail. Across the 129 scoring units, the expert rubrics contain 329 explicit required points. 105 scoring units require between 2 and 5 details; missing one can turn an otherwise useful explanation into Partial.
3. The answer is distributed across different systems. Almost half of the checklist—62 of 129 facts—requires warehouse investigation, where the agent must discover the right tables and sometimes run the right read-only analysis rather than follow a named code path.

If complete and partial coverage are shown together, Pavo found at least some of 54 facts, compared with 39 for Cursor and 38 for Claude Code. We do not use that looser number as the headline because a missing condition can be operationally important. It does, however, show why the strict score should be read as complete checklist recall, not conventional question-answer accuracy.

Difficulty also varies sharply by project. Pavo fully covered 13 of 19 facts in the code-oriented Autoplay scope, but only 1 of 34 in the warehouse-heavy Sticky Notifications scope. The combined 24 of 129 is therefore a weighted picture of several very different research environments—not a universal measure of how often the underlying model is right.

The honest reading is still demanding: 24 of 129 is not a solved problem. The result says that Pavo's specialized workflow recovers more complete knowledge than the 2 general coding workflows in this evaluation, while leaving substantial room to improve—especially in analytics warehouses.

What did the test ask each system to do?

Each system received a project brief, access to the relevant code and analytics warehouse, and the same required 8-part structure covering topics such as architecture, data, metrics, experiments, failures, and history. The 129 expert facts stayed hidden until the books were finished.

The hidden checklist was manually annotated, not automatically extracted. For every fact, the benchmark authors reviewed the underlying source material and recorded the details a complete answer must include, interpretations it must not make, the supporting evidence, and a fact-specific scoring rule. Those human-authored annotations define how the labels are assigned.

An annotated fact might be the rule that chooses a recommendation, the exact definition of a metric, the table that feeds a report, the connection between 2 services, or a caveat from an experiment.

A separate AI judge—Gemini 3 Flash Preview, with randomness turned off—then applied those annotations by comparing each full book with one hidden fact at a time. Humans created the gold facts and rubrics; the AI judge assigned the per-book verdicts. The facts and scoring instructions were never shown to the systems generating the books.

The method shares an important instinct with SpreadsheetBench: test realistic work and inspect the artifact it produces, rather than grading how persuasive a response sounds. It also follows the coverage, groundedness, and source-quality framing in Anthropic's guide to evaluating AI agents.

A coding task and a knowledge book have different finish lines

A coding agent is usually done when it has answered a question, changed a file, or passed a test. Its investigation may be excellent, but the path it took is temporary. It can paraphrase a constant, skip a nearby branch that was irrelevant to the immediate task, or omit the exact table name and still complete the job.

Pavo is not done until that investigation becomes reusable knowledge. The book has to work for 3 audiences at once:

1. A teammate needs an explanation they can skim and trust.
2. An AI assistant needs exact file names, tables, metrics, and relationships.
3. Everyone needs a citation when the answer matters.

That makes the book the first place to look, not the final authority. If an answer is present and cited, a person or agent can use it quickly and inspect the source. If it is missing, they know to reopen the live system. The goal is simple: make more important answers available without hiding where they came from.

How Pavo builds the book

Connect to the sources the team actually uses

Pavo starts by checking which sources are available for that organization. A team using GitHub and BigQuery gets those sources. A team using Snowflake and dbt gets those instead. A data team without a code repository is not forced through a code-first workflow.

Depending on the team, the investigation can include code repositories, analytics databases, dbt models and lineage, dashboards, saved queries, experiments, pull requests, and project documents.

Keep investigation and writing together

The same AI agent that discovers a fact writes it into the book. Earlier versions split research, extraction, reconciliation, and writing across separate calls. Every handoff compressed the source a little more. Individual facts survived, but their order, conditions, and relationships often did not.

The current process keeps one continuous investigation. The agent takes inventory, follows promising leads, reads original sources, drafts sections, checks them, and repairs problems without discarding the mental model it has built. This follows the broader principle in Anthropic's guide to building effective agents: add coordination layers only when measured performance justifies them.

Use search for leads, then check the original source

Search can reveal an analyst's query, an old pull request, a dashboard definition, or a file the brief never mentioned. But search results and summaries are snapshots, not proof.

Pavo uses search to discover where to look, then opens live code and reads from the warehouse to establish what is true now. It begins broadly, then follows imports, database schemas, data lineage, and read-only query results into the details.

Save small sections with their evidence

Pavo saves one small section at a time instead of attempting the whole guide in one enormous response. Each section has a stable identity and a clear structure—paragraphs, lists, tables, code, notes, or quotes—so it can be checked and replaced without rewriting everything around it.

Evidence is stored with the section, not added as decoration later. A citation records the kind of source, a reference that can be opened again, and the relevant excerpt. A code citation must point to a file read during the run. A database citation must match a result that the run actually returned.

Check gaps and repair them before publishing

Before finishing, Pavo checks for missing citations, references that cannot be reopened, excerpts that do not match, important numbers or formulas without support, claims that say more than their evidence, and sections that rely too heavily on one kind of source.

Those findings go back to the same writer, which can collect more evidence and revise only the affected section. The finished book then powers both the human-readable guide and Ask Pavo, the question-answering experience over that guide. People and AI assistants work from the same shared artifact.

What this changes for a team

The benchmark measures fact capture, but the product value is reduced rediscovery. A maintained guide changes the starting point for common work:

The source breakdown is more useful than the headline

Pavo led the 2 general coding agents on facts backed by code, analytics data, and documents. But the shape of the result matters more than that clean sentence.

• Code was the strongest large source. Pavo captured 18 of 64 code-backed facts, and 37 included a specific source reference.
• The document result is too small to generalize. Pavo captured 2 of only 3 document-backed facts.
• Analytics data remained difficult for every system. Pavo captured 4 of 62 warehouse-backed facts; Cursor captured 1; Claude Code captured 2.

The warehouse-heavy Sticky Notifications scope is the clearest failure case. Pavo and Claude Code each captured 1 of 34 claims; Cursor captured 0. That is not a victory worth celebrating. It is a diagnostic.

Code gives an agent names, imports, and call relationships to follow. A large analytics database can contain thousands of plausible tables, multiple definitions of the same business concept, and important differences in how rows are grouped. Finding a relevant table is not the same as running the precise query that proves an operational fact.

That failure has already shaped the roadmap: broader catalog discovery, stronger use of live read-only queries, clearer limits on query cost, and citations that preserve the exact returned values behind a claim.

What this result does not prove

This is an internal product-development evaluation, not an independent public leaderboard. The result is useful evidence for improving Pavo, but it should be read with these limits:

• Each system has 1 selected strict run per project scope. Both generation and AI judging can vary between runs; repeated trials would make the comparison more reliable.
• The base model family is matched, but the surrounding workflows, tools, and source access differ. Those differences are the product being tested, but the result cannot be credited to one feature alone.
• The benchmark measures facts captured and sources named. It does not yet compare speed, cost, ease of use, or success on later tasks.
• The 4 project scopes come from 2 companies and do not represent every technology stack.
• An AI judge is useful but fallible. It can miss equivalent wording, penalize phrasing too harshly, or disagree with itself across runs.

Parlance Labs' study of automated evaluations found that automated reviewers can recover human-labeled failures and discover new ones, while still missing failures that require product or domain context. We therefore use the judge as a signal, not an oracle. People inspect the misses, form general hypotheses, run controlled comparisons, and keep the benchmark answers out of the book-generation prompts.

The larger lesson: durable context is a product capability

The usual enterprise AI debate asks which model can answer the hardest question. Organizations also need a system that turns an expensive investigation into reusable, inspectable context. Otherwise every agent pays the same discovery cost, repeats the same mistakes, and builds on a slightly different interpretation of the system.

Pavo's advantage is not simply that it can read a repository. Cursor and Claude Code can do that. The difference is that Pavo treats understanding as a shared product with a lifecycle:

• It connects to the sources each organization actually has.
• Research and writing stay in one continuous investigation.
• Original sources outrank convenient summaries.
• Small sections can be saved, checked, and repaired independently.
• Citations can be reopened by people and software.
• One guide becomes shared memory for teammates and AI assistants.

The benchmark shows that this specialization already matters. Its weakest result—warehouse-backed knowledge—also shows how far the system has to go. That is the standard we want for tribal knowledge: not a confident-looking document, but a cited and continually improvable guide to how a team's system actually works.

Benchmark notes

The evaluation code and hidden expert facts are private because they contain customer operational details. We publish the aggregate method, results, and caveats so the comparison can be interpreted without exposing that knowledge.