Better offline tests lead to more production wins

A product team can generate recommendation ideas much faster than it can test them. A ranking rule takes days to build. A reliable A/B test can take weeks, needs real traffic, and competes with every other experiment on the roadmap.

That makes offline evaluation valuable for a simple reason: it lets a team reject weak ideas before users ever see them. But this only works when the offline test predicts the metric the team wants to move in production.

In this project, the first offline benchmark did not. It rewarded the recommendation system already in production, disagreed with the first online experiment, and would have sent the team toward the wrong candidates. Randomized traffic gave us better evidence. We used it to rebuild the benchmark, replay more than a dozen policies, and find an offline measure that tracked production much more closely.

There were more ideas than A/B-test slots

The product was a high-traffic consumer feed. The team had a strong incumbent recommender and many plausible ways to improve it: broader intent signals, different candidate generators, personalized ranking, deliberate discovery, and combinations of those ideas.

The hard part was not producing another candidate. It was deciding which candidates deserved production traffic. Sending every idea into a live test would create a long experiment queue and expose users to approaches that could have been rejected cheaply.

A useful offline benchmark would turn this portfolio into a shortlist. A misleading benchmark would make the experiment queue worse by confidently promoting the wrong work.

The first offline test rewarded the system already in production

The initial evaluation used engagement from production logs as its answer key. That sounds reasonable until we remember how those logs were created: the incumbent policy chose what users saw, and users could only engage with those exposed items.

The benchmark therefore mixed up two different facts: whether an item was relevant and whether the existing system had chosen to display it. Unseen content looked like rejected content. New policies were graded using evidence collected by the policy they were trying to replace.

This is why the incumbent appeared strongest in the most policy-dependent evidence. When we moved toward evidence collected outside its usual choices, the apparent gap narrowed and sometimes reversed.

Randomization turned an A/B test into better evaluation data

The first online experiment did more than compare recommendation approaches. It introduced controlled variation into what appeared in the visible slate. That gave some items exposure they would not normally receive and made the resulting behavior less dependent on the incumbent policy.

We could now separate three questions that the original logs had mixed together: Was the item available? Was it shown? What did the user do after seeing it?

What “replay” means

For each logged session, we kept the candidate set that had actually been available. We then asked every candidate policy to rank that same set. If the item the user later chose moved toward the top, the policy received a better offline score.

This is often called counterfactual evaluation: asking what a different policy would have done in a situation we already observed. Here it was a practical replay, not a claim that we could perfectly predict every unseen production outcome.

We tested five definitions of success

Better data was only half the problem. The team also needed the right target. “Did the user engage somewhere?” and “Did this recommendation surface create meaningful consumption?” are not the same question.

We evaluated the same policies against five views of relevance, ordered by how strongly the incumbent policy influenced the labels.

The exercise changed the conversation. We were no longer asking for the single official offline metric. We were asking which combination of evidence and metric best predicted the production decision.

The best benchmark was the one closest to the product decision

The broad benchmark asked whether a policy could surface anything a user engaged with anywhere in the product. It was easy to compute, but it was a weak proxy for whether a recommendation improved the target surface.

The production-specific benchmark narrowed both sides of the comparison. Offline, it used engagement from the surface being optimized. Online, it used consumption created by that same surface. That tighter definition produced a much more useful policy ordering.

The improved benchmark still did not predict every position exactly. Two similar policies swapped second and third place. That was a useful expression of uncertainty: they were close enough that offline evidence could shortlist both, while the online test decided between them.

The benchmark became a filter, not a replacement for production

A benchmark that tracks production changes the order of work. The team can explore many approaches offline, remove candidates that fail obvious checks, and reserve live traffic for a small set with credible evidence.

Online experiments remained the source of truth. They measured effects that replay could not: new retrieval behavior, changes in user response, system latency, novelty, and downstream product outcomes. Offline evaluation made those experiments more selective; it did not make them unnecessary.

Better offline tests make every online test more valuable

The speed benefit is easiest to understand as experiment capacity. If a team can test only a few candidates at a time, screening the portfolio offline can remove entire rounds of live experimentation.

The important gain is not merely that one project finishes sooner. A stronger shortlist raises the chance that each online test contains a real winner. Teams spend less traffic rediscovering obvious failures and more traffic distinguishing between credible alternatives.

Every A/B test should improve the next benchmark

A fixed benchmark gets stale. Products change, users change, and teams start optimizing new outcomes. The benchmark therefore needs the same feedback loop as the production system.

After each experiment, we compare the offline prediction with what happened online. Agreements increase confidence. Disagreements are more valuable: they reveal a missing cohort, a weak label, a guardrail the benchmark ignored, or a part of the serving policy the replay could not reproduce.

The result is a Frontier Benchmark: not a frozen dataset, but a measurement system that keeps earning the right to guide production decisions.

What we learned

• Production logs are not neutral ground truth. They contain the choices of the policy that produced them.
• Randomization is measurement infrastructure. An exploratory arm can improve future offline evaluation, not only the experiment in which it runs.
• The target must match the product decision. Broad engagement was easier to measure but less useful than a surface-specific production outcome.
• Offline evaluation should shortlist, not declare victory. Close candidates should advance together when the benchmark cannot separate them confidently.
• The real goal is a higher experiment hit rate. Better offline tests help teams put stronger ideas into production and ship metric wins more consistently.

Limits

The exploratory traffic was partially randomized rather than fully randomized across every possible item and position. That reduced dependence on the incumbent policy but did not remove every source of bias.

Replay could rerank only items present in the logged candidate set. It could not estimate outcomes for unseen items or a completely different retrieval system. The production baseline was also not perfectly reproducible from every logged session.

Finally, the offline–online rank comparison contained four policies. The change from weak alignment to strong directional alignment was useful for choosing the next experiment, but it should not be treated as a general statistical result.

Those limits are why the loop matters. Offline evaluation decides what deserves a live test. Production decides what is true. The benchmark learns from the difference.