Case study
Dashverse
Short-video entertainment · recommendation systems
One improvement loop lifted Shortfree Autoplay activationby +3.1pp, live in production
Pavo built an offline-eval engine to screen ~50 candidate policies without live traffic, then shipped the variant that lifted Autoplay activation +3.1pp in production.
- Surface
- Autoplay (Shortfree)
- Target metric
- Activation rate
- +3.1ppAutoplay activation · live in production
- +2.6→+3.1ppOffline prediction vs live result
- 5Counterfactual estimators · DR locked
- ~50Candidate policies screened offline
The problem
Autoplay was under-activating, in two fixable ways
Shortfree's Autoplay picks the next show for a viewer, and one number says whether the pick was right:
- 01Activation—did the viewer open the surfaced show and finish it that day.
- 02A v3 bandit made that pick in production. It worked, but it was quietly under-activating, and closing the gap meant first finding where the activation was leaking.
Two causes came out of the analysis:
- 01The policy kept re-serving shows a viewer had already finished, which suppressed activation instead of driving it.
- 02It ran one exploration strategy for three very different viewers—cold, medium, and mature—spending exploration on cold users, where it barely helps, while exploitation does little for the mature ones.
A single team owned the fix, without the bandwidth to work through candidates fast enough.
The approach
Pavo ran the full improvement loop on one surface
Six stages: learn the system, find the gap, build the measurement, generate candidates, screen them offline, and ship only what earned live traffic.
- 01
Onboarded system knowledge
- An onboarding agent read Dashverse's warehouse—tables, events, metrics—and the reels-recsys repository.
- It compiled both into a structured, queryable System Knowledge Book.
- Every claim carries evidence, provenance, confidence, and freshness.
- 02
Identified the gap
- Clustered users and shows against activation as the target—user × show, show × show.
- Turned raw production noise into a ranked matrix of where the policy was losing activation, and for whom.
- 03
Built an offline-eval engine
- Before any A/B test, we built a way to score any policy on logged data, so live traffic only ever goes to likely winners.
- A random-exploration arm logged a golden dataset—context → action → reward, with known p(shown).
- Five counterfactual estimators scored any policy on that log, no live traffic needed.
- Doubly-Robust matched the live answer-key best, and was locked.
- 04
Generated ~50 candidates
- Three families: heuristics on top of v3, a Thompson-sampling + LGBM bandit, and a deep MLP over four predicates.
- Roughly fifty candidate policies aimed at the gap—not a bet on a single idea.
- 05
Screened offline, ranked to ship
- Every candidate replayed on the locked dataset, then ranked by offline lift, model simplicity, and implementation cost.
- Survivors shipped as plain pickled files with an encoded surrogate—no heavy serving infrastructure.
- 06
Shipped the winner
- Survivors were promoted to live A/B.
- On Pavo's recommendation, Dashverse stopped re-serving already-watched shows.
- Rolled out to all users on Jun 4.
Gap identification
One policy, three different viewers
Pavo's gap-identification pass clustered users and shows against activation—user × show, show × show—and turned raw production noise into signal. The shape it surfaced: one exploration budget can't fit three viewer types that behave differently.
Cold
v3 over-explores. New viewers want proven hits; exploration is largely wasted on them.
Medium
About where v3's single budget sits—the one cohort it roughly fits.
Mature
v3 under-explores. Mature viewers have seen the hits, so exploitation earns little—they need discovery.
Budget positions are illustrative; the point is that the right exploration budget differs by cohort while v3 applied one to all. The same read also exposed the re-serve bug—already-watched shows being surfaced again, suppressing activation.
Offline evaluation
Before any A/B test, we built a way to score any policy on logged data—so live traffic only ever goes to likely winners
The random arm's logs became a locked golden dataset with known p(shown). Every policy is replayed against it and scored by the panel below; Doubly-Robust matched the live answer-key best, so it was locked as the decision estimator.
| Estimator | What it does | Why it matters |
|---|---|---|
| Replay | Averages reward on rows where the new policy picks the same show | Simple, but discards most of the log |
| IPS | Reweights every reward by 1 ÷ how likely that show was | Uses every row; high variance |
| SNIPS | IPS divided by the total weight | Same idea, far less noisy |
| Direct Method | Trains a reward model, asks it what the new policy would score | Low variance; wrong if the model is wrong |
| Doubly-Robust | Model prediction plus an IPS correction on its errors | Right if either part is right—the one we locked |
Known p(shown) from the random arm is what makes counterfactual scoring valid—every downstream number traces back to that locked table. The panel above is representative; the production dataset carried 144 features against each grain.
Validation
The eval was checked against history before it screened anything new
Before trusting the offline engine on new ideas, Pavo replayed Dashverse's own past production launches through it—by viewer cohort—and confirmed the offline scores reproduced what those launches actually did online.
Offline DR prediction· online actual
| Past launch (replayed offline) | Overall | Cold | Medium | Mature |
|---|---|---|---|---|
| V3 → V8 (random arm) | −6.2−7.5 | −7.0−6.5 | −6.0−8.2 | −5.9−5.9 |
| V8 → V9 | +6.9+5.8 | +11.7+6.5 | +5.8+9.3 | +5.5+4.8 |
| V3 → V9 | −0.8−1.7 | +1.00.0 | −0.6+1.1 | −1.5−1.2 |
Each cell: offline replay (DR) prediction over the launch's actual online result, by cohort—indexed, not percentage points. The predictions recover the known ordering of launches Dashverse had already shipped—the random arm costs activation, V9 recovers it—which is what earned the eval the right to screen new candidates. Cold users move most, where the policy has the most to learn.
Exploration
A suite, tried in parallel—offline eval decides who gets live traffic
Pavo didn't bet on one idea. Roughly fifty policies were generated across three families, then every one was replayed on the locked dataset. A few clearly separated; those advanced to A/B.
- 01
Heuristics on v3
- Rules layered on the v3 bandit
- v3 minus already-watched
- Drop shows the viewer has already finished
- + rule variants: cold-explore caps, recency
- 02
Bandit + GBM
- Per-user personalization
- Thompson-sampling + LGBM bandit
- Trained on v8-arm data—the personalization layer is the key
- + tuning variants
- 03
Deep model
- Learned representation
- MLP · 4 predicates
- user × session × show × past-show, on v8-arm data
- + architecture variants
How they scored · offline replay (DR)
MLP · 4 predicates (v8-arm)
Thompson sampling + LGBM (v8-arm)
v3 minus already-watched
Explore-less on cold (rule)
Popularity re-rank (rule)
Bandit variant
Offline activation lift (DR estimate) · illustrative ranking; ~50 policies were tried, the top few advanced.
The screen was unforgiving: near-clone pickers were rejected (users reject a show they just watched), popularity-maximizers served the same show to everyone, and same-trope 'continuity' picks did worst. What survived was affinity with guardrails—skip already-watched and near-duplicate shows—and the MLP reranker.
Screening
Picking the winner which earns the right to be shipped
Winning offline isn't enough to ship. Pavo ranked the survivors by performance first, then by how little could go wrong in production—which is why a one-line fix went live ahead of the higher-scoring MLP.
- 01
Best on offline eval
Start from the offline winners—performance comes first.
- 02
Prefer the simplest model
Fewer moving parts wins the tie-breaks.
- 03
Lowest implementation cost
Favor what can go into production fast and safely.
- 04
Deploy as plain files
The winner ships as a pickled model plus an encoded surrogate—no heavy serving infrastructure to stand up.
The result
The eval called it, and it held live
Activation here is same-day finale conversion—open the surfaced show and finish it that day. The winner ran as a live A/B, then rolled to all users on Jun 4; activation stepped cleanly on rollout and held. Offline predicted +2.6pp; live delivered +3.1pp.
- Same-day finale conversion (live)+3.1pp
- Offline prediction (DR)+2.6pp
Activation held flat for days, then a clean step on the Jun 4 rollout to 100% that held. Exact denominators and confidence intervals remain Dashverse's; figures are approved point estimates pending final confirmation.
In closing
A one-off win became a continuous learning loop
In one engagement Pavo learned Shortfree's system, found where Autoplay was leaking activation, built the offline-eval engine to screen ideas safely, and shipped the change that moved the metric—predicted +2.6pp, delivered +3.1pp live. Every stage is now reusable infrastructure: the next idea runs through it in days.
More case studies
Teams using Pavo to move the metrics that matter, in production, at scale.
- A 25M-user edutainment app$100M+ ARR · 100M downloadsHow category-level personalization lifted notification CTR +43% and doubled second-video starts+43%Notification CTRRead case study
- Feed Personalisation 2Technical case-study rewriteHow Pavo’s Segment-Adaptive Personalization Lifted Time Spent by 4.5%+4.5%Platform-wide time spentRead case study
