How Pavo’s Segment-Adaptive Personalization Lifted Time Spent by 4.5%

A large consumer platform used Pavo to improve the first personalized feed people saw when they opened the product. The current production system was sensible: retrieve content near a user’s recent history, blend similarity with a quality prior, and diversify the final slate. It was fast, explainable, and already strong for a meaningful part of the audience.

The problem was that it treated recommendation as one problem. In practice, a person with almost no history, someone whose tastes are just beginning to form, a regular user, and a power user are constrained by different things. The first needs coverage without much personal signal. The last often needs novelty more than another accurate prediction of familiar interests.

Across three progressive experiments, Pavo turned those differences into an adaptive recommendation policy. The platform reconstructed the existing system, evaluated fourteen intervention approaches, trained five model families on a production-scale feature set, and tested specialized treatments by behavioral stage. The final design retained the current production strategy where it remained hard to beat and introduced learned ranking or explicit discovery only where the evidence supported it.

A single recommender was hiding four product problems

The original path was a conventional two-stage system. Recent consumption supplied a small set of seeds. Semantic retrieval produced candidates. A global score combined similarity and historical quality before a simple interleave produced the feed. Every eligible user passed through broadly the same logic.

That design was strongest when a person had stable, recent preferences and enough unconsumed content near those preferences. It weakened at both ends of the journey. With little history, similarity had almost nothing to work with. With extensive history, it repeatedly returned safe material from an increasingly exhausted neighborhood.

We built stage definitions from behavior rather than demographics. The useful variables were the amount and recency of consumption history, the density and consistency of preference signals, the stability of category affinity, and the failure mode visible in the existing feed. Exact production thresholds are confidential; the operational distinction is more important than the cut points.

Turning behavioral stages into modeling problems

Pavo assembled a training set at production scale with user history, content attributes, exposure context, and user–content cross-features. User representations combined longer-run preference structure with recent behavior. Content representations combined semantic embeddings with category, topic, format, duration, language, freshness, and historical quality signals.

Pavo trained factorization machines, DeepFM, a deep-and-cross network, a two-tower model, and a multi-task network. The multi-task model shared a trunk across three outcomes—completion, skipping, and meaningful engagement—then combined the heads into a ranking score. The system could therefore learn from more than a single click label while preserving distinct signals for depth and dissatisfaction.

The first model looked excellent for the wrong reason

Our first learned models reported very strong classification performance. They were also poor rankers. Random negatives made the task too easy: the model learned to separate plausible catalog items from obviously irrelevant items, not to distinguish between two items a user might realistically have seen.

Pavo rebuilt the training and evaluation design around impression-level negatives, same-neighborhood hard negatives, user-level holdouts, and rank-sensitive metrics. Training AUC stopped being the headline. Hit rate, NDCG, score separation, calibration, and segment-level replay became the decision surface.

Production logs were also shaped by what the current system had chosen to expose. We used several relevance views—including behavior after a rejection and explicit-intent signals—to avoid prematurely eliminating approaches that differed from the current policy. The benchmark remained a supporting instrument; the main question was still which strategy improved each user stage online.

Three experiments changed the design of the system

The experiments were deliberately progressive. The first asked whether a credible alternative could beat production anywhere. The second tested specialized heuristic, learned, discovery, and exploratory treatments across maturity stages. The third narrowed the portfolio into a maintainable adaptive policy.

What did not work

We initially expected the learned ranker to replace the current production system for most users. That did not happen. It added value when sparse but usable preference signals were forming, yet brought little incremental value to the established cohort. For power users it could become too conservative, repeatedly scoring familiar high-affinity material above the novelty those users actually needed.

• Easy random negatives produced impressive AUC and weak ranking; the training design was replaced.
• A completion-heavy blend improved a quality metric while narrowing the slate too aggressively; its weight was reduced.
• Randomized exploration produced valuable labels but was retained as an evidence-collection mechanism, not a permanent user experience.
• A learned ranker that improved one cohort was stopped for power users when plays, breadth, and watchtime moved in the wrong direction.
• The existing production strategy remained the choice for established users because added complexity did not earn material lift.

From a global ranker to an adaptive production policy

The final architecture did not become a collection of unrelated models. A common layer computed user state from recent and longer-run behavior, assigned a stable route, and logged route and policy identity with every exposure. Users near a boundary stayed on a conservative fallback until their evidence was strong enough to justify a change.

Cold-start users received a quality-controlled, diverse slate. Emerging users received learned ranking that combined short- and long-run intent. Established users stayed on the strong production baseline. Power users received explicit discovery from under-explored parts of the catalog. All routes shared exclusions, serving, experiment assignment, monitoring, and rollback behavior.

The resulting metric gains

The programme produced two kinds of evidence. The strongest early treatment effect showed that substantial feed lift was available in the right slice. The later adaptive strategy showed that segment-specific policies could improve broader product outcomes without sacrificing cancellation or uninstall guardrails.

What carries forward

The most useful result was not a new universal ranker. It was a clearer description of the recommendation problem. Personalization depth should grow only when the available signal supports it. Discovery should be treated as a first-class objective when familiarity becomes the constraint. A strong production baseline should remain in place wherever a new system cannot earn its operational cost.

That changes how the next improvement cycle begins. Instead of asking which model should replace production, the team can ask which user stage is constrained, which mechanism could remove that constraint, what evidence would distinguish the candidates, and which route can be changed without destabilizing the rest of the system.