Impression Share Prediction: An Offline Evaluation Task for Ranking Systems
Best AI papers explainedShow Notes
Researchers from Meta Platforms propose a novel offline evaluation task called impression share prediction to better anticipate how new ranking models redistribute traffic across different business objectives. Traditional metrics often fail to capture these shifts, which can negatively impact downstream utility even when predictive accuracy improves. To address this, the authors developed a structural causal model that identifies how model signals and delivery capacity interact to determine impression allocation. Their framework includes a Random Forest regressor for established models and a specialized encoder-conditioned architecture to handle the complex dynamics of newly introduced models. This system significantly reduces prediction error compared to standard baselines, particularly during the critical first hour of a model's deployment. Ultimately, this approach provides practitioners with vital visibility into a candidate model's allocation behavior before proceeding to expensive online A/B testing.
More from Best AI papers explained
Automated research agents powered by large language models often experience performance plateaus during open-ended discovery tasks as their accumulated history makes their edits overly conservative. To overcome this limitation, researchers introduce a novel strategy called forgetting, which periodically discards previous agent sessions and candidate search histories while retaining only the single best solution found so far. When combined with a collaborative multi-agent team setup that allows asynchronous communication, this reset mechanism clears out unproductive contextual baggage and unblocks fresh exploration. Experimental evaluations across diverse domains—including heuristic programming, algorithm design, kernel optimization, and machine learning training—demonstrate that teams utilizing forgetting consistently outperform single-agent baselines and parallel configurations. Ultimately, the findings emphasize that managing what an automated discovery agent is allowed to remember is just as critical to its overall success as the amount of computational resources provided.
AI-driven scientific discovery increasingly relies on automated verification to evaluate candidate solutions, but the scarcity of reliable automated metrics makes human expert review essential for validating complex research. Because generating candidate solutions often outpaces human review capacity, OSVERIFY introduces an innovative algorithmic framework that separates verification into hierarchical error proposal and error validation to maximize the rejection of invalid solutions while preserving rare valid ones. Experiments across mathematical benchmarks demonstrate that this approach significantly reduces human review workload compared to standard voting baselines while maintaining strict false-rejection constraints. Furthermore, the application of OSVERIFY to real-world research successfully identifies subtle mistakes in published papers and helps resolve open conjectures in the Langlands program. Ultimately, this methodology facilitates human-AI research collaboration by providing a structured, scalable solution to the verification bottleneck in open-ended scientific domains.
This paper introduces a scalable algorithm called Skills Estimate Empowerment (SEE) that bridges unsupervised skill learning and empowerment maximization to train intelligent agents in complex, open-ended environments. By treating mutual information skill learning as a lower bound for potential empowerment, the method utilizes a two-stage policy where a potential policy navigates toward controllable states while effective skill policies explore local options. Through experiments in challenging settings such as Craftax-Classic, AntSoccer, and BuilderBench, the authors demonstrate that this approach successfully overcomes previous computational limitations. Consequently, the trained agents exhibit emergent behaviors like resource gathering, tool crafting, environmental modification, and survival without relying on external rewards.
This paper introduces HybridAgents, a novel framework that blends continuous latent reasoning with explicit text interactions within a unified generation framework. By utilizing a lightweight adapter called HybridLink, the system seamlessly transitions between internal thinking modes and external tool usage without losing crucial context. The model leverages continuous working memory to maintain multi-turn hybrid generation, effectively eliminating the need for traditional token-space reconstruction across different conversational steps. Furthermore, specialized post-training protocols involving Hybrid Supervised Fine-Tuning and Hybrid Reinforcement Learning stabilize generation and scale reasoning capabilities. Extensive empirical evaluations demonstrate that this architecture significantly outperforms standard language models, achieving superior accuracy while concurrently reducing inference latency, token consumption, and computational overhead.
This research demonstrates that unfaithful large language model service providers can artificially inflate the number of reasoning paths generated during test-time compute to overcharge users. By utilizing a specific algorithm to generate and strategically reorder extra paths, a provider can make every reported output appear necessary under adaptive stopping rules. Furthermore, the authors show that this manipulation enables providers to successfully evade exact likelihood-ratio audits. Experiments across multiple open and distilled model families reveal that these additional reasoning paths follow a heavy-tailed distribution, meaning substantial overcharging remains possible even under strict auditing constraints.