-
SPICE: Self-Play In Corpus Environments Improves Reasoning
An overview of our paper, SPICE (Self-Play In Corpus Environments). SPICE is the next step in our line of work on self-play for large language models, following SPIRAL. Where SPIRAL had a model sharpen its reasoning by playing a handful of fixed zero-sum games against itself, SPICE grounds the self-play loop in a far larger world: a single model plays a Challenger that mines a document corpus to pose grounded reasoning tasks, and a Reasoner that solves them without the document. Grounding in an external corpus removes the hallucination amplification and information symmetry that make closed-loop self-play plateau, and lets the model pose problems it could not have invented from its own weights alone, yielding consistent gains across mathematical (+8.9%) and general (+9.8%) reasoning.
-
SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning
An overview of our paper, SPIRAL - Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning. In this paper, we introduce SPIRAL, a framework where self-play on zero-sum games incentivizes models to develop reasoning capabilities by automatically selecting generalizable Chain-of-Thought patterns from pretrained LLMs. This framework demonstrates that competitive game dynamics drive the discovery of reasoning strategies that transfer to mathematical and general reasoning benchmarks, serving as an initial exploration toward integrating self-play into the LLM self-improvement pipeline.
-
TorchOpt: An Efficient Library for Differentiable Optimization
An overview for our NeurIPS 2022 Workshop OPT paper, TorchOpt - An Efficient Library for Differentiable Optimization. In this paper, we introduce TorchOpt, a PyTorch-based efficient library for differentiable optimization. This library provides unified and expressive differentiable optimization programming abstraction, and high-performance distributed execution runtime.