-
SPADE: Self-Play in Adaptive Synthetic Executable Environments
An overview of our paper, SPADE (Self-Play in Adaptive Synthetic Executable Environments). SPADE is the next step in our line of work on self-play for large language models, following SPIRAL and SPICE. Where SPIRAL sharpened reasoning inside a handful of fixed zero-sum games, and SPICE grounded the self-play loop in a document corpus to pose harder tasks, SPADE lets a single model design the training worlds themselves: an Environment Designer writes complete, long-horizon environments as executable Python implementing a Gym-style reset()/step() interface, and a Reasoning Agent learns to act in them. A hint-based regret reward trains the Designer to keep every environment solvable but right at the edge of the Agent's ability, so the curriculum keeps moving as the learner improves. The same recipe lifts both reasoning and multi-turn agentic tool use, by +5.3 over the strongest fixed-environment baseline at 30B scale, with a margin that grows with model size.
-
SPICE: Self-Play In Corpus Environments Improves Reasoning
An overview of our paper, SPICE (Self-Play In Corpus Environments). SPICE is the next step in our line of work on self-play for large language models, following SPIRAL. Where SPIRAL had a model sharpen its reasoning by playing a handful of fixed zero-sum games against itself, SPICE grounds the self-play loop in a far larger world: a single model plays a Challenger that mines a document corpus to pose grounded reasoning tasks, and a Reasoner that solves them without the document. Grounding in an external corpus removes the hallucination amplification and information symmetry that make closed-loop self-play plateau, and lets the model pose problems it could not have invented from its own weights alone, yielding consistent gains across mathematical (+8.9%) and general (+9.8%) reasoning.
-
SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning
An overview of our paper, SPIRAL - Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning. In this paper, we introduce SPIRAL, a framework where self-play on zero-sum games incentivizes models to develop reasoning capabilities by automatically selecting generalizable Chain-of-Thought patterns from pretrained LLMs. This framework demonstrates that competitive game dynamics drive the discovery of reasoning strategies that transfer to mathematical and general reasoning benchmarks, serving as an initial exploration toward integrating self-play into the LLM self-improvement pipeline.
-
TorchOpt: An Efficient Library for Differentiable Optimization
An overview for our NeurIPS 2022 Workshop OPT paper, TorchOpt - An Efficient Library for Differentiable Optimization. In this paper, we introduce TorchOpt, a PyTorch-based efficient library for differentiable optimization. This library provides unified and expressive differentiable optimization programming abstraction, and high-performance distributed execution runtime.