SPADE: Self-Play in Adaptive Synthetic Executable Environments

An overview of our paper, SPADE (Self-Play in Adaptive Synthetic Executable Environments). SPADE is the next step in our line of work on self-play for large language models, following SPIRAL and SPICE. Where SPIRAL sharpened reasoning inside a handful of fixed zero-sum games, and SPICE grounded the self-play loop in a document corpus to pose harder tasks, SPADE lets a single model design the training worlds themselves: an Environment Designer writes complete, long-horizon environments as executable Python implementing a Gym-style reset()/step() interface, and a Reasoning Agent learns to act in them. A hint-based regret reward trains the Designer to keep every environment solvable but right at the edge of the Agent's ability, so the curriculum keeps moving as the learner improves. The same recipe lifts both reasoning and multi-turn agentic tool use, by +5.3 over the strongest fixed-environment baseline at 30B scale, with a margin that grows with model size.

Motivation: The Environment Is Still Fixed

Continuous self-improvement requires an ever-expanding pool of self-generated, diverse, adaptive goals. That single sentence is the thread that runs through this whole line of work. In SPIRAL , a model sharpened its reasoning by playing a handful of fixed zero-sum games against itself. In SPICE , we grounded the self-play loop in a document corpus, so a Challenger could mine real text to pose problems a Reasoner could not have invented from its own weights. Each step removed a different ceiling on how far a closed loop could climb.

I ended both of those posts with the same promise. SPIRAL’s closing section imagined “self-play between actor and environment, where the environment itself becomes another learner that generates increasingly challenging problems.” SPICE’s closing section said the constraint that remained was the environment itself, still fixed, and that the next step was “to make the environment a learnable component that co-evolves with the agent, so the curriculum scales with the learner rather than being capped by a fixed corpus or a fixed game.” SPADE is that step.

The reason the environment matters now more than ever is that the target has moved. The frontier is no longer single-turn reasoning; it is agents that act over long horizons, calling tools, browsing, and operating software across many turns . Web-scale pretraining data, the fossil fuel of AI, is largely exhausted , and every gain from experience is still tied to a supply of environments with verifiable rewards that is small, hand-built, and fixed. An agent stops improving once it exhausts them. Human-curated environments scale only as fast as people can write them. Frozen synthetic generators produce a distribution that never adapts, so the model soon outgrows it. And closed-loop self-play, ungrounded, is bounded by the invisible leash : it cannot pose a challenge beyond its own knowledge. Every one of these keeps the goal distribution fixed while the learner scales past it.

SPADE trains one LLM to design its own environments and improve on held-out benchmarks in both the games and tool-use settings. A single policy plays two roles: an Environment Designer that writes an executable environment with a privileged hint, and a Reasoning Agent that solves it with and without the hint. The return gap rewards the Designer (hint-based regret), task completion rewards the Agent, and both update the same weights.

Core Insight: An Environment Is Just Code

SPADE's Core Insight: Let a single model write its own training environments as executable code. In the Environment Designer role, the model emits a complete Python program implementing the Gym-style reset()/step() interface, a full multi-turn MDP with its own state transitions, reward function, and verification logic. In the Reasoning Agent role, the same model learns to act in it. Because any computable MDP can be written as a program, the space of environments the Designer can reach is bounded only by the model's coding ability, not by a hand-designed parameterization. And because the Designer is trained, by a hint-based regret reward, the environment distribution co-evolves with the Agent instead of staying frozen.

This is the move I kept circling in the earlier posts, now made concrete. SPIRAL’s environments were three human-written games. SPICE’s were questions mined from documents. Both are still, in the end, worlds someone else specified. The code-as-environment representation lets the model specify the world itself. A single interface, reset() returns the first observation and step(action) returns the next observation, a reward, and whether the episode is done , spans everything from a one-step math check to a twelve-turn tool-use task where the reward pays out only after earlier actions have changed hidden state. You get to freely express the transition function and the reward function in ordinary Python, so the same training pipeline covers single-turn reasoning and long-horizon agentic interaction without changing anything but the generated code.

Making the designer a learner, rather than a fixed generator, is the oldest idea in this lineage and the one I find most beautiful. PowerPlay imagined a single system that continually invents the simplest problem it cannot yet solve; asymmetric self-play split that into a proposer and a solver; unsupervised environment design and POET co-evolved agents with the worlds they train in. SPADE carries that principle into an unbounded, code-defined environment space, with the designer and the agent being the same set of weights. Designing an environment, in the end, is like setting up a scientific experiment: the Designer proposes a new, self-contained setting, and the Agent learns by acting in it and seeing what holds.

Research Questions

RQ1: Does training a model to design environments beat training it on a fixed set of environments, even a strong one?
RQ2: Does the Designer actually adapt to the Agent, keeping the curriculum at the frontier as the Agent improves?
RQ3: Which ingredients (Designer training, corpus grounding, environment memory) actually carry the gain?
RQ4: Does the same recipe reach real multi-turn agentic tool use, not just reasoning?

The SPADE Framework

The SPADE framework. Top: the Environment Designer conditions on an environment memory and a pretraining corpus to emit an executable environment together with a privileged hint. Bottom: the Reasoning Agent plays that environment with and without the hint; the return gap is the Designer's hint-based regret, and task correctness is the Agent's reward. Both rewards update the shared policy via GRPO.

Dual-Role Self-Play and Code-as-Environment

A single policy acts in two roles, selected by a system prompt: designing environments or solving them. In the Designer role it produces an executable environment as a Python program whose step() encodes both the transition function and the reward function, and it also writes a short privileged hint for that environment. In the Agent role, the same weights interact with the environment turn by turn, receiving observations and rewards. Both roles share parameters and are updated jointly with GRPO , with role-specific rewards: the Designer receives hint-based regret, the Agent receives task correctness from the environment’s own reward function. Every candidate environment passes syntax, executability, and solvability checks before it enters the training pool, so the Agent only ever trains on programs that actually run and can actually be solved.

Here is the flavor of what the Designer emits, a Wordle-style deduction game as a single stateful class in the exact format it produces:

class WordleEnv:
  def reset(self, seed=None):   # initial state
    self.target = choice(WORDS); self.turns_left = 6
    return "Guess a 5-letter word in 6 tries.", {}

  def step(self, guess):   # (obs, reward, terminated, truncated, info)
    self.turns_left -= 1
    fb = feedback(guess, self.target)
    if fb == "GGGGG": return fb, 1.0, True, False, {}
    return fb, 0.0, False, self.turns_left == 0, {}

The real environments the Designer writes at 30B scale are far richer than this: a family-dispute negotiation with hidden legal state that only pays out once the title is secured, or a thermodynamics lab where the Agent must run a three-step gas cycle and discover that the net entropy change is zero. Same interface, unbounded content.

Hint-Based Regret: Aiming at the Frontier

The whole system lives or dies on one question: how do you reward the Designer? A pure adversary is free to make environments unsolvable; a cooperative designer can inflate the Agent’s reward without teaching it anything. SPADE’s answer is hint-based regret. For each environment, the Designer also writes a privileged hint, a sentence or two of strategy or a partial solution, and the reward is the gap between the Agent’s success with the hint and without it:

r_D(e) = mean_return_with_hint(e) − mean_return_without_hint(e)

Three regimes fall out of this one number. A high gap means the environment is right at the frontier: solvable with the hint, not without, exactly where the Agent has the most to learn. A low gap with high returns means the Agent has already mastered it. A low gap with low returns even with the hint means the environment is intractable and teaches nothing. Rewarding the gap steers the Designer toward the first regime and away from the other two. This is the minimax-regret objective of PAIRED , but without a separate antagonist network: the hint-equipped Agent serves as the upper-bound policy, and the same weights play every part. For a policy that can ignore a hint, extra information cannot lower its expected return, so regret is non-negative in expectation, which is what keeps the dynamics constrained rather than adversarial.

Corpus for Breadth, Memory for Difficulty

A generator conditioned only on its own output has no source of novelty outside its weights, so it narrows toward the patterns it already favors, the invisible leash again. SPICE established the fix for task generation: ground every round in freshly sampled human corpus. SPADE carries that principle to environment generation, where a sampled passage seeds an executable MDP rather than a question-answer pair. But SPADE adds a second input that SPICE did not have: an environment memory. The corpus supplies breadth, deciding what the environments are about; the memory supplies difficulty, deciding how hard they are. It is a buffer of previously generated environments annotated with their regret scores and skill tags, so each round the Designer starts from the high-regret environments the Agent currently finds hard, varying them, rather than starting from scratch and re-posing what the Agent has already outgrown. The corpus keeps the environments diverse; the memory keeps the difficulty pinned to the moving frontier. As we will see, removing either one costs real points.

RQ1: Does designing environments beat training on fixed ones?

We train three Qwen3 backbones (4B, 8B, and 30B-A3B, the last our primary model) and compare against two fixed-environment baselines that share SPADE’s base model, GRPO hyperparameters, and compute budget: RLVE , a set of 400 hand-engineered verifiable environments with difficulty scheduling, and a static pool generated once by a frontier model. Every benchmark is held out from training.

Training on diverse self-designed games improves science reasoning, code generation, and procedural reasoning while competition math is preserved (Qwen3-30B-A3B). No science questions, no real code, and no benchmark-style problems appear in the generated games, yet held-out GPQA-Diamond (+5.4), LiveCodeBench-v6 (+4.1), and the four Reasoning-Gym categories (+5.8 to +18.3) all climb through the full run. The dashed line is the untrained base.

At 30B-A3B, SPADE reaches a suite average of 58.3: +8.1 over the base model and +5.3 over the strongest fixed-environment baseline, and the margin holds at every scale. The generated games contain none of the evaluated content, yet the reasoning skills transfer to mathematics, science, and code. The most striking part is what happens as models get bigger.

SPADE's average gain over base grows with model size, from +5.2 at 4B to +8.1 at 30B-A3B, while the matched-budget fixed-environment baseline stays near +1.2 at every size. A static pool is a fixed signal that larger models fit quickly and stop learning from; an adaptive curriculum keeps generating environments at the frontier, and larger models benefit more from it.

This is the clearest evidence for the whole thesis. A fixed set of environments is a fixed amount of signal: a bigger model fits it faster and then plateaus, so the gain stays flat near +1.2 no matter the scale. An adaptive curriculum keeps producing new frontier environments as the Agent climbs, so the gain grows with capability. The better the learner, the more a moving curriculum is worth.

RQ2: Does the Designer actually adapt to the Agent?

An adaptive curriculum is only real if the Designer keeps supplying environments the Agent can actually learn from, and keeps making them harder as the Agent improves. The generation prompt is byte-identical across all 400 training steps, so anything that changes traces to Designer training, not to new instructions. Three independent, code-level measurements move together.

A trained Designer stops revealing the solution method in the prompt. Physics environments at steps 20, 192, and 384: the share whose opening observation prints the governing formula falls from 25% to 5% over 473 environments. Rightmost, in red: at matched steps, an ablation with the corpus removed emits the same RotatingMazeEnv 41 consecutive times.

First, environments stop giving the answer away: the fraction of physics environments that print the governing formula in the opening observation falls from 25% to 5%. Second, winning increasingly requires interaction: the share of environments that pay out only after earlier commands have changed hidden state, rather than grading a single submitted answer, rises from 43.5% to 55.3%. Third, rewards become more finely graded, from 3.7 to 5.8 distinct levels per environment, while the program length stays flat (325 to 333 lines, correlation 0.05), so the added structure comes from smarter grading, not longer code. Across the run, SPADE keeps roughly a third of every batch of environments in the learnable band, where the Agent wins between 20% and 80% of the time, and holds it there through step 400.

That red panel above is my favorite failure. Strip out the corpus grounding and the Designer collapses onto a single idea: over steps 290 to 312 it emits the exact same rotating-maze environment forty-one times in a row. Grounding is what keeps the worlds diverse; on an embedding map, corpus-grounded environments fill the space (an effective-diversity score of 0.68) while the ungrounded ones huddle into a single blob (0.04). The corpus buys breadth, and the regret reward turns that breadth into difficulty.

You can see this directly. Below is a 2D map of 4,976 environments the Designer wrote across training, each embedded with SBERT and projected with t-SNE, one dot per environment. Toggle the runs: full SPADE (purple) spreads across the whole space, while the no-corpus ablation (red) stacks its near-duplicates into dense clumps, the collapse made visible as saturation.

Open the full interactive explorer → (click any environment there to read its generated code, hint, and agent trajectories)

Every dot is one executable environment the Designer wrote (SBERT embedding, t-SNE to 2D). Toggle runs in the legend and hover a dot for its class name and training step. Full SPADE fills the space; the no-corpus ablation collapses onto repeated near-duplicates. The per-environment source code, hints, and trajectories live on the project page.

The Agent’s side of the loop shifts to match. Early in training it reasons entirely up front and cannot recover once the interface rejects its answer; by the middle it tests short hypotheses and revises them as results come back; late in training it gathers evidence first and derives once, when the evidence is enough. It learns to act on what the environment tells it, not just to derive in advance, while keeping its long-form derivation ability intact.

RQ3: Which ingredients matter?

Removing Designer training drops self-play far below the untrained base; removing any single component (corpus grounding or environment memory) stays above base but peaks early and fades. Full SPADE keeps improving to the end of training. One curve per ablation variant, Qwen3-30B-A3B, games setting; the dashed line is the untrained base.

The three ingredients are not optional; removing any one costs at least 5.9 suite-average points. The sharpest result is about training the Designer at all. Freeze it and drop the memory, and the model gets worse than no training: the suite average falls 12.6 points below the untrained base, because a frozen designer keeps generating environments the Agent has already outgrown. Even replacing the self-play Designer with a stronger frozen frontier model, while keeping corpus and memory, recovers only about 40% of SPADE’s gain and does not improve code at all. The frozen and single-component variants all peak early and then fade; full SPADE keeps climbing to the end. The value is not in any one clever environment. It is in a designer that adapts to the Agent as the Agent learns.

RQ4: Does it reach real multi-turn agentic tool use?

Everything above is the games setting: reasoning skills learned in self-designed puzzles. The same recipe, with the Designer instead writing tool-use environments (a set of simulated tools in function-calling format, a backend state they modify, and several natural-language user instructions that arrive one at a time), lifts real agentic benchmarks too.

Benchmark (30B-A3B)	Base	+ SPADE	Δ
BFCL v4 (multi-turn)	49.0	54.7	+5.7
τ2-bench	49.0	52.6	+3.6
ACEBench-Agent	62.0	75.9	+13.9

Self-designed tool-use environments improve multi-step agentic interaction (Qwen3-30B-A3B). The size of the gain tracks how closely a benchmark's task structure matches the generated environments: ACEBench-Agent gains most (+13.9), its stateful, multi-step tasks mirroring the generated pattern of a database, a tool schema, and a multi-call goal. At 4B, BFCL v4 multi-turn gains +10.3. SPADE leads dedicated tool-use data-synthesis systems on both BFCL multi-turn and ACEBench-Agent.

This is the answer to why the environment matters in the agentic era. The benchmarks that gain most are the ones whose long-horizon, stateful structure the Designer learned to reproduce. The structural training signal, learning to plan across turns and act on returned state, transfers to real tool use where domain-specific data collection does not reach. One interface, from a single-step math check to a twelve-turn tool-calling task, and the same self-play loop trains both.

Where This Is Heading: Automating the Rest of the Pipeline

It is worth stepping back to say what SPADE is a piece of. Building a frontier model has become a recognizable pipeline: pretraining, then mid-training, then a post-training stage of RL on environments with verifiable rewards. Each stage is increasingly a known recipe, executed by hand. Recursive self-improvement, read plainly, is what happens when the AI starts walking through that recipe and automating it, stage by stage, instead of waiting for us to.

Where this goes next. SPADE automates one specific thing: the scaling of environments, which is one part of the post-training stage. Today those environments are hand-curated or frozen; SPADE makes the model design and grow them itself, so the curriculum scales with the learner. But environment design is only the first component to fall. The reward-shaping and verification logic still lives inside human-written code; the curriculum schedule is a fixed rule; and the learning algorithm itself, the update rule that turns experience into weight changes, is a fixed, human-authored GRPO. Each of these is a stage of the recipe waiting to be handed to the model. The arc that began with SPIRAL playing fixed games, moved through SPICE grounding in a corpus, and reaches SPADE designing its own worlds, points at a system that eventually authors not just the environments but the rules by which it learns from them.

Read as one line, the trajectory is simple: self-play inside fixed games, then self-play grounded in a corpus, then self-play in worlds the agent designs and grows for itself. Each step hands one more piece of the training loop to the learner. SPADE hands it the environments. The pieces that remain, the reward logic, the curriculum, and finally the update rule, are the map for everything after.

Conclusion

SPADE makes environment design a learnable component of post-training through self-play. A single model plays an Environment Designer that writes complete environments as executable Python and a Reasoning Agent that learns to solve them, and a hint-based regret reward keeps every environment at the Agent’s frontier. Concretely, SPADE:

1. Beats fixed-environment training, and the margin grows with scale. +5.3 over the strongest fixed-environment baseline at 30B-A3B (+8.1 over base), while a matched-budget fixed pool stays near +1.2 at every size, because an adaptive curriculum keeps producing frontier environments a bigger model can still learn from.

2. Produces a genuinely adaptive curriculum. With training alone, the Designer stops revealing solutions (formula reveals fall 25% to 5%), makes winning require interaction (43.5% to 55.3%), and holds roughly a third of every batch in the learnable band through the whole run. Remove the corpus and it collapses to the same environment 41 times in a row.

3. Needs all three ingredients. Designer training, corpus grounding, and environment memory each carry at least 5.9 suite-average points; a frozen designer, even a stronger frontier one, recovers only about 40% of the gain.

4. Reaches real multi-turn agentic tool use. The same recipe lifts BFCL v4 multi-turn (+5.7) and ACEBench-Agent (+13.9) at 30B, with the largest gains where the benchmark’s long-horizon structure matches the environments the Designer learned to write.

The takeaway continues the one from SPIRAL and SPICE. A self-improving system needs an ever-expanding supply of its own goals, and the way to get one is to stop treating the environment as fixed. Point the model at the task of building its own worlds, and the curriculum opens back up.

---

The agenda that began with SPIRAL and continued through SPICE was always heading here: a model that generates experience of its own, not just reads ours. SPADE lets it build the worlds it learns in. The next step lets it write the rules it learns by.