RSI-PDEbench
RSIRecursive self-improvement on scientific code

RSI-PDEbench: A Benchmark for Self‑Improving Agents in PDE Solving

RSI-PDEbench gives a self-improvement harness one foundation model, a fixed budget and a set of identify-then-solve PDE cases. The harness must turn feedback into a solver program. That solver is then scored, after every round, on a hidden test split that no run ever sees.

The RSI loop Each round the model proposes a solver update, the harness self-tests it on every training case, a fixed validation gate judges it, and the best solver so far is kept. The kept solver is scored on the hidden test after every round. Hidden test scored every round Propose Identify PDE, evolve true coefficient, write the solver Self-test every training case Validate fixed gate, ≤ 30 rounds Keep the best so far RSI recursive self-improvement
Reference solutions of the 42 problem templates
1001
PDE cases

Each case gives 256 spatial-temporal samples of an unknown equation's solution, without information about its class and coefficients. A recursive self-improving loop helps agents identify the equation, evolve the correct coefficients and write accurate code to solve the PDE.

Authors

  • Binghang Lu1,*Purdue
  • Lang Cao2,†UIUC
  • Changhong Mou3,†USU
  • Yige Sun4Independent
  • Cindy Kong1Purdue
  • Zheyuan Deng5Brown
  • Tian Yuan4Independent
  • Bing Hu4Independent
  • Runyu Zhang6MIT
  • Na Lu7ASU
  • Guang Lin1,*Purdue
* Corresponding authors: lu895@purdue.edu † Equal contribution

Affiliations

  1. 1Purdue University
  2. 2University of Illinois Urbana-Champaign
  3. 3Utah State University
  4. 4Independent researcher
  5. 5Brown University
  6. 6Massachusetts Institute of Technology
  7. 7Arizona State University

What it measures

Every case is an identify-then-solve problem. A harness is judged by the solver it keeps, on hidden cases it never sees, after every round of self-improvement.

Data samples256 spatial-temporal samples of an unknown equation's solution, plus black-box forcing, initial and boundary data.
IdentifyChoose the equation's class from a library and fit its coefficients.
SolveCompute the solution on the whole domain, within the case's time limit.
PassSpace–time and final-time relative L2 ≤ 2e-3, with initial and boundary data exact to 1e-10.
Headline 1

Held-out pass rate

passed ÷ hidden-test cases

The share of hidden-test cases passed by the solver the harness retained after N rounds. A round is one gated validation submission.

Headline 2

Gain over a single run

harness − single run

The held-out pass rate minus that of a single-run entrant with the same model: one model call, frozen as returned. This is what the self-improvement loop adds.

Headline 3

Dev–test gap

validation − held-out

The validation pass rate minus the held-out pass rate of the same solver. A large gap means the loop overfit to the cases it could see.

The RSI-PDE harness

A frozen foundation model improves its solver program P, instruction policy π and memory M. The supervisor, controller and acceptance rules stay fixed.

RSI-PDE harness: context components H1–H6, frozen LLM, supervisor reply checks H7, experiment, analysis and proposal actions H8–H10, self-test H11 and validation gate H12. Accepted solver, policy and memory return to the next round; hidden testing is evaluator-only.
RSI-PDE harness (paper Figure 3). One round follows steps 1–7: read context, act in the sandbox, self-test, validate, then retain an accepted candidate. The model rewrites its policy and memory through accepted proposals; rejection reasons return as feedback. Hidden-test scores stay with the evaluator. Download PDF

The RSI-PDE loop

Identify the equation with residual loss, fit its coefficients, solve it, and use public relative L² errors and the validation gate to guide the next refinement.

Recorded Opus 5.5 solver proposals in selected rounds 2, 3, 4, 7 and 9. Code edits appear beside public validation errors, pass counts and acceptance decisions. The retained solver improves from 16 to 18 passes out of 20; a rejected round keeps the previous program.
Recorded Opus 5.5 code evolution on 20 public validation cases in the Full classical/fractional group. Selected rounds 2, 3, 4, 7 and 9 show coefficient fitting, solver refinement and a rejected proposal; retained macro relative L² falls from 6.19 × 10−3 to 6.87 × 10−4, while validation passes rise from 16 to 18. Code excerpts are abridged; errors and gate decisions come from the saved run. Pause or select a round to inspect its code. Source details
RSI-PDE workflow: 256 spatial-temporal samples and known public inputs, a frozen LLM with evolving solver, policy and memory, residual screening of the top five PDEs, coefficient fitting and solving, relative L2 testing, self-test and validation feedback, with a toy Burgers example across five rounds.
Download GIF
An animated walkthrough of the loop from paper Figure 4. Residual screening keeps the five lowest-loss PDE candidates; coefficient fitting and public relative L² tests refine the equation and solver before the validation gate. Accepted P, π and M carry into the next round. The bottom row is an illustrative Burgers case, with later coefficient values chosen to illustrate refinement, rather than benchmark measurements. Inputs are 256 spatial-temporal samples: Full is noise-free (full0), Noisy uses 50% RMS noise, and Hard-OOD uses 1% RMS noise. Download PDF

What's in RSI-PDEbench

There are 42 problem templates. They range from scalar diffusion, transport and fractional problems, through chaotic, incompressible, phase-field and electromagnetic systems, to data-driven problems whose inputs are real fields. Every template appears once in training, once in validation and five times in the hidden test. One seeded generator makes all of them, so the splits share one distribution.

Equation families

Calibration

Before any model was run, fixed solvers from earlier 2-hour runs solved all 1001 cases of the expanded release: 2,856 sandboxed solves, with no model calls. The probes separate clearly. Templates that every strong probe solved were hardened for the benchmark.

The pass rate of each probe solver over all variants of the calibrated templates. The 10 templates of Problems 12–16 had no earlier solver, so the baseline runs calibrate them.

Where the headroom is

The pass rate per template and probe, on the colour scale above, hardest templates first. Hover over a cell for its value. Templates marked * have harder numbers in the benchmark, because every strong probe solved all their calibration variants.

Baseline results

Paper Figure 1: passed-case bar charts on the left and learning curves on the right, with Opus 5.5 above GPT-5.5 and Full, Noisy and Hard-OOD columns. RSI-PDE is red; native curves are black and native bars are gray.
Paper Figure 1: final passed-case counts and learning curves across Full, Noisy and Hard-OOD. Each half has Opus 5.5 above GPT-5.5. Red denotes RSI-PDE; native curves are black and native bars are gray. Labels show gains in passed cases. Axes use local ranges; cropped bar baselines are marked. The native Opus Full curve whisker is the observed 187–198 range over three runs. Download PDF

Results per PDE

Residual check

How well each hidden-test solution satisfies the true equation, beside its relative L2 error. The residual needs no reference solution. It is an extra measure, not the pass test.

Normalized residual: r = ‖Σk Tk(u)‖ / Σk ‖Tk(u)‖ ∈ [0, 1], using the true PDE terms and coefficients, including forcing. Medians cover returned solutions (all, passed and failed); ρ is Spearman correlation with relative L² error. Post-hoc reruns use no model calls; counts remain official (two reruns differ by 1–2 cases). Sparse time outputs can impose a ~10−3 residual floor, limiting comparisons among very accurate solutions. Elliptic point outputs have no residual; single-run rows include only parts that returned code.

Get started

Clone the repository and build the environment. Run your harness, or one of the built-in entrants, on a GPU node with bubblewrap. The maintainers score submissions on the hidden test.

git clone https://github.com/jeremylu916/RSI-PDE.git && cd RSI-PDE
pip install -e .
export PDE_RSI_PROJECT_ROOT=$PWD   # the code defaults to the original cluster path
# run a built-in entrant on all 42 templates (RSI-PDE, Claude Code or Codex; six cores, a GPU)
python scripts/bench.py run --entrant pde-rsi-v3.4 --model opus55 --cores 0,1,2,3,4,5 --output runs/bench-v1-my-run
# maintainers: score every round on the hidden test, then build the table
python scripts/bench.py score --runs runs/bench-v1-my-run --cores 6,7,8,9
python scripts/bench.py report

Specification

docs/rsi-pde-bench.md: the rules, budget, splits, scoring and calibration.

Datasheet

Composition, provenance and licence. The DiffusionPDE inputs are CC BY-NC-SA 4.0.

Equations

pde_benchmarks.tex (Problems 1–16) and the data-driven problems.