What Self-Improvement (RSI) Research Offers Agent Builders Who Never Train Models

Research date: Sep 19, 2026. Direction set in a scoping exchange: the reader builds agents on pretrained models, does not train LLMs, and was not yet clear what RSI is — so this post answers one question: if you never train a model yourself, does the self-improvement / RSI literature give you anything for building agents? The deep history, the complete paper table, and the full citation apparatus are in the backing survey (research-workspace/rsi-self-improvement-evidence-survey-2026-09-19.md in the site repo). Every quote is verified against a fetched page, a fetched scan, or a journal record (method notes at the end); recommendations about applying these patterns to agent setups are labeled interpretation. A follow-up round (Sep 20, 2026) extended the human-audited variant below with a third reading and verified precedents; its evidence note is research-workspace/rsi-human-feedback-test-compilation-evidence-2026-09-20.md in the site repo.

RSI in plain words

Self-improvement, in the sense used in this literature, is a loop: a system generates candidate improvements, something measures whether they are better, the winners are kept, and the process repeats. “Recursive” self-improvement (RSI) means the loop feeds its own outputs back in — the thing that got better becomes the thing doing the improving. The term predates LLMs by more than half a century: I.J. Good described machines designing better machines in 1965 — “the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control” (Advances in Computers vol. 6, p. 33; scan-verified) — and Schmidhuber formalized it in 2003 as a machine that “rewrites any part of its own code as soon as it has found a proof that the rewrite is useful” (arXiv:cs/0309048). No weights, no gradients, no language models in either definition.

Three verified examples calibrate the phrase, one per era:

• 2017, no LLM: AlphaGo Zero learned superhuman Go “based solely on reinforcement learning, without human data, guidance or domain knowledge beyond game rules” (Silver et al., Nature 550:354–359, abstract via Europe PMC/MEDLINE).
• 2023, LLM as a frozen generator: FunSearch paired “a pretrained LLM with a systematic evaluator”; the best programs feed back into the pool — “creating a self-improving loop” (Nature 625:468–475, DeepMind blog). The model never got better; the programs did.
• 2025, the loop aimed at an agent: the Darwin Gödel Machine is “a self-improving system that iteratively modifies its own code (thereby also improving its ability to modify its own codebase) and empirically validates each change using coding benchmarks” (arXiv:2505.22954).

Two flavors of the loop are active in 2026, which is why the term feels murky: (a) the model trains on itself — its own generated data and rewards update its weights (most of the 2024–2026 papers); (b) everything else improves — game policies, programs, kernels, or the agent’s own harness — with the model frozen. Both are RSI; only (a) requires a training stack.

Short answer

Yes — but not through the part everyone hypes. The wave of papers about models fine-tuning themselves is one cell of the self-improvement space, and it is the one cell that needs a training stack. The parts that transfer to agent building are the loop and the verifier — and the published evidence shows those parts, not the weight updates, produce most of the measurable wins. Four mechanisms run on a frozen model; one mechanism you cannot import. Your own agent loop — described next — is a legal instance of the first kind.

Worked example: an error-fixing agent loop

Consider this architecture (described generically): a process collects system errors and writes them as questions; a fixer agent (frozen LLM + tools) works on them as answers; a verifier agent checks whether each fix is correct; the traces (error, fix, verdict) accumulate; and an enhancer uses those traces to improve the fixer. Automatically, end to end.

Is this RSI? Yes, structurally — every component maps to a verified pattern:

Two things decide whether this loop actually works:

1. What “enhance” means. Fine-tune the fixer’s weights on traces → training-flavor RSI, which needs a training stack. Update the fixer’s harness — memory of past errors, skills, prompts, tool configs, selected by the verifier’s verdicts → the DGM cell, which runs entirely on frozen models: the DGM “empirically validates each change using coding benchmarks”, and its underlying foundation model never changes (arXiv:2505.22954). The second is what an API-only agent builder can run today. Interpretation: DGM at lab scale; the same loop at deployment scale is mechanically identical but unverified in the literature.
2. The verifier is the weak link. If the fixer and verifier are the same model — or even the same family — the loop rebuilds a documented failure mode: “an LLM evaluator scores its own outputs higher than others’ while human annotators consider them of equal quality”, with the bias strength correlating “linearly” with the evaluator’s ability to recognize its own outputs (Panickssery, Bowman, Feng, arXiv:2404.13076). Mitigations, strongest first (interpretation, grounded in the cited mechanisms): (a) a deterministic check first — most system errors admit one: reproduce the error, apply the fix, re-run; that is the AZR executor pattern, “a code executor to both validate proposed code reasoning tasks and verify answers” (arXiv:2505.03335), and it is strictly stronger than any judge; (b) verifier from a different model family; (c) the verdict used as a filter (reject and retry), never as a self-reinforcing training target; (d) periodic human calibration on a sample.

A free bonus in this design: real errors arrive at the natural difficulty frontier. R-Zero pays its Challenger “for proposing tasks near the edge of the Solver capability” (arXiv:2508.05004); a production error stream is that, sourced from reality instead of invented — the property the literature has to engineer, this architecture gets from the environment.

The human-audited variant. If a human rates (or comments on) fixes instead of a verifier agent — (error, fix, human rating) → enhancer — the loop is no longer fully self-improving: humans are in the signal path. Three readings, honestly separated:

• Human ratings replace the verifier → this is the human-feedback improvement loop (iterative RLHF-style retraining), not RSI in the literature’s sense — the papers defining the field explicitly position against human signals: “without relying on any external data” (AZR), “generates its own training data from scratch” (R-Zero), “only unlabeled test data” (TTRL, arXiv:2504.16084).

• Human ratings only audit and calibrate the verifier on a sample, while the loop runs model-verified between audits → an RSI loop with a quality anchor. “Semi-RSI” is a fair coinage for it — but it is the author’s label, not a literature term, and the verified mitigations above are what make the anchoring work.

• Human comments are compiled into a durable verifier artifact — a regression test — and the human exits the signal path for that error class. The loop error → fixer → human comment → compiled test → growing suite gating future fixes is a composition of verified practices, none of them new individually. Production failures as the task source: “failures become test cases, test cases prevent regressions, and metrics replace guesswork” (Anthropic, Demystifying evals for AI agents, Jan 9 2026); “each failure is a ready-made test case for the next cycle” (Google, agent-quality flywheel, Jun 30 2026). Bugs-as-tests is textbook SRE: “start documenting all reported bugs as test cases. If every bug is converted into a test, each test is supposed to initially fail because the bug hasn’t yet been fixed” (Google SRE book, ch. 17). A human stated worry compiled into a gating metric ships in Google’s flywheel skill — a plain-language concern becomes a custom rubric “that I can count, gate on (act if more than 20% come back IGNORED), and track cycle over cycle”, with “It proposes; you approve. Human-in-the-loop, not hands-off.” (same Google post). The near-novel node is automatic compilation of a free-text comment into an executable test: the closest verified work is academic — SWT-Bench’s agents “formalize user issues into test cases” and its “generated tests are an effective filter for proposed code fixes, doubling the precision of SWE-Agent” (arXiv:2406.12952); Issue2Test generates “a test that fails, and that fails specifically for the reason described in the issue” (arXiv:2503.16320) — plus eval tooling where the durable artifact is a dataset rather than a repo test (LangSmith: “Add runs that received negative feedback to test against”, “exporting annotated runs directly to datasets”; Braintrust: “Add human feedback, labels, and corrections to build test cases”). No verified project runs the full loop end to end.

  The design risk concentrates in the compiler: a bad synthesized test poisons the gate permanently, and the verifier-side literature supplies an admission checklist (evidence and labels in research-workspace/rsi-human-feedback-test-compilation-evidence-2026-09-20.md in the site repo):

  • Fail pre-fix, pass on the reference fix — SWE-bench admits only instances “with at least one test where its status changes from a fail to pass”; validation compares outcomes “both before and after applying the PR’s patch”, repeated until “instances with consistent results across all runs” remain (arXiv:2310.06770, SWE-bench-Live §3.4). For this loop the human-approved fix is the gold patch; validating a compiled test only against the fixer’s own fix is circular (interpretation).
  • Keep passing behavior green — PASS_TO_PASS tests check “that existing unrelated functionality in the codebase has not been broken” (OpenAI, SWE-bench Verified).
  • Audit the oracle against what the comment actually says — reject tests “easily gamed” (an explicit OpenAI annotation category), and OpenAI’s 2026 taxonomy of flawed tests: 35.5% “narrow” (enforcing implementation details absent from the spec) and 18.8% “wide” (checking unrequested functionality). Human ratification is not a one-time fix: even after a 93-developer, 3-annotator campaign flagged “unit tests that may unfairly mark valid solutions as incorrect” on 61.1% of samples, the 2026 audit still found 59.4% of hard failures had flawed tests — so the suite needs an audit-and-removal loop, not just admission criteria (openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/, Feb 23 2026).
  • Triage the comments — Anthropic’s own eval guidance treats user feedback as “sparse and self-selected” and notes “users rarely explain why something failed”: bug-class comments compile into tests; taste and environment-specific one-offs should land in the fixer’s prompts or memory instead (interpretation on sourced caveats).
  • Keep the grader decoupled from the fixer — “The optimizer never grades its own work… An optimizer that grades itself learns to game the metric” (Google flywheel); “Give Claude something that produces a pass or fail, and the loop closes on its own” (Claude Code docs); OpenAI’s Codex Security ships the same shape — a fix “adds a focused regression test that fails before the fix and passes after it”, and when no test is feasible it “records the proof gap and provides the strongest repeatable validation artifact instead” (fix-findings docs).

  Read structurally, this third reading moves the loop out of the “human in the signal path” family altogether: after compilation the human is out of the path, the verifier is a deterministic artifact, and the loop sits in the AZR-executor/DGM-benchmark cell of the matrix above — with the benchmark itself grown from production errors. Anthropic’s long-running-agent harness already runs this shape internally: a fail→pass feature list gates its fixer agents, and the rules say “It is unacceptable to remove or edit tests because this could lead to missing or buggy functionality” (Anthropic, effective harnesses, Nov 26 2025).

Labels: the classification of these three variants is derivation from the cited mechanisms, not a sourced ruling; no verified paper rules on this exact production pattern. The third reading’s per-link precedents are verified (Anthropic/Google/OpenAI official guidance, SWE-bench machinery, ReCrash ECOOP 2008, Defects4J, Google SRE ch. 17 — details in the follow-up note), but the end-to-end production pattern — a human comment auto-compiled into a committed repo test that gates merges — is not verified to exist as a shipped system in the sources gathered this session.

The four mechanisms you can use without training anything

Each: the sourced mechanism, the agent-setup reading (interpretation), and the failure mode.

1. Verifier-gated candidate generation — “generate many, let a check pick.” The one ingredient every successful verified system shares. STaR keeps “the rationales that ultimately yielded correct answers”; AZR’s executor validates and verifies; Ansor “fine-tunes the sampled programs with evolutionary search and a learned cost model”; MLGO’s models “achieve up to 7% size reduction” in LLVM (STaR, AZR, Ansor OSDI ‘20, MLGO). In your setup: sample N candidates from the frozen model; let tests, lints, benchmarks, or a script’s exit code decide. Failure mode: only works where checking is cheap and honest — the verifier gap is the boundary of the whole literature.

2. Self-generated evals (the Challenger/Solver pattern, minus the RL). R-Zero’s “Challenger… rewarded for proposing tasks near the edge of the Solver capability” lifted Qwen3-4B-Base “+6.49 on math-reasoning benchmarks and +7.54 on general-domain reasoning benchmarks” with “any pre-existing tasks and labels” absent; SPICE mines a corpus for the same effect (+8.9% / +9.8%) (R-Zero, SPICE). In your setup: have a model mine your docs, repo, and backlog into hard questions; keep only tasks that discriminate between agent configurations — an eval suite that compounds (eval-driven workflow). Failure mode: the Challenger can generate hard-but-wrong or degenerate tasks; no verified evidence the pattern transfers to taste-dependent domains.

3. Harness iteration under a benchmark (DGM-style, minus the lab). The DGM improved SWE-bench “from 20.0% to 50.0%” and Polyglot “from 14.2% to 30.7%” by having a foundation model mutate the agent’s own code, with “safety precautions (e.g., sandboxing, human oversight)” (arXiv:2505.22954). In your setup: treat the agent’s harness (prompts, skills, memory files, tool configs) as the improvement target, the eval suite as the reward, every change re-passing a fixed benchmark with a rollback path (companion survey found this loop deployed in memory/skills form). Failure mode: benchmark-overfitting — the harness learns to please the eval, not the user; hence the DGM authors’ sandboxing and oversight.

4. Improving compute and code around the model. AlphaEvolve found a 4×4 matrix multiplication procedure “using 48 scalar multiplications; offering the first improvement, after 56 years, over Strassen’s algorithm in this setting”, improved data-center scheduling (“continuously recovers, on average, 0.7% of Google’s worldwide compute resources”), and “accelerated the training of the LLM underpinning AlphaEvolve itself” (arXiv:2506.13131, DeepMind blog). The pre-LLM versions are pure search: Ansor’s schedules improved kernel speed “up to 3.8×, 2.6×, and 1.7×”; AlphaDev’s sorting routines are “now available in the LLVM libc++ standard sorting library” (DeepMind blog, Jun 7 2023). Even OpenAI’s internal “RSI Index” benchmark suite measures “kernel optimization, tiny training runs” (vendor claim; see our benchmark review). In your setup: build scripts, kernels, CI heuristics, eval harnesses — all improvement targets with objective scores. Failure mode: objective scores invite hacking; a kernel that wins the microbenchmark can lose the workload.

What you cannot import

• Weight-space self-training. STaR, AZR, R-Zero, SPICE, and SEAL all end in a training loop — SEAL’s self-edits “result in persistent weight updates” (arXiv:2506.10943) — with GPU budgets and reward design an API-only builder doesn’t have. The loop and the verifier survive; the fine-tuning does not.
• The model as its own reward. Self-Rewarding LMs use “LLM-as-a-Judge prompting to provide its own rewards during training” (arXiv:2401.10020). You can use LLM-as-judge without training, but treat it as a directional filter, not a measurement — the self-preference bias above is the documented reason.

The matrix that justifies “yes”

Read the last column: four of the mechanisms above run on frozen models, and the two strongest verifiable wins (AlphaEvolve, DGM) are loops an agent builder can run. (Full matrix with the remaining rows: research-workspace/rsi-self-improvement-evidence-survey-2026-09-19.md in the site repo.) Label: the mapping of published mechanisms to your setup is interpretation built on sourced mechanisms; the sourced facts are the mechanisms and their measured results, not the transfer.

Limits, honestly

• Published recursion is shallow. STaR runs a handful of iterations; Self-Rewarding three; SEAL two ReST-EM rounds. Nothing verified demonstrates open-ended self-improvement — that remains a forecast (Good 1965; Bostrom’s “seed Artificial Intelligence”, archived OUP catalogue), not a result.
• The verifier gap decides everything. Cheap, honest checks (tests, executors, game outcomes, benchmarks) are where every verified win happened; subjective domains are where self-judge loops stall or drift (arXiv:2404.13076). Design the loop around what you can measure.
• If you build the loop, build the guardrails. The DGM authors ran with sandboxing and human oversight — the floor, not the ceiling, for a loop that edits its own harness (evals as the measuring side).

Methodology and Sources

Method notes. Research date Sep 19, 2026; follow-up round Sep 20, 2026. Quotes are verbatim from fetched pages, fetched scans, or journal records. Access caveats: nature.com/science.org article pages redirect to an IDP login and were not directly fetchable, so those items are verified via official DeepMind blogs (which link the exact article URLs), Europe PMC/MEDLINE records, and Crossref/OpenAlex metadata — labeled journal-record verification rather than page fetch. In the follow-up round, openai.com returned HTTP 403 to direct fetches, so both SWE-bench Verified posts were verified via a reader proxy (r.jina.ai) of the identical URLs; dl.acm.org 403’d and PDF content-types were rejected by the fetcher, so those paper claims come from arXiv abstract pages or the Semantic Scholar API. The follow-up round’s per-source verification record (parent-verified vs. delegated-verified URLs) is in research-workspace/rsi-human-feedback-test-compilation-evidence-2026-09-20.md. The complete evidence survey (deep history, full paper table, access caveats) is in research-workspace/rsi-self-improvement-evidence-survey-2026-09-19.md. Everything labeled interpretation is the author’s transfer of a sourced mechanism to an agent-building setup; the architecture classifications in the worked example are derivations from cited mechanisms, not sourced verdicts; vendor numbers (RSI Index) are vendor claims.

Sources

• Good 1965: scan (Language Log) · Quote Investigator, scan-verified · DOI record
• Schmidhuber Gödel machine: arXiv:cs/0309048 · author’s page
• AlphaGo Zero: Nature 550:354–359 · Europe PMC/MEDLINE abstract · DeepMind blog
• FunSearch: Nature 625:468–475 · Crossref abstract · DeepMind blog
• STaR arXiv:2203.14465 · Self-Rewarding arXiv:2401.10020 · TTRL arXiv:2504.16084 · Absolute Zero arXiv:2505.03335 · SEAL arXiv:2506.10943 (project page) · R-Zero arXiv:2508.05004 · SPICE arXiv:2510.24684 · survey arXiv:2507.21046
• Self-preference bias: arXiv:2404.13076 (Panickssery, Bowman, Feng, Apr 2024)
• Ansor: USENIX OSDI ‘20 · arXiv:2006.06762 · MLGO: arXiv:2101.04808
• AlphaDev: Nature 618:257–263 · DeepMind blog · AlphaTensor: Nature 610:47–53
• AlphaEvolve: arXiv:2506.13131 · DeepMind blog
• Darwin Gödel Machine: arXiv:2505.22954 · Sakana AI blog
• Bostrom: archived OUP catalogue page · nickbostrom.com
• Site companions: Self-Improving Agent Frameworks vs. Smart Memory Management · Evals and Agent Memory · Frontier LLM Releases of Summer 2026 (RSI Index figures — vendor claim)
• Follow-up round (Sep 20, 2026), comment→test-compilation precedents: Anthropic Demystifying evals for AI agents (Jan 9, 2026) · Effective harnesses for long-running agents (Nov 26, 2025) · Claude Code best practices · Google agent-quality flywheel (Jun 30, 2026) · harness engineering (Sep 9, 2026) · SRE book ch. 17 (© 2017) · Testing blog: flaky tests (Apr 17, 2017) · OpenAI SWE-bench Verified (Aug 13, 2024; reader-proxy verified) · Verified retirement audit (Feb 23, 2026; reader-proxy verified) · evaluation best practices · Codex Security fix-findings · SWE-bench arXiv:2310.06770 · SWE-bench-Live arXiv:2505.23419 · SWT-Bench arXiv:2406.12952 · Issue2Test arXiv:2503.16320 · Long & Rinard arXiv:1602.05643 · Xiong et al. arXiv:1706.09120 · Qi et al. DOI 10.1145/2771783.2771791 (via Semantic Scholar) · Barr et al. oracle survey DOI 10.1109/TSE.2014.2372785 (via Semantic Scholar) · DeFlaker DOI 10.1145/3180155.3180164 (via Semantic Scholar) · ReCrash, ECOOP 2008 · Defects4J · BugSwarm · LangSmith evaluation concepts · LangSmith annotation queues (Oct 26, 2023) · Braintrust annotate docs · TDD-Bench Verified arXiv:2412.02883 · AssertFlip arXiv:2507.17542 · PRIMG arXiv:2505.05584 · TestGenEval arXiv:2410.00752