Safety for recursive self-improving AI — taxonomy, risk discovery, and evaluation.
When AI systems begin to improve themselves — writing memories, learning skills, updating their own models and evaluators — safety is no longer a property of a single checkpoint. Evolutionary Safety studies how safety properties are preserved, weakened, inherited, and restored across the entire improving process.
Everything we build lives in the open on GitHub. Whether you want to read, or to run — pick your entry.
A continuously updated awesome list for the emerging literature on safety under self-improvement — organized along the five-domain taxonomy of our survey, so you always know where a paper, benchmark, or defense fits.
YUVANE — an open-source evaluation system for measuring how safety changes across self-improvement cycles: not just whether a system is safe today, but what each accepted update does to safety tomorrow. Under active development — the public release is coming soon.
A perspective from our survey: safety analyzed as a property of an evolving process, not of a static system.
Evolutionary Safety studies how safety properties are preserved, weakened, or restored as AI systems retain and build on the results of their own improvement processes — across updates, interacting systems, and descendant lineages.
— Evolutionary Safety of Recursive Self-Improving AI: Taxonomy, Risk Discovery, and Evaluation (2026)
A safety failure becomes evolutionarily consequential when its effects persist, influence later selection, or propagate to successors. The risk life cycle has five stages:
A candidate change is proposed: a memory write, a skill, a parameter or evaluator update.
The evaluator admits it — a high task score can hide a weakened safety check.
The change persists in memories, skills, tools, workflows, or model weights.
Descendants and downstream workflows build on the retained artifact.
The risk spreads across artifacts, models, and interacting systems.
Recurring forms of safety degradation — a diagnostic ladder from early drift to spread.
Goals, constraints, and priorities quietly shift across repeated updates.
Small deviations compound as later updates build on them.
Misleading or poisoned material enters what the agent retains.
Capabilities improve while previously reliable safeguards degrade.
Biased criteria reshape what gets selected and retained.
Risks persist and spread across artifacts and successors.
The taxonomy locates risks by their carriers — the components that move a change into the next round of improvement.
Memories, skills, tools, and workflows carry lessons — and failures — forward.
Fine-tuning, self-training, and merging encode change into parameters and descendants.
Judges and experience streams shape which candidates survive.
Execution layers decide which safety controls can actually be enforced.
The updater itself — and adapting peers — change the rules of improvement.
Four complementary units, from a snapshot to a family tree; governance lives inside the loop, not after it.
Risk and control readouts at a point in time.
Before/after comparison around one accepted change.
Accumulation and delayed effects across rounds.
Propagation across descendants and derivation graphs.
Where the research frontier currently sits.
The mechanisms that make safety evolutionary are no longer hypothetical — they are shipping in research systems today.
Reflexion-style memories, Voyager-style skill libraries, and feedback-driven tool evolution make one interaction change the starting conditions of the next.
A shortcut that gets stored and reused survives its own episode. Deleting the source does not delete the descendants.
A final benchmark number cannot show erosion between checkpoints. Safety must be followed across states, updates, trajectories, and lineages.
Project homepage launched. One place for the survey, resources, and the evaluation system.
YUVANE evaluation system in preparation. States, updates, trajectories, and lineages — made measurable. Public release coming soon.
Survey preprint out. Evolutionary Safety of Recursive Self-Improving AI: Taxonomy, Risk Discovery, and Evaluation.
If you use the resource list or the evaluation system in your work, please cite our survey:
@article{gong2026evolutionary,
title = {Evolutionary Safety of Recursive Self-Improving AI:
Taxonomy, Risk Discovery, and Evaluation},
author = {Gong, Chang and Bi, Jingping and Yao, Di
and Liang, Xinjian and Xiang, Chao and Guo, Ruijie},
year = {2026},
journal = {arXiv preprint arXiv:XXXX.XXXXX},
note = {Institute of Computing Technology,
Chinese Academy of Sciences}
}