Something quietly strange is happening to the release calendar. Not long ago, a major AI model arrived every six to twelve months. Now the gap is measured in weeks.
The name for where this is heading is recursive self-improvement, or RSI. The idea is old, a mathematician sketched it in 1966, imagining a machine that could design a better machine, which could then design a better one and so on, faster each time. For sixty years it stayed a thought experiment. Now it is starting to happen for real. It is worth understanding calmly, without the hype or the doom, because underneath the jargon it is something every delivery team already knows well – Agile.
You Already Run this Loop
Agile, stripped to its engine, is a feedback cycle. That is building a small thing, putting it in front of reality, looking honestly at what came back, and adjusting the next cycle accordingly. The retrospective is the ceremony where a team improves the way it works, not just the product. The whole philosophy rests on the belief that a team that inspects and adapts will get better at getting better over many iterations.
This is exactly the core concept of recursive self-improvement. The only difference is who is holding the pen.
In Agile, humans run the loop. Humans decide what “done” means, humans sit in the review, humans own the retrospective. RSI is what you get when the build, the measurement and the retrospective all move inside the machine. The system that did the work also runs the retro on itself and ships its own improvement before the next stand-up would even have happened. Seen this way, RSI is not an alien concept. It is the Agile loop with the human ceremonies automated away. That framing is useful precisely because it tells you what is gained (speed) and what is quietly lost (the checkpoints those ceremonies always were).
What “improves itself” Actually Means
“The AI improves itself” is a sentence that hides a lot of engineering. In practice, a self-improving system can change itself at three different layers:
- It can rewrite its own weights. This is the deepest layer. The model updating its own parameters through continued training on its own experience or on data it generated. This is powerful and also the hardest to do safely, because a bad update degrades the model itself, and you cannot always see it coming.
- It can rewrite its own prompts and code. This is the layer doing most of the real work today. The model doesn’t touch its weights; it edits the scaffolding around itself, e.g., its prompts, its instructions, the code of the agent it runs inside. This is the same territory as loop engineering. You stop hand-prompting the agent and instead design a system that prompts the agent, checks the result, and decides the next move. When that system starts editing its own prompts and scaffolding to do the job better, you have a modest, bounded form of self-improvement.
- It can build and patch its own tools. The system writes or fixes the tools and controllers it depends on, e.g., a better search function, a more reliable deploy script, a patched robotics controller learning from live telemetry. It gets smarter not by thinking harder but by giving itself sharper instruments.
Sitting across all three is a memory layer. It’s the accumulated record of what worked and what didn’t, so the system stops re-learning the same lesson every session. And above all of it is the one that actually moves the release calendar: the meta-loop, where AI improves the pipeline that builds the next AI.
The Checkpoint You’re Automating Away
Here is the part that connects back to something I posted earlier, in Govern the Action, Not the Agent.
That post argued that the workable way to govern autonomous AI is not to police the agent’s reasoning but to gate the moment of consequential, irreversible action, the same way we govern physicians, judges and financial officers. We don’t audit their every thought; we require evidence at the point where their decision becomes real. Govern the action, not the agent.
Recursive self-improvement is the sharpest test of that idea. Here is why. Let’s go back to the retrospective. In Agile, this ceremony was never only about improvement, it’s also a governance checkpoint. It was a recurring moment where humans looked at the work, judged whether it was good, and decided whether to continue. Every sprint review, every definition of done, every stakeholder demo was a place where human judgement re-entered the loop.
RSI removes those moments. When the system runs its own retrospective on its own schedule, the natural human checkpoints simply aren’t there anymore. And this is precisely why “govern the agent” fails as a strategy. You can’t supervise a reasoning process that iterates a thousand times before you’ve finished your coffee. The only thing that scales is to stop trying to govern the agent and instead put an immovable checkpoint at the action, e.g., the deploy, the release, the irreversible change to the next model, where a human-defined gate says this may proceed, or this may not.
In other words, the faster the loop learns to run itself, the more the earlier argument holds. The checkpoint has to move off the agent and onto the action. Speed doesn’t remove the need for governance.
What Stays Human
If you take the Agile lineage seriously, the answer to “what should humans keep” is already written in the practice. The retrospective can be automated. The judgement of value should not be. The definition of done is not a technical parameter. It is an act of accountability, and accountability has to sit with someone who can be held to it.
So the practical posture, whether you’re running a delivery team or wiring up an agent that improves itself, is the same: let the loop run fast, but keep the gate. Design the checkpoint into the action, define “done” like someone who intends to stay accountable for it.
The machine can run the retrospective. It still shouldn’t be the one deciding what the retrospective was for.