Most "self-improving" AI agents don't actually improve
Out of 9 running in production, only 2 had a key element: verifier grounded in the real world
Exec summary
Despite the hype, public examples of true self-improving loops are few and far between.
A self-improving loop is only as good as its verifier, and a verifier is only as good as the real-world data grounding it.
Building verifiers is hard, slow, and complicated, and where most of the competitive opportunity is today. Most AI engineering is trying to side-step them.
Before you sponsor a project that builds one, align the resources: (1) what real-world data grounds the verifier? (2) who provides expert feedback on failures and what does that cost?
Loops are having a moment. The poor souls who take it upon themselves to be their organization’s Most Plugged-In AI Person were in for a treat over the last few weeks: Boris Cherny, (creator of Claude Code), shared his Steps of AI Adoption, a de facto blueprint of what it takes to move past the bottlenecks in automating work; co-founder of Arize AI Aparna Dhinakaran broke down the theory of loops. Gergely Orosz of The Pragmatic Engineer quickly followed with a piece rightfully dispelling some of the hype.
No doubt many organizations are experimenting with self-improvement. But if you look for documented cases of self-improving loops in production, the list gets short quickly; among 9 use cases I found and looked into, only two disclose enough to verifiably call the loop “self-improving”.
Verifier is key
In its simplest form, a loop has three components:
The execution loop, which is whatever the automation actually does; say, monitors the bug queue and fixes them;
The verifier - checks the output of the execution loop against some standard you established (if in your mind verifier is served well by golden-data-evals alone - please read along);
Feedback - if the verifier flags an issue with the execution loop, the context of the issue is fed back into the execution loops for another spin.
And it’s in the verifier the real bottleneck of “self-improving” lives.
Why verifiers are expensive and difficult
Verifier, despite what seems to be an increasing trend in the industry, must be grounded in a steady trickle of real-world data; if it isn’t, it can (and will) optimize in a random direction.
For deploying code, this “steady stream of real-world data” is relatively straightforward, even if it requires some creative stitching: tests can pass or fail, production telemetry can catch anomalies, code history can be audited, A/B tests can run, and user feedback and bugs queue can be monitored.
As you move further outside of code and into knowledge work, figuring out the “steady trickle of real-world data” needs some creativity.
This need for creativity is why I like to see how various businesses have tackled the problem, but evidence on in-production loops is pretty thin. Kudos to the folks behind the two examples I break down below: Replit’s product development agent and Tax AI agent, built by OpenAI’s FDEs (PS. are FDEs a revival of 2010s implementation consulting?).
Agent drafts, human ships: current state of self-improving loops
Exhibit 1. Replit’s self-improving coding agent
The use case for Replit’s agent is straightforward: identify product issues, write the code and submit the build. From here, verifier takes over, including Replit’s benchmark built specifically for vibe-coding: it simulates the app and user’s clicks within it; if passed, the proposed change runs through A/B testing, and results are sent to a human for review to ship, iterate, or drop.
For me, an even more interesting element of Replit’s loop is Telescope - their proprietary traces clustering tool1, which allows to analyze traces at scale without unnecessarily exposing user data to random eyeballs. Plus, it looks cool.
Exhibit 2. Tax AI agent, built by OpenAI x Thrive
Tax AI set out to tackle the ambiguous and spottily-documented process of tax preparation. Since wading through hundreds of documents to prepare a draft is the most labor-intensive part of the CPA’s job, the appetite for automation was high enough to dedicate the time to weeks of painful - but effective - mechanical turking.
What I delightfully noted in OpenAI’s use case is that the agent improved drastically post-launch2. The percentage of drafts that were at least 75% correct went from roughly a quarter at launch to 86% - vast majority - at six weeks. The result comes from FDEs pulling teeth asking CPAs to explain each decision where their judgment differed from the agent.
Think about this process: one tax draft has dozens - or hundreds - of fields, each requiring data and judgement - sometimes formal, sometimes not. Imagine those six weeks and the amount of good will required on both sides, the FDEs sending draft after draft to the CPAs, asking them to explain why and how the agent is wrong, field-by-field. Slow, complicated, and expensive, and inevitable in navigating the muck layer of knowledge work.
This is the exact challenge with self-improving agents: is it possible to make agent’s failures visible and understandable? Is it feasible to get the context from the experts on enough of the failures to meaningfully improve the agent? This time and buy-in might be one of the biggest bottlenecks in AI adoption today.
If your team has solved this creatively, tell me more. If your team is struggling with figuring this out, also reach out, I’d love to workshop some solutions.
What about automated evals?
Inevitably, someone will say: we already have verifiers - we have an automated eval system, thank you very much, all taken care of.
Auto-evals are sort of a trojan horse of agent development - they are cheap, fast, simple, privacy-preserving, loops can spin indefinitely without being slowed down by humans - tempting to set them up and call it a day.
Hamel Husain, an ML engineer known for his work on LLM evals, recently ran an open test that proved autoevaluators did relatively well on known criteria. But because of criteria drift, it’s in the “undefined” where the customers weep and dollars leak:
The premise of criteria drift is that you don’t know how to evaluate an agent until your human eyeballs observe it interacting with the real world.
Even if you assume you do know all possible failure modes of your agent, there’s the increasingly ripe problem of reward hacking, where an agent optimizes for a score but not the intended behavior - for example, if rated on politeness, the agent might hang up on an angry customer; if rated on sales, the agent might give 99% discounts, or close $0 sales, or offer hallucinated perks - all of which you can catch, but not until you know it can happen.
Which needs real-world input.
Autoevals are one way of shooting yourself in the foot and sidestepping the verifier problem instead of solving it; other ways include: LLM-as-judge, synthetic data, “we’ll add evals later”, and golden data that stays static.
Don’t let the hype tell you that everyone’s automated their business with loops already. Before you sponsor a self-improving loop project, make sure you understand the messy middle between the launch and results, and that you’ve aligned the resources: (1) what real-world data grounds the verifier? (2) who provides expert feedback on failures and what does that cost?
That is, only if you have good verifiers.
I write for people betting careers and businesses on AI from the perspective of having shipped across three waves of emerging tech in startups and Big Tech - if that’s you, subscribe to get your weekly under-5-minutes-to-read-weekly-email.
Disclaimers: opinions my own. Data for this piece was sourced from public sources and private AI communities.
Built based on Anthropic’s Clio paper, an interesting solution for companies (rightfully) iffy about giving their development teams broad access to traces.
There’s a lesson here for any exec sponsoring an agentic team or builder working on self-improving loops: the launch result is a start, not the end. The only numbers that matters is how quickly you get to a launch, and how quickly you improve from there. The results at launch are nearly irrelevant.