We Built a Fully Autonomous CRO Agent. Three Things Broke.

A wave of tools now promises the same thing: point AI at your store, and it will discover what is leaking revenue, implement the fix, run the A/B test, launch the winner, and drive your numbers up. End to end. No humans in the loop.

It is a beautiful pitch. We believed in it enough to build it.

Back in January, we set out to build a fully autonomous CRO system, one that could go from diagnosis to deployed change without a person touching it. Not as a demo. As the actual product. It broke in three specific places, and those three lessons changed how we think about the entire category.

The short version

• Finding the real conversion leak was harder than generating the fix.
• Implementation needed brand taste and human judgment, not just functional correctness.
• Maintenance turned a one-time fix into long-term ownership of someone else's storefront.

First, the part everyone underestimates: the finding

Start with the constraint everyone forgets. You do not have unlimited traffic.

You cannot run 100 A/B tests in parallel. Most brands cannot run five with real statistical significance. A store doing 8,000 sessions a month waits weeks for a single test to reach confidence, and every day the worse variant is live, you are sending real shoppers through it. At today's acquisition costs, that is thousands of dollars a day burned to learn something a good simulation should have caught for free.

That is exactly why synthetic shoppers are appealing, and why even Shopify is moving in that direction. Earlier this year, Shopify introduced SimGym, an AI Research Preview app that creates AI shoppers to browse products, navigate collections, add items to cart, and provide qualitative feedback before changes go live. It is a smart idea from the company with the most commerce data on the planet.

Shopify's own framing is the important part: research preview. The engineering write-up is directionally exciting, but it also makes clear that the hard work is the simulation infrastructure itself. Simulated shoppers are useful when they make a better pre-launch signal. They are dangerous when teams treat them as measurement.

That is not a knock on Shopify. It is the opposite. If even the team with that much data is still framing AI shoppers carefully, that tells you the real work was never running a simulation. It is building the harness that makes the output trustworthy. The hard part of AI-powered CRO was never the fixing. It is the finding. And making the finding trustworthy is the whole game.

That is the public version of a problem we hit privately. What follows is our own scar tissue.

Wall 1: Raw simulation is confidently wrong until you harness it

This was the first wall, and the most important.

Out of the box, AI analysis and user simulation are non-deterministic. Run the same audit twice and you can get different findings. The model also brings its own baggage: stereotypes about how "a mobile shopper" or "a deal seeker" behaves, and a tendency to over-anchor on whatever detail it noticed first.

The output looks authoritative. Clean report, confident tone, a specific number attached. And a meaningful share of raw, ungrounded output is simply wrong.

Here is the trap. If you bolt full automation on top of a discovery layer you have not hardened, you have not saved anyone time. You have built a machine that ships wrong conclusions faster, at scale, with conviction.

Trustworthy simulation is not a switch you flip. It is a system you build: grounding every finding in real data, cross-checking it against multiple signals instead of one, and keeping a human in the loop to catch what the model is confidently wrong about. So we put our effort there, into making the diagnosis trustworthy, rather than into automating deployment on top of one that is not.

Wall 2: Implementation is taste, not just function

Some fixes are objective. A broken size filter, overlapping form fields on mobile, a checkout step that does not fire. Those are right or wrong, and an agent can handle them.

Most CRO changes are not like that. Most of them are editorial. How a value proposition is phrased. Which proof point leads. How a product page feels. Every brand has a point of view about its own taste that no brand-guideline document fully captures. We would generate an on-brand page, and the brand would have a specific, valid opinion the guidelines never mentioned.

So we built a workflow where the brand could generate, iterate, and finalize the content as part of the process. The human approval gate stopped being a courtesy. It became a quality requirement. Autonomy can produce the draft. It cannot own the taste.

Wall 3: Maintenance is a liability you inherit

This one we did not see coming.

We had early wins with smaller brands where our agents generated on-brand content and pages automatically. It worked. And the moment it worked, the request changed: would we maintain it? Keep it compatible with the next theme update, with every app change, with the other platforms they wanted to expand to.

Full autonomy demands full trust, and full trust quietly turns into full ownership. "We will just do it for you" becomes "we now own your store's maintenance, indefinitely." Even when our changes were correct, we inherited the long-tail overhead of keeping them alive on a platform that updates constantly. That cost is real, and it is exactly what fully-autonomous claims leave out of the pitch.

There is a quieter fourth wall worth naming: closing the loop. To know whether an autonomous change actually worked, you have to measure it. But most A/B testing platforms gate programmatic access behind their enterprise tiers, so an agent on a mid-tier plan cannot even read or write its own tests. It is no accident that the companies moving fastest here own the entire stack, so they never have to ask anyone for access.

What we actually did about it

Both paths pointed to the same conclusion. The prerequisite for any trustworthy end-to-end system is not a better deployment pipeline. It is a discovery layer whose output you can actually trust: findings that are accurate, actionable, and ranked by clear impact.

Brands do not lose money for lack of an autonomous button. They lose it by acting on the wrong diagnosis, or by being too scared of breaking something to act at all. Get the diagnosis right, and every action that follows, whether human or agent, becomes safe and high-ROI.

So that is where we put our weight. Diagnosis first. Autonomy is earned downstream, not assumed upfront. We still believe the end-to-end vision is the right direction. We just do not think you can get the next move right without real confidence in what that move should be.