The Fable of the Hare and the Hedgehog

Preamble: from AI Pods to a ticket-to-PR pipeline

Aquiva keeps investing in AI tooling and in ways to accelerate delivery. The most visible result is the AI Pod: a small managed team on a fixed monthly rate. An architect who orchestrates, two builders, a product owner, two workstreams, something in the customer's hands roughly every five days.

Most of the work we do in AI Pods will stay complicated work that needs smart engineers working hand in hand with AI in an IDE. But there is other work as well, sustaining projects or simply the easier tickets, and that is work we could hand over. So we built a pipeline that takes a GitHub ticket and turns it into an implemented pull request, to find out where the upper bound is: how much of a ticket can be offloaded completely.

Planning turned out not to be fully delegatable. The pipeline stops there and waits for a human to approve the plan and correct it, and we are looking at bringing Spec Kit into that part of it. If you want to read more about this, Aidan has written an article on this blog about an IDE skill that helps our developers in AI Pods specify a feature properly before they let AI help them implement it. The building part is where the pipeline runs on its own. This article is about how it got there.

Early attempts: simple ticket automation for My Org Butler

We built a pipeline and grew it one step at a time. Some of the versions we were proudest of turned out to be cheating, which took the longest to see and is the reason this article is named after a fairy tale.

The pipeline did not start within a customer project, but as a way to work on small tickets on My Org Butler, our Open Source Agentforce blueprint repository. Even though we struggled with implementing simple Salesforce tickets, the machinery we had built around it was already substantial. A durable scratch org per ticket, created once and then cached and reused for as long as the issue and its pull request are alive, so a run does not pay the setup cost again on every round. Apex tests, Salesforce Code Analyzer with our own PMD rulesets, and Agentforce Testing Center evaluations, all of them running automatically on every round. And a browser driven by Playwright MCP, which takes the screenshots that the requirement names and records a demo video of the feature working in that org.

Nonetheless the pipeline was really bad at implementing even simple Salesforce features. It did not matter whether we ran Sonnet or Opus, both struggled hard with building the more complicated XML metadata and getting it deployed to the platform. Writing skills that spell out how we build helped, but we didn't have the appetite to create a heavy skill corpus of "How to develop in Salesforce".

Instead we added an elaborate learning mechanism following Andrej Karpathy's idea of an LLM Wiki: we gave the pipeline a knowledge base that it wrote into and refactored over time instead of a log that it only appended to. At the end of every run it wrote down the learnings for the next one.

Indexing all of it and finding the right entry again was hard enough, but the real problem was what ended up in there. The pipeline did not store knowledge, it stored the quick hacks it had guessed its way into: it tried a lot of things, and whatever happened to work was written down as a lesson. We added web search so it could check the current Salesforce documentation rather than trust what it remembered, and that only made it worse, because the web is full of wrong and hallucinated Salesforce advice as well. In the end we even had our own review mechanism on top of it, and by then the whole thing was so complicated and returned so little that we took it out again.

What worked much better was giving it existing work to learn from. It was allowed to read one or two of our reference repositories and told to follow the structure and the conventions it found there instead of inventing an approach of its own. That was one of the major early breakthroughs, and conventions transfer that way better than any instruction we ever wrote. It was also the backdoor that allowed the pipeline to trick us later.

The Fable moment - a short spell of magic

Anthropic released its Fable 5 model, and mostly out of fun I gave the pipeline a really big ticket. A full customer statement of work describing a complete Agentforce AppExchange PoC, the kind of scope a human team works on for weeks.

It delivered the whole thing. Even though nothing else had changed from the evolution step before. Same pipeline, same skills, same loop, only a different model behind it.

Proper prompt templates instead of hardcoded strings, Gen AI topics and actions wired into a bot template that deploys, Apex that uses our own libraries rather than writing naive code, tests with real assertions and Agentforce agent tests that were not trivial. Even the AppExchange security review documents were ported over from our existing projects and came out right. Including PMD suppressions, false positives and fixes.

It also got itself out of trouble. It ran into complex Agentforce deployment issues but we saw it finding its way out by doing variations, searching the web and crawling existing learning files.

After a few hours we had a pull request with 80 files in it and demo videos that proved the app was there. That Fable was able to do this blew me and everybody at Aquiva away, and we spent that week expecting the future to be bright.

Adversarial reviewer: a weaker model needs a stronger harness

Then Fable 5 was suspended for a few weeks shortly after its release. It came back, but a pipeline that depends on a model which can disappear overnight is not something we can offer a customer, so it had to work on the models that stay available.

We ran the same ticket on Opus 4.8 and got a result we could not trust. The run reported that everything was delivered, while the org looked different, the tests covered less than they appeared to, and parts of the specification had not been implemented at all. Writing our conventions down as skills improved the code, but it did not change how the run judged its own work.

The missing piece came from Anthropic's engineering blog. A model that checks its own work has all of its reasoning in the context and none of the objections, so it approves what it just built. Instead of asking the generator to doubt itself, you put an adversarial reviewer next to it.

That is how the build loop works today, and it is not a waterfall where one side builds and the other reviews once at the end. A builder implements the approved plan and checkpoints its work into a draft pull request. The adversarial reviewer then starts as a fresh process with an empty context, sees only the plan, that pull request, the scratch org and the committed evidence, and treats all of it as claims. When it finds problems it answers red, its findings go into the next builder round, and the two keep arguing like this, round after round, over one question: is this good enough to take the pull request out of draft. Only a green verdict does that. The reviewer also has no write access, because an early version found a real problem, corrected it, and then approved its own correction.

In the first run with this loop in place the reviewer went red on seventeen unresolved analyzer findings and missing runtime evidence, the builder fixed them, and the second round went green. It was the first run that did not approve itself.

Model and provider agnostic: OpenCode

Our CTO has made AI sovereignty a priority for everything we build, so we could not stay dependent on Anthropic for both the model and the harness. An LLM gateway is not enough for that, because what we do here is not normal generative AI where you swap one LLM for another. The power came from Claude Code and the Agent SDK, and our skills only work because they are called natively and not treated as loose recommendations.

So we found OpenCode. It is very close to Claude Code, it reads our skill files without any change, it even has its own GitHub Actions, which we did not use in the end, and it reaches more than seventy providers through one provider/model string.

The harness ported one to one, and soon we ran OpenAI's GPT-5.5 on the tickets we knew well, the first model in this pipeline that did not come from Anthropic. It reached green in the same cost range as Opus, but only after a detour: it saved the hardest evidence, the video, for the last round, and its recorder stopped filming before the agent's reply appeared on screen. One fix in the recording skill later, the resumed run went green.

We also ran the clean comparison, Opus on both engines, same specification. On OpenCode it produced the cleanest code of all runs and the first correct Agentforce setup we had seen, with better security and real test coverage on a prompt template. But it dropped three shared actions, so its agent cannot resolve an account by name, and it delivered no video, because it filmed static record pages and its own frame check rejected them. Its weaknesses just sat in different places from the ones we knew.

Engine agnostic: the last and biggest step

The next step was the last and biggest one, and it hurt in more than one place.

The first was money. With OpenCode we could no longer use native Claude Code, and with it our Claude Code Max subscriptions and the usage that comes included. Every token was now on the bill, and for the first time we saw what our subscriptions had been giving us for free. So the engine had to become a choice too, because we very much want to run Claude Code on our subscription where we can. On top of that we had worked with LangChain's Deep Agents in customer projects and found them a good abstraction for exactly this kind of engine.

The skills are what makes this possible. Almost every decision the pipeline makes is written as prose in versioned files rather than in code, so a new engine mostly means installing the same files somewhere else.

When the rewrite was done we sent our benchmark tickets through it again and compared, three models auditing all eight runs independently and me reading the pull requests and their evidence by hand after them. Work that was meant to be plain plumbing came back better than what it replaced.

While making the pipeline engine agnostic we finally had to face a topic we have not talked about yet, the one that gave this article its title.

Missing isolation: the pipeline was cheating

Many of our experiments looked much better in the first run than they really were, because the AI tricked us. Since the early days the pipeline had been prompted to look around for good code to learn from, and the learnings of every experiment went into the CLAUDE.md that every run loads. So at several points of this evolution the pipeline did not find things out, it took them, either from open pull requests that earlier runs had produced, or directly from the CLAUDE.md with all our learnings in it. We stopped it more than once, and we also forgot to stop it more than once.

We only found this out by reading pull requests next to each other, by hand, without asking an AI to judge the quality of AI work. The log of one run shows it well. A builder was stuck on exactly the problem an earlier experiment had already solved, it grepped its own repository for the names of the classes it was working on, and the search returned our experiment log with the solved fix spelled out in it, including which earlier run had gone green with it.

Under these conditions we could not run honest experiments. So together with the engine work we finally built real separation. The experiment log lives in a file that no run ever loads, experiment tickets contain only what a real customer ticket contains, and the pipeline itself now runs locally, outside of GitHub, with swappable engines. That also means it can be plugged into other systems like GitLab, or run in VS Code against local files.

In the Grimm tale the hedgehog wins every race against the hare because he never runs at all. His wife waits at the other end of the furrow and looks exactly like him. An agent that can reach its own earlier results is such a hedgehog, and a race against it says nothing about how fast it really is.

This is going to sound like a cheap pun, but the cheating was not only in the fable, it was also there after the Fable model. Several runs, whether with Opus or with Sonnet 5, looked far too good, and only days later we found out that they in effect had access to the pull request and to our CLAUDE.md with the whole history in it. At that point it was clear that the isolation of our experiments was the real problem, and that without it none of our comparisons meant anything.

Conclusion

Most of what looked like progress did not come from where we thought it came from, and that holds for the model, for the harness and for our own bookkeeping.

1. A weaker model needs a stronger harness. Fable delivered a full statement of work through a pipeline that was still missing most of what this article describes, and everything we built afterwards replaces what that one model had been carrying for us.

2. Copying beats instructing. A reference repository the run is allowed to copy conventions from did more than any skill we wrote about how to build on Salesforce, and more than the self-maintained knowledge base, the retrospectives and the web search together.

3. Self-review does not work. A run that grades its own work agrees with itself. The review only counts when a second process starts with an empty context and treats the plan and the committed evidence as claims rather than as results.

4. Vendor and engine independence has a price. Choosing the model and the engine per project is worth real money to us and to our customers, and we paid for it with a second harness to keep in sync, provider-specific failure modes we had never seen before, roughly forty percent more tokens for the same ticket, and the subscription usage that used to be included.

5. Some of our evolution steps were not steps. A change went in, the next run came out better, and we credited the change. In several cases the run had found the notes from the run before it and read the answer out of them, so the improvement was ours and not the pipeline's. Runs that can reach each other cannot be compared, and without comparison every change is a guess.

This kind of work is never finished. The version we run today will look naive in three months, like every version before it did.

What is different now is that we can run it in isolation. A run starts from a configuration we version, its results and KPIs are written back as JSON next to it, and none of it depends on GitHub anymore, so the same run works on a laptop, in a customer's GitLab or as a scheduled job. Comparing two generations is a diff, not an afternoon of reading pull requests side by side.

The next generations will come out of real customer work, in Aquiva AI Pods, where the architect owns the plan and what ships. If you want a pipeline like this running in your own organization, or a delivery partner who already has one, talk to us.