The AI coding setup that caught its own mistakes
I used OpenCode, a terminal interface for coding agents, to split planning, building and criticism into separate roles. The work stayed visible in separate panes, and every handoff landed as a file. On Polster’s Docker support the setup added 6,416 lines, and it kept finding reasons not to ship them yet. This is the hands-on follow-up to Field Note 01.
On a building site, nobody asks the architect to pour the concrete, inspect the foundation and sign the safety certificate as well. Software teams know this when people do the work, but they seem to forget it as soon as an AI model is involved. We give one agent a big prompt, ask it to plan, build, test, review and declare success, and then we’re surprised when the review sounds like a defence of its own work. I wanted a setup with different incentives, where one agent plans, another builds, a third tries to reject the result, and the orchestrator decides whether to continue or stop.
I think the useful unit in AI coding is the handoff, because that’s the part you can actually review.
— lesson from the Polster buildA feature big enough to test the method
The test case was Docker-first support for Polster, a Python CLI that generates Dagster data projects. It was a lot more than a wrapper script, since it touched runtime selection, the generated Compose files, POSIX and PowerShell launchers, project upgrades, CI, image security, documentation and the native Python fallback. The implementation plan ended up at 584 lines and 13 tasks.
Real work on a real branch
The first feature commit was big enough that a friendly “looks good” review would have meant nothing. The setup had to check the architecture, command behaviour, migration safety, release gates and the path a user would actually run.
Plan
584 lines · 13 tasks
Explicit files, dependency policy, failure modes, verification commands, and a twelve-part quality rubric.
Feature commit
53 files changed
6,416 insertions and 189 deletions across launchers, templates, upgrades, tests, documentation, and CI.
Two follow-up commits
Smoke + Compose hardening
First 10 files, +242/−37, after the user path failed. Then 5 files, +35/−7, for compatibility with Docker Compose 2 and newer.
Outcome
Merged, public release held
PR #3 got merged after the compatibility fix. The Docker static gate still failed, no v0.2 release was published, and the multi-OS release matrix was still unfinished.
Plan → Build → Review → Fix, with one final owner
I kept the workflow narrow on purpose. The planner turns the goal into a build contract and the builder implements that contract. The reviewer then grades the diff against it, and has to either write a numbered feedback file or print a standalone GRADE: PASS. Failed reviews go back to the builder, and only after a pass does the orchestrator run its own checks and decide what can honestly be claimed.
Four roles, four different incentives
The loop works because “done” becomes a list of artifacts and gates that another role can reject.
01 · Planner
Defines the contract
Produces plan.md, with the architecture, sequence, exact tests, edge cases, dependency limits and pass criteria.
02 · Builder
Produces the diff
Implements task by task. It can explain why it deviated, but it doesn’t get to grade its own work.
03 · Reviewer
Tries to stop the release
Audits correctness, security, maintainability, test coverage, and scope. Every issue needs a file reference and an exact repair.
04 · Orchestrator
Owns continue, hold, or stop
Routes the work, spots stalls, enforces scope, runs its own verification and reports the remaining risk, without quietly turning it into “done.”
The files matter more than the chat
Long agent conversations are a pretty bad place to keep project state. They get truncated, they’re hard to diff, and they’re full of statements that look like decisions but aren’t. So this setup moves the important state onto disk. The plan’s actual pass rule was blunt, “Every category below is mandatory. One failed category means the result is not GRADE: PASS.” A fresh agent can come in with no memory of the conversation, read that contract and the latest feedback file, look at the repository and carry on.
| Artifact | What it controls | Why plain chat is weaker |
|---|---|---|
plan.md |
Scope, architecture, dependencies, tests, edge cases, and the definition of pass. | Goals in a chat drift. A file can be diffed, reviewed, attached to a fresh session and checked against the code. |
feedback-round-N.md |
Numbered defects with file references and repair instructions. | “A few issues remain” isn’t something you can act on, and it can’t prove that every issue was dealt with. |
| Git diff | The actual implementation boundary. | Agent summaries often leave out files, generated artifacts, permission changes and dependency drift. |
| Test output | Evidence that a specific command ran in a specific environment. | “Tests pass” is just a sentence. The exit code, the count and the command output are the actual evidence. |
| Smoke report | The user path: install, initialise, rebuild, run, stop, and clean. | Unit tests prove the contracts, but they don’t prove that the product feels coherent when you start from an empty directory. |
Different contexts create useful friction
A model that reviews its own implementation has already committed to a story, where the design is sensible, the changes are intentional and the remaining gaps are probably the environment’s fault. A separate reviewer starts from a different question, which is what would make this unsafe to merge. I think that change in attitude is the main advantage. What the extra compute really buys is disagreement.
Why the loop beats one large prompt
The advantage comes from ordinary engineering discipline, expressed through context boundaries and explicit permissions. There’s nothing mystical about models collaborating here.
Incentive separation
The critic is rewarded for rejection
The reviewer didn’t spend hours producing the diff, so it has no sunk-cost reason to protect the implementation.
Context reduction
Each role sees a smaller job
The planner thinks about architecture, the builder about implementation and the reviewer about what could fail. That means less leaking between roles and less story to defend.
Bounded repair
Fix rounds do not redesign the feature
The feedback file defines what needs repairing. “Fix every numbered item, add no dependency, expand no scope” is a much stronger brief than “make it better.”
Independent evidence
Claims can be checked outside the model
Git, pytest, Ruff, Docker, HTTP health checks and filesystem sentinels are deterministic witnesses, so the orchestrator doesn’t have to trust a completion summary.
Why I kept every agent in a visible TUI
Files keep the state between sessions, but the TUI solves a different problem, which is seeing the run while it’s happening. The agents ran in separate OpenCode terminal interfaces inside one herdr workspace, a terminal multiplexer that kept every pane side by side. I could see which role was active, whether tests were moving, whether an approval dialog had blocked the process and whether an agent had wandered off into an unrelated failure. Being able to see it doesn’t make the agents any better, but it does make it a lot cheaper to step in.
Same workspace
No hidden worker fleet
Planner, builder, reviewer and orchestrator stay side by side, so you can look at the whole system without hunting across tabs or reading a generated summary.
Stall detection
Silence becomes observable
A frozen subagent, a repeated viewport, a model limit or an unchanged file timestamp becomes something you can recover from, instead of a timeout nobody sees.
Controlled recovery
Interrupt, restart, resume
A short resume brief can name the branch, the completed tasks, partial work, the next action, constraints and the required completion marker.
Human authority
Autonomy stays bounded
Safe edits and tests inside the repository can go ahead. Destructive operations on the host, publishing and scope changes stay outside what the agent is allowed to do on its own.
The setup earned its keep by saying no
The first review didn’t come back with polite suggestions. It came back with 17 findings. Among them were native environments in the wrong place, incomplete upgrade semantics, a stale-plan overwrite risk, remote repositories pushed before generation finished, broken release checksums, missing tag-version validation and real-daemon coverage gaps. Two more rounds brought the list down. Then a smoke test run the way a real user would found five bugs outside the planned happy path. The counts below come from the local feedback files I kept, which were left out of the product branch on purpose.
| Gate | Result | What it exposed |
|---|---|---|
| Review round 1 | FAIL · 17 findings | Runtime isolation, upgrade safety, push ordering, architecture and port handling, probe safety, CI gaps, checksum paths, tag consistency, and incomplete release evidence. |
| Review round 2 | FAIL · 6 findings | A missing initial commit before remote push, over-managed project metadata, a release path that could bypass daemon E2E, and three focused test gaps. |
| Review round 3 | FAIL · 2 blockers | Formatting still failed on changed files, and the current host still lacked complete cross-shell and real-daemon release evidence. |
| User smoke | FAIL · 5 findings | Piped init aborted, rebuild forced a registry pull, failing child commands returned zero, empty materialisation crashed inside Dagster, and pytest flags required a -- separator. The hardening diff added allow_extra_args and tests for both ways of invoking it. |
| Post-hardening verification | PASS locally | The focused suite passed, with the PowerShell launcher tests skipped on this Linux host. Ruff, formatting, shell syntax and the real piped-init/rebuild/materialise/test path all passed. |
| End-to-end confidence run | PASS locally | Init, doctor, rebuild, list-assets, validate, bronze-to-gold materialisation, UI health, stop, persistence, and clean all passed with a local image. |
| Merge | PR #3 merged | A final five-file compatibility commit fixed Docker Compose 2-and-newer detection before the feature branch was squash-merged into main. |
| Release path | HELD | The Docker static CI job failed, release jobs were skipped, no v0.2 release was published and the Linux/macOS/Windows evidence was still incomplete. So a merged feature didn’t get called a public release. |
Where this setup is the wrong tool
The loop adds model calls, wall-clock time, artifacts and coordination. It also brings the risk of reviewing for the sake of it, where an agent produces twenty technically plausible objections without making the product any better. I think the setup is worth it when a hidden defect would cost more than another review round, and it’s wasteful when the change is small, reversible and easy to test directly.
| Situation | Use one agent | Use the orchestration loop |
|---|---|---|
| One-file fix with an obvious failing test | Default | Mostly ceremony |
| Cross-platform feature touching runtime, CI, and migrations | High blind-spot risk | Default |
| Early experiment you may discard tomorrow | Default | Too slow |
| Release path with security or persistence claims | Self-review is too weak | Default |
| No independent test oracle exists | Fast, uncertain | Still uncertain, since more agents don’t create ground truth |
| Plan is vague or internally wrong | Likely scope drift | Confidently wrong at scale |
Make mistakes cheap to catch before you make agents more autonomous
The Polster case supports a smaller claim than “AI agents can replace a software team”, and I think a more defensible one. The controls a software team uses can make AI coding a lot safer. Planning, implementation, criticism and release authority shouldn’t collapse into one context just because the worker is a model. The same separation that helps human teams helps agent teams too.
The reusable part now lives in a skill, a small instruction package that writes the workflow down, so I don’t have to rely on remembering one good session. It defines the plan-build-review loop, the required artifacts, the TUI layout in one workspace, the failure markers, the recovery path and the final independent checks. The next build starts with those lessons already in place, instead of rediscovering them through another broken launcher.
Overall, I think orchestration should make agent work easy to inspect, easy to interrupt and hard to misrepresent. A system that writes code quickly is useful, but a system that can produce evidence, reject its own work and keep the reason for that rejection is what I’d actually call an engineering system.
— end —