The AI coding setup that caught its own mistakes
I used OpenCode, a terminal coding-agent interface, to separate planning, building, and criticism; kept the work visible in separate panes; and made every handoff land as a file. On Polster’s Docker support, the setup added 6,416 lines—and repeatedly found reasons not to ship them yet. This is the operational sequel to Field Note 01.
No serious construction project asks the architect to pour the concrete, inspect the foundation, and sign the safety certificate. Software teams understand this when people do the work, then forget it when an AI model enters the room. We give one agent a large prompt, ask it to plan, implement, test, review, and declare success, then act surprised when the review sounds like a defence of the implementation. I wanted a setup with different incentives: one agent plans, another builds, a third tries to reject the result, and the orchestrator owns the decision to continue or stop.
The useful unit of AI coding is not the agent. It is the reviewable handoff.
— lesson from the Polster buildA feature large enough to expose the method
The test case was Docker-first support for Polster, a Python CLI that generates Dagster data projects. The feature was not a wrapper script. It touched runtime selection, generated Compose files, POSIX and PowerShell launchers, project upgrades, CI, image security, documentation, and the native Python fallback. The implementation plan ran to 584 lines and 13 tasks.
A real branch, not an agent demo
The first feature commit was large enough that a friendly “looks good” review would have been meaningless. The setup needed to inspect architecture, command behaviour, migration safety, release gates, and the path a user would actually run.
Plan
584 lines · 13 tasks
Explicit files, dependency policy, failure modes, verification commands, and a twelve-part quality rubric.
Feature commit
53 files changed
6,416 insertions and 189 deletions across launchers, templates, upgrades, tests, documentation, and CI.
Two follow-up commits
Smoke + Compose hardening
First: 10 files, +242/−37 after the user path failed. Then: 5 files, +35/−7 for Docker Compose 2-and-newer compatibility.
Outcome
Merged; public release held
PR #3 merged after the compatibility fix. The Docker static gate still failed, no v0.2 release was published, and the multi-OS release matrix remained unfinished.
Plan → Build → Review → Fix, with one final owner
The workflow is deliberately narrow. The planner turns the goal into a build contract. The builder implements that contract. The reviewer grades the diff against it and must either write a numbered feedback file or print a standalone GRADE: PASS. Failed reviews go back to the builder. Only after a pass does the orchestrator run independent checks and decide what can honestly be claimed.
Four roles, four different incentives
The loop works because “done” is not a feeling inside one context window. It is a sequence of artifacts and gates that another role can reject.
01 · Planner
Defines the contract
Produces plan.md: architecture, sequence, exact tests, edge cases, dependency limits, and pass criteria.
02 · Builder
Produces the diff
Implements task by task. It may explain deviations, but it does not get to grade its own work.
03 · Reviewer
Tries to stop the release
Audits correctness, security, maintainability, test coverage, and scope. Every issue needs a file reference and an exact repair.
04 · Orchestrator
Owns continue, hold, or stop
Routes the work, detects stalls, enforces scope, runs independent verification, and reports residual risk without laundering it into “done.”
The files matter more than the chat
Long agent conversations are poor project state. They are easy to truncate, difficult to diff, and full of statements that look like decisions but are not. This setup moves the important state onto disk. The plan’s actual pass rule was blunt: “Every category below is mandatory. One failed category means the result is not GRADE: PASS.” A fresh agent can enter with no conversational memory, read that contract and the latest feedback file, inspect the repository, and continue.
| Artifact | What it controls | Why plain chat is weaker |
|---|---|---|
plan.md |
Scope, architecture, dependencies, tests, edge cases, and the definition of pass. | Chat goals drift. A file can be diffed, reviewed, attached to a fresh session, and checked against the code. |
feedback-round-N.md |
Numbered defects with file references and repair instructions. | “A few issues remain” is not actionable and cannot prove that every issue was addressed. |
| Git diff | The actual implementation boundary. | Agent summaries routinely omit files, generated artifacts, permission changes, and dependency drift. |
| Test output | Evidence that a specific command ran in a specific environment. | “Tests pass” is a sentence. Exit code, count, and command output are evidence. |
| Smoke report | The user path: install, initialise, rebuild, run, stop, and clean. | Unit tests prove contracts. They do not prove that the product feels coherent when used from an empty directory. |
Different contexts create useful friction
A model reviewing its own implementation has already committed to a story: the design is sensible, the changes are intentional, and the remaining gaps are probably environmental. A separate reviewer starts from a different question: what would make this unsafe to merge? That change in posture is the main advantage. The extra compute is not buying more code. It is buying disagreement.
Why the loop beats one large prompt
The advantage comes from ordinary engineering discipline expressed through context boundaries and explicit permissions—not mystical collaboration between models.
Incentive separation
The critic is rewarded for rejection
The reviewer did not spend hours producing the diff. It has no sunk-cost reason to protect the implementation.
Context reduction
Each role sees a smaller job
The planner reasons about architecture. The builder reasons about implementation. The reviewer reasons about failure. Less role leakage, less narrative baggage.
Bounded repair
Fix rounds do not redesign the feature
The feedback file defines the repair surface. “Fix every numbered item, add no dependency, expand no scope” is a stronger brief than “make it better.”
Independent evidence
Claims can be checked outside the model
Git, pytest, Ruff, Docker, HTTP health checks, and filesystem sentinels are deterministic witnesses. The orchestrator does not need to trust a completion summary.
Why I kept every agent in a visible TUI
Files preserve state between sessions. The TUI solves a different problem: seeing the run while it is happening. The agents ran in separate OpenCode terminal interfaces inside one herdr workspace—a terminal multiplexer that kept every pane side by side. I could see which role was active, whether tests were moving, whether an approval dialog had blocked the process, and whether an agent had wandered into an unrelated failure. Visibility leaves capability unchanged and makes intervention cheaper.
Same workspace
No hidden worker fleet
Planner, builder, reviewer, and orchestrator stay side by side. The user can inspect the system without hunting across tabs or reading a synthetic summary.
Stall detection
Silence becomes observable
A frozen subagent, repeated viewport, model limit, or unchanged file timestamp is a recoverable state—not an invisible timeout.
Controlled recovery
Interrupt, restart, resume
A concise resume brief can name the branch, completed tasks, partial work, next action, constraints, and required completion marker.
Human authority
Autonomy stays bounded
Safe in-repository edits and tests can proceed. Destructive host operations, publishing, and scope changes remain outside the agent’s implied authority.
The setup earned its keep by saying no
The first review did not return tasteful suggestions. It returned 17 findings: native environments created in the wrong place, incomplete upgrade semantics, stale-plan overwrite risk, remote repositories pushed before generation finished, broken release checksums, missing tag-version validation, and gaps in real-daemon coverage. Two more rounds reduced the list. Then a real-user smoke test found five bugs outside the planned happy path. The counts below come from the retained local feedback files; those files were intentionally excluded from the product branch.
| Gate | Result | What it exposed |
|---|---|---|
| Review round 1 | FAIL · 17 findings | Runtime isolation, upgrade safety, push ordering, architecture and port handling, probe safety, CI gaps, checksum paths, tag consistency, and incomplete release evidence. |
| Review round 2 | FAIL · 6 findings | A missing initial commit before remote push, over-managed project metadata, a release path that could bypass daemon E2E, and three focused test gaps. |
| Review round 3 | FAIL · 2 blockers | Formatting still failed on changed files; the current host still lacked complete cross-shell and real-daemon release evidence. |
| User smoke | FAIL · 5 findings | Piped init aborted, rebuild forced a registry pull, failing child commands returned zero, empty materialisation crashed inside Dagster, and pytest flags required a -- separator. The hardening diff added allow_extra_args plus tests for both invocation forms. |
| Post-hardening verification | PASS locally | The focused suite passed; PowerShell launcher tests were skipped on this Linux host. Ruff, formatting, shell syntax, and the real piped-init/rebuild/materialise/test path passed. |
| End-to-end confidence run | PASS locally | Init, doctor, rebuild, list-assets, validate, bronze-to-gold materialisation, UI health, stop, persistence, and clean all passed with a local image. |
| Merge | PR #3 merged | A final five-file compatibility commit fixed Docker Compose 2-and-newer detection before the feature branch was squash-merged into main. |
| Release path | HELD | The Docker static CI job failed, release jobs were skipped, no v0.2 release was published, and the Linux/macOS/Windows evidence remained incomplete. A merged feature was not renamed a public release. |
Where this setup is the wrong tool
The loop adds model calls, wall-clock time, artifacts, and coordination. It also creates the risk of ceremonial reviewing: an agent can produce twenty technically plausible objections without improving the product. The setup is justified when the cost of a hidden defect is higher than the cost of another review round. It is wasteful when the change is small, reversible, and easy to test directly.
| Situation | Use one agent | Use the orchestration loop |
|---|---|---|
| One-file fix with an obvious failing test | Default | Mostly ceremony |
| Cross-platform feature touching runtime, CI, and migrations | High blind-spot risk | Default |
| Early experiment you may discard tomorrow | Default | Too slow |
| Release path with security or persistence claims | Self-review is too weak | Default |
| No independent test oracle exists | Fast, uncertain | Still uncertain — more agents do not create ground truth |
| Plan is vague or internally wrong | Likely scope drift | Confidently wrong at scale |
Do not optimise agent autonomy. Optimise the cost of catching a mistake.
The Polster case supports a narrower, more defensible claim than “AI agents can replace a software team”: software-team controls can make AI coding materially safer. Planning, implementation, criticism, and release authority should not collapse into one context just because the worker is a model. The same separation that helps human teams helps agentic ones.
The reusable part is now a skill—a small instruction package that codifies the workflow—rather than a memory of one successful session. It defines the plan-build-review loop, the required artifacts, the same-workspace TUI layout, the failure markers, the recovery path, and the final independent checks. The next build starts with those lessons already encoded instead of rediscovering them through another broken launcher.
Orchestration should make agent work inspectable, interruptible, and hard to misrepresent. A system that writes code quickly is useful. A system that can produce evidence, reject its own work, and preserve the reason for that rejection is an engineering system.
— end —