Vol. 01 · No. 08
— Field Notes —
August 2026
A Field Note on Agentic Software Engineering

The AI coding setup that caught its own mistakes

I used OpenCode, a terminal interface for coding agents, to split planning, building and criticism into separate roles. The work stayed visible in separate panes, and every handoff landed as a file. On Polster’s Docker support the setup added 6,416 lines, and it kept finding reasons not to ship them yet. This is the hands-on follow-up to Field Note 01.

On a building site, nobody asks the architect to pour the concrete, inspect the foundation and sign the safety certificate as well. Software teams know this when people do the work, but they seem to forget it as soon as an AI model is involved. We give one agent a big prompt, ask it to plan, build, test, review and declare success, and then we’re surprised when the review sounds like a defence of its own work. I wanted a setup with different incentives, where one agent plans, another builds, a third tries to reject the result, and the orchestrator decides whether to continue or stop.

I think the useful unit in AI coding is the handoff, because that’s the part you can actually review.

— lesson from the Polster build
01 · The build

A feature big enough to test the method

The test case was Docker-first support for Polster, a Python CLI that generates Dagster data projects. It was a lot more than a wrapper script, since it touched runtime selection, the generated Compose files, POSIX and PowerShell launchers, project upgrades, CI, image security, documentation and the native Python fallback. The implementation plan ended up at 584 lines and 13 tasks.

Polster’s Docker support

Real work on a real branch

The first feature commit was big enough that a friendly “looks good” review would have meant nothing. The setup had to check the architecture, command behaviour, migration safety, release gates and the path a user would actually run.

Plan

584 lines · 13 tasks

Explicit files, dependency policy, failure modes, verification commands, and a twelve-part quality rubric.

Feature commit

53 files changed

6,416 insertions and 189 deletions across launchers, templates, upgrades, tests, documentation, and CI.

Two follow-up commits

Smoke + Compose hardening

First 10 files, +242/−37, after the user path failed. Then 5 files, +35/−7, for compatibility with Docker Compose 2 and newer.

Outcome

Merged, public release held

PR #3 got merged after the compatibility fix. The Docker static gate still failed, no v0.2 release was published, and the multi-OS release matrix was still unfinished.

Figure 1 The bigger the change, the more the loop is worth. On a one-file fix it’s mostly ceremony, but across 53 files and two operating systems, splitting up the jobs is just basic risk control.
02 · The loop

Plan → Build → Review → Fix, with one final owner

I kept the workflow narrow on purpose. The planner turns the goal into a build contract and the builder implements that contract. The reviewer then grades the diff against it, and has to either write a numbered feedback file or print a standalone GRADE: PASS. Failed reviews go back to the builder, and only after a pass does the orchestrator run its own checks and decide what can honestly be claimed.

The operating loop

Four roles, four different incentives

The loop works because “done” becomes a list of artifacts and gates that another role can reject.

01 · Planner

Defines the contract

Produces plan.md, with the architecture, sequence, exact tests, edge cases, dependency limits and pass criteria.

02 · Builder

Produces the diff

Implements task by task. It can explain why it deviated, but it doesn’t get to grade its own work.

03 · Reviewer

Tries to stop the release

Audits correctness, security, maintainability, test coverage, and scope. Every issue needs a file reference and an exact repair.

04 · Orchestrator

Owns continue, hold, or stop

Routes the work, spots stalls, enforces scope, runs its own verification and reports the remaining risk, without quietly turning it into “done.”

Figure 2 Splitting the roles only helps if the authority is split too. If the builder can declare its own pass, the extra agent names are just decoration. The reviewer has to be able to fail the work, and the orchestrator has to be able to keep it failed.
03 · The control plane

The files matter more than the chat

Long agent conversations are a pretty bad place to keep project state. They get truncated, they’re hard to diff, and they’re full of statements that look like decisions but aren’t. So this setup moves the important state onto disk. The plan’s actual pass rule was blunt, “Every category below is mandatory. One failed category means the result is not GRADE: PASS.” A fresh agent can come in with no memory of the conversation, read that contract and the latest feedback file, look at the repository and carry on.

Artifact What it controls Why plain chat is weaker
plan.md Scope, architecture, dependencies, tests, edge cases, and the definition of pass. Goals in a chat drift. A file can be diffed, reviewed, attached to a fresh session and checked against the code.
feedback-round-N.md Numbered defects with file references and repair instructions. “A few issues remain” isn’t something you can act on, and it can’t prove that every issue was dealt with.
Git diff The actual implementation boundary. Agent summaries often leave out files, generated artifacts, permission changes and dependency drift.
Test output Evidence that a specific command ran in a specific environment. “Tests pass” is just a sentence. The exit code, the count and the command output are the actual evidence.
Smoke report The user path: install, initialise, rebuild, run, stop, and clean. Unit tests prove the contracts, but they don’t prove that the product feels coherent when you start from an empty directory.
Figure 3 The working tree is the shared memory. Files survive context resets, model switches, crashes and handoffs between people. The feedback files stayed local and untracked, while the plan and the product code were committed. That way the orchestration history stayed available, without shipping review scratch files as part of the product.
04 · Why it works

Different contexts create useful friction

A model that reviews its own implementation has already committed to a story, where the design is sensible, the changes are intentional and the remaining gaps are probably the environment’s fault. A separate reviewer starts from a different question, which is what would make this unsafe to merge. I think that change in attitude is the main advantage. What the extra compute really buys is disagreement.

Four structural advantages

Why the loop beats one large prompt

The advantage comes from ordinary engineering discipline, expressed through context boundaries and explicit permissions. There’s nothing mystical about models collaborating here.

Incentive separation

The critic is rewarded for rejection

The reviewer didn’t spend hours producing the diff, so it has no sunk-cost reason to protect the implementation.

Context reduction

Each role sees a smaller job

The planner thinks about architecture, the builder about implementation and the reviewer about what could fail. That means less leaking between roles and less story to defend.

Bounded repair

Fix rounds do not redesign the feature

The feedback file defines what needs repairing. “Fix every numbered item, add no dependency, expand no scope” is a much stronger brief than “make it better.”

Independent evidence

Claims can be checked outside the model

Git, pytest, Ruff, Docker, HTTP health checks and filesystem sentinels are deterministic witnesses, so the orchestrator doesn’t have to trust a completion summary.

Figure 4 The setup works because it’s adversarial on purpose. It assumes builders miss things, reviewers can be vague, conversations lose state and automated tests leave out user behaviour, and each layer covers a different one of those failures.
05 · Visibility

Why I kept every agent in a visible TUI

Files keep the state between sessions, but the TUI solves a different problem, which is seeing the run while it’s happening. The agents ran in separate OpenCode terminal interfaces inside one herdr workspace, a terminal multiplexer that kept every pane side by side. I could see which role was active, whether tests were moving, whether an approval dialog had blocked the process and whether an agent had wandered off into an unrelated failure. Being able to see it doesn’t make the agents any better, but it does make it a lot cheaper to step in.

Same workspace

No hidden worker fleet

Planner, builder, reviewer and orchestrator stay side by side, so you can look at the whole system without hunting across tabs or reading a generated summary.

Stall detection

Silence becomes observable

A frozen subagent, a repeated viewport, a model limit or an unchanged file timestamp becomes something you can recover from, instead of a timeout nobody sees.

Controlled recovery

Interrupt, restart, resume

A short resume brief can name the branch, the completed tasks, partial work, the next action, constraints and the required completion marker.

Human authority

Autonomy stays bounded

Safe edits and tests inside the repository can go ahead. Destructive operations on the host, publishing and scope changes stay outside what the agent is allowed to do on its own.

Figure 5 Observability is part of orchestration. A headless pipeline is great once the process is stable, but while you’re still learning the method, visible panes show you the failure patterns that should later become rules in the skill.
06 · The evidence

The setup earned its keep by saying no

The first review didn’t come back with polite suggestions. It came back with 17 findings. Among them were native environments in the wrong place, incomplete upgrade semantics, a stale-plan overwrite risk, remote repositories pushed before generation finished, broken release checksums, missing tag-version validation and real-daemon coverage gaps. Two more rounds brought the list down. Then a smoke test run the way a real user would found five bugs outside the planned happy path. The counts below come from the local feedback files I kept, which were left out of the product branch on purpose.

Gate Result What it exposed
Review round 1 FAIL · 17 findings Runtime isolation, upgrade safety, push ordering, architecture and port handling, probe safety, CI gaps, checksum paths, tag consistency, and incomplete release evidence.
Review round 2 FAIL · 6 findings A missing initial commit before remote push, over-managed project metadata, a release path that could bypass daemon E2E, and three focused test gaps.
Review round 3 FAIL · 2 blockers Formatting still failed on changed files, and the current host still lacked complete cross-shell and real-daemon release evidence.
User smoke FAIL · 5 findings Piped init aborted, rebuild forced a registry pull, failing child commands returned zero, empty materialisation crashed inside Dagster, and pytest flags required a -- separator. The hardening diff added allow_extra_args and tests for both ways of invoking it.
Post-hardening verification PASS locally The focused suite passed, with the PowerShell launcher tests skipped on this Linux host. Ruff, formatting, shell syntax and the real piped-init/rebuild/materialise/test path all passed.
End-to-end confidence run PASS locally Init, doctor, rebuild, list-assets, validate, bronze-to-gold materialisation, UI health, stop, persistence, and clean all passed with a local image.
Merge PR #3 merged A final five-file compatibility commit fixed Docker Compose 2-and-newer detection before the feature branch was squash-merged into main.
Release path HELD The Docker static CI job failed, release jobs were skipped, no v0.2 release was published and the Linux/macOS/Windows evidence was still incomplete. So a merged feature didn’t get called a public release.
Figure 6 A good agent setup gives you useful refusals. The 6,416 inserted lines were the first output, but I think the more valuable output was the list of concrete reasons why those lines weren’t safe to call finished yet.
07 · Honest limits

Where this setup is the wrong tool

The loop adds model calls, wall-clock time, artifacts and coordination. It also brings the risk of reviewing for the sake of it, where an agent produces twenty technically plausible objections without making the product any better. I think the setup is worth it when a hidden defect would cost more than another review round, and it’s wasteful when the change is small, reversible and easy to test directly.

Situation Use one agent Use the orchestration loop
One-file fix with an obvious failing test Default Mostly ceremony
Cross-platform feature touching runtime, CI, and migrations High blind-spot risk Default
Early experiment you may discard tomorrow Default Too slow
Release path with security or persistence claims Self-review is too weak Default
No independent test oracle exists Fast, uncertain Still uncertain, since more agents don’t create ground truth
Plan is vague or internally wrong Likely scope drift Confidently wrong at scale
Figure 7 More agents aren’t automatically more reliable. The setup needs a sharp plan, deterministic checks, a reviewer willing to reject and an orchestrator that can tell an environment gap from a code defect, without hiding either of them.
Closing observation

Make mistakes cheap to catch before you make agents more autonomous

The Polster case supports a smaller claim than “AI agents can replace a software team”, and I think a more defensible one. The controls a software team uses can make AI coding a lot safer. Planning, implementation, criticism and release authority shouldn’t collapse into one context just because the worker is a model. The same separation that helps human teams helps agent teams too.

The reusable part now lives in a skill, a small instruction package that writes the workflow down, so I don’t have to rely on remembering one good session. It defines the plan-build-review loop, the required artifacts, the TUI layout in one workspace, the failure markers, the recovery path and the final independent checks. The next build starts with those lessons already in place, instead of rediscovering them through another broken launcher.

Overall, I think orchestration should make agent work easy to inspect, easy to interrupt and hard to misrepresent. A system that writes code quickly is useful, but a system that can produce evidence, reject its own work and keep the reason for that rejection is what I’d actually call an engineering system.

— end —

The AI coding setup that caught its own mistakes · A Field Note Set in Shadows Into Light Two & Cause

← Back to all posts