In September 2026, nine researchers from UMass Amherst, Emory and UNC Charlotte published the most careful harness study I’ve seen: 176 matched settings, four models, two benchmarks, one component changed at a time (Fan et al., 2026).
One row in their results table deserves more attention than it gets. Nemotron-3 550B on SWE-Bench Verified with a 32K context window and no context management resolved 6.40% of issues. The same model, with the same tools and the same budget, but with a running summary of older turns, resolved 58.40% (Fan et al., full results).
That 52-point swing came from a context-management layer that takes a few dozen lines in most harnesses. It’s larger than the 43-point gap between the best scores of the strongest and weakest models in the whole study.
This issue is about that layer and the rest of the code around a model: the harness. It’s written to be used, not skimmed. You get:
A working definition of an agent harness.
The 10 components every production harness ends up with, each with the failure it prevents, the code, and the measured effect from controlled studies.
A tested, single-file reference harness in plain Python (no frameworks) that you can read in one sitting and point at any OpenAI-compatible model.
An evidence scorecard that ranks the components.
A method for measuring your own harness changes properly, including where the harness doesn’t matter.
What an agent harness actually is
A language model is a stateless function. It takes a list of messages and returns one message. It can’t run a command, read a file, remember the last call, or notice that it’s been trying the same broken fix for twenty minutes.
Everything that turns that function into something that finishes a task is the harness:
Agent = model + harness. The harness is the loop that calls the model, the tools it can use, what it sees after each action, what gets kept or dropped from its context, the rules that stop it from looping or quitting early, and the limits on what it’s allowed to do.
The terms overlap in practice. “Scaffold” usually means the same thing. “Framework” (LangGraph, the OpenAI Agents SDK, the Claude Agent SDK) is a library you build a harness with. “Context engineering” is one part of the harness: deciding what the model sees on each call.
LangChain’s team put the practical version clearly. Keeping gpt-5.2-codex fixed and changing only the system prompt, tools and middleware, they moved their coding agent from 52.8% to 66.5% on Terminal Bench 2.0, from outside the top 30 to the top 5 on the leaderboard (LangChain).
The research community has started treating this as a reporting problem. Harness-Bench ran 5,194 trajectories across 106 tasks and concluded that agent capability should be reported at the model-plus-harness level, not the model level (Harness-Bench). A companion paper’s title says it directly: “Stop Comparing LLM Agents Without Disclosing the Harness” (arXiv 2605.23950).
So when someone tells you model A beats model B at agentic work, the first question is: in which harness?
How to read this issue
Each component below follows the same pattern:
The failure: what goes wrong without it.
The code: an excerpt from the reference harness. The full file is about 370 lines, standard library only, and has a test suite that exercises every component with a scripted model, so it runs without an API key. The full file is printed in the appendix at the end of this post.
The evidence: the best controlled number I could find, with the setup, so you can judge whether it applies to you.
A note on evidence quality. I’ve prioritized studies that change one thing at a time with the model held fixed. Vendor blog posts are useful for mechanisms, but many don’t report controlled numbers. When a number comes from a small sample, I say so.
Component 0: The loop
The failure: there isn’t one yet. This is the minimum: call the model, run the tools it asked for, append the results, repeat.
for step in range(budget.max_steps):
msg = model(ctx.view(record), tool_schemas) # the model sees a view, not the raw record
record.append(msg)
for call in msg.get("tool_calls") or []:
result = getattr(tools, call["function"]["name"])(**json.loads(call["function"]["arguments"]))
record.append({"role": "tool", "tool_call_id": call["id"], "content": result})Every serious harness is this loop plus a set of hooks around it. One design decision in these six lines matters more than it looks: the model receives ctx.view(record), not record. The harness keeps the complete history for tracing and debugging, and builds a smaller working view for each model call. Component 4 depends on that separation.
Two more rules that belong in the loop from day one:
Tool errors go back to the model, not up the stack. A
FileNotFoundErroris information the agent can act on. A crashed harness is not.The loop, not the model, decides when the run ends. More on that in Component 6.
Component 1: The tool surface
The failure: the agent has the wrong set of actions. Too few, and it can’t do the task. Too many or too specialized, and it wastes steps choosing between them, misuses them, or fills its context with tool definitions.
The SWE-agent team at Princeton called this the agent-computer interface (ACI) and showed it matters as much for models as user interfaces do for people. With GPT-4 Turbo on SWE-bench Lite, their custom interface resolved 18.0% of issues, versus 11.0% for the same model with only a shell (SWE-agent paper).
But the direction of the effect depends on the model. That’s the most useful finding from Fan et al. At a 128K budget, they compared a full predefined tool set with a bash-only interface (Fan et al.):
The model ranking flips. With full tools, Mistral beats Nemotron-3 550B. With bash only, Nemotron wins by 24 points. Bash-only also cut Nemotron-3 550B’s mean cost per task from $2.33 to $1.11. The authors’ conclusion: predefined tools help models with weaker bash skills, while bash-capable models do well with bash alone at much lower cost, especially on command-line tasks.
Vercel saw the same pattern in production. Their text-to-SQL agent had 17 specialized tools. They replaced them with bash access to a file system containing the schema and SQL files. On their five benchmark queries with Claude Opus 4.5, success went from 80% to 100%, average time from 274.8s to 77.4s, and tokens used fell by about 37% (Vercel). Five queries is a small sample, but the direction matches the controlled study.
What the reference harness does: six tools, each designed around a known failure.
fn("read_file", "Show 100 numbered lines of a file starting at `start`.", ...)
fn("grep", "Regex search across .py files. Returns at most 50 hits.", ...)
fn("edit_file", "Replace one exact occurrence of `old` with `new`. Python edits are syntax-checked.", ...)
fn("run", "Run a shell command in the repo root. Output is truncated.", ...)
fn("update_todo","Replace the todo list: [{task, done}].", ...)
fn("finish", "Declare the task complete, with a one-paragraph summary.", ...)Rule of thumb: start with bash plus a file editor. Add a specialized tool only when your traces show the model repeatedly failing at something a tool would make trivial. If you have more than about 10 tools, or tool definitions above about 10K tokens, load them on demand instead (see Component 9).
Component 2: Observation shaping
The failure: one tool result floods the context. A test run prints 4,000 lines, a cat returns a 3,000-line file, or a search returns 900 matches. The useful signal is buried, and later turns are paid for with tokens that carry no information.
This is where the SWE-agent ablations are most instructive, because they isolate small design choices (SWE-agent paper):
Two lessons. There’s a sweet spot: too little context and the agent can’t orient, too much and it drowns. And a badly shaped tool can be worse than no tool. Iterative search scored below having no search at all, because agents dutifully inspected every result until they ran out of budget.
The code: every tool result passes through a head-and-tail truncation, and the file viewer tells the model where it is.
def head_tail(text: str, cap: int) -> str:
"""Keep the beginning and end, drop the middle with a visible marker."""
if len(text) <= cap:
return text
keep = max(cap // 2 - 40, 0)
omitted = len(text) - 2 * keep
return f"{text[:keep]}\n...[{omitted} chars omitted]...\n{text[-keep:]}"
def read_file(self, path: str, start: int = 1) -> str:
...
return (f"[{path}: lines {start}-{end} of {len(lines)}; "
f"{start - 1} above, {len(lines) - end} below]\n{body}")Head and tail matters because command output puts the important parts at the edges: what ran at the top, the error and exit code at the bottom. The explicit omission marker tells the model something was removed, so it can ask for more instead of assuming it saw everything.
Component 3: Guarded edits
The failure: the agent writes a syntax error into a file, then spends the next five turns debugging a problem it created, often while believing the original bug is still the issue.
SWE-agent’s editor runs a linter on every edit. If the edit introduces a syntax error, it’s rejected, the file stays unchanged, and the model sees the error. That guardrail alone was worth 3 points: 18.0% with linting versus 15.0% without. Removing the custom editor entirely dropped the score to 10.3% (SWE-agent paper).
The code:
def edit_file(self, path: str, old: str, new: str) -> str:
src = self._path(path).read_text()
if src.count(old) != 1:
return f"Edit rejected: `old` must match exactly once, found {src.count(old)} matches."
candidate = src.replace(old, new, 1)
if path.endswith(".py"):
# compile the candidate in a temp file; on failure, return the error and keep the file unchanged
...
return f"Edit rejected, file unchanged. Syntax error:\n{...}"
self._path(path).write_text(candidate)
return "Edit applied.\n" + self.read_file(path, start=max(1, line - 3))[:1500]Three details worth copying:
Require a unique match. String replacement that silently edits the wrong occurrence is one of the most common silent failures in coding agents.
Reject rather than warn. A warning leaves a broken file on disk. A rejection keeps the workspace in a known-good state.
Echo the edited region back. The model confirms what changed without spending a separate read call.
For other languages, swap py_compile for the fastest checker you have: tsc --noEmit on the single file, gofmt -e, a JSON or YAML parser.
Component 4: The context manager
The failure: the conversation outgrows the context window, and the run dies halfway through the task. Or it doesn’t die, but the model’s attention degrades as the context fills with stale tool output, a pattern Chroma documented as context rot.
This is the component with the largest measured effect, and the evidence is unusually good.
Fan et al. compared five strategies across four context budgets (Fan et al.):
T0: no context management; the run stops when the window is full.
T1, elision: replace old tool-result bodies with short stubs.
T2, elision + recall: same, but store the originals and give the model a
recall_eventtool to fetch them back.T3, summarization: fold older messages into a running natural-language summary.
T4, staged: elide at 60% of the window, summarize at 85%.
Selected results on SWE-Bench Verified (% resolved):
Four findings from the paper that should change how you build:
Most of the benefit comes from preventing overflow. Without management, a tight window kills the run. With any management, it survives.
The benefit shrinks as the window grows. At 32K, context management is the difference between a working agent and a broken one. At 128K, it’s worth a few points for strong models, and mainly becomes a cost lever.
Staged elision, then summarization, gave the best overall efficiency. Cheap mechanical trimming handles most turns. The expensive LLM summary only runs when trimming isn’t enough.
Making elided content recoverable didn’t help. The
recall_eventtool added machinery that models rarely used, with no accuracy gain.
A second study found the same effect with a purely mechanical approach. On 169 SWE-bench Verified tasks with a 20,480-token window, Lewis compared the full transcript with a “working view” in which older tool results are shortened on a “half-life” schedule (as a result’s age doubles, its character cap halves), combined with a stall detector and command safeguards. The mean fraction of target tests fixed rose from 28% to 49%, and complete solutions from 43 to 72. With a 262K window, outcomes were close, and the treatment used 7.2% fewer prompt tokens per turn (Lewis, 2026).
Vendor numbers point the same way. Anthropic reported that context editing alone improved agentic search performance by 29% over baseline, 39% combined with a memory tool, and cut token use by 84% in a 100-turn web search evaluation (Anthropic). Even the 2024 SWE-agent paper found that collapsing observations older than the last five beat keeping the full history, 18.0% versus 15.0% (SWE-agent paper).
The code: a two-stage manager in the style of T4, using the half-life schedule for the elision stage.
@dataclass
class ContextManager:
window_chars: int = 60_000
soft: float = 0.6 # start eliding here
hard: float = 0.85 # summarize the middle here
keep_recent: int = 4 # newest tool results always shown in full
base_cap: int = 2_000
summarize: Callable[[list[dict]], str] | None = None
def view(self, record: list[dict]) -> list[dict]:
msgs = [dict(m) for m in record] # the full record is never modified
if self.size(msgs) < self.soft * self.window_chars:
return msgs
# Stage 1: half-life elision of old tool results
tool_idx = [i for i, m in enumerate(msgs) if m["role"] == "tool"]
for age, i in enumerate(reversed(tool_idx)):
if age < self.keep_recent:
continue
tier = (age - self.keep_recent + 1).bit_length() - 1 # 0,1,1,2,2,2,2,...
cap = max(self.base_cap >> tier, 120)
msgs[i]["content"] = head_tail(msgs[i]["content"], cap)
# Stage 2: summarize the middle only if elision wasn't enough
if self.summarize and self.size(msgs) > self.hard * self.window_chars:
...
return msgsIn the test suite, 12 tool results of about 4,000 characters each produce this working view, oldest to newest: 198, 448, 448, 448, 448, 948, 948, 1,948, then the four newest in full. Total context falls from 48,301 to 22,063 characters, with no model call and the full record preserved for the trace.
What not to shorten: the task statement, the model’s own messages, and the plan. Lewis’s treatment shortened only tool-result bodies and left everything else untouched.
Component 5: The stall detector
The failure: the doom loop. The agent runs the same failing command, gets the same error, makes a tiny variation, and repeats. LangChain saw agents make small variations on the same broken approach “10+ times in some traces” (LangChain).
Models are poor at noticing this about themselves, because each turn looks reasonable in isolation. The harness can see the pattern for free, because it’s in the execution record.
Lewis’s detector used fixed rules with no model call: a repeated failing command, the same error several times, or repeated reads without an edit. When it fired, the harness delivered a fixed message telling the model the same attempt had produced the same failure and to try something different. Across 168 sessions at the 20K budget, it fired in 31 of them (Lewis, 2026). The paper reports its effect as part of the combined treatment above, not in isolation.
LangChain’s LoopDetectionMiddleware works the same way: it counts edits per file and, after N edits to the same file, adds a message suggesting the agent reconsider its approach.
The code:
@dataclass
class StallDetector:
repeat_limit: int = 3 # same action + same result N times in a row
edit_limit: int = 5 # N successful edits to one file
read_limit: int = 8 # N reads/searches with no edit
def observe(self, name: str, args: dict, result: str) -> str | None:
key = name + dumps(args)
self.history.append((key, result[:500]))
...
last = self.history[-self.repeat_limit:]
if len(last) == self.repeat_limit and len(set(last)) == 1:
return ("The same action produced the same result "
f"{self.repeat_limit} times. Do something different.")Two implementation details that matter:
Deliver interventions after all tool results for that turn. Most chat APIs require every tool call to be followed by its result before any other message. Inserting a nudge in between produces an API error.
Log every firing. The trace of when the detector fired, and what the agent did next, is the best data you’ll have for tuning thresholds.
Component 6: The verification gate
The failure: the agent writes a plausible fix, reads its own code, decides it looks right, and declares victory without running anything. Or it runs the tests, sees a failure it doesn’t like, and finishes anyway.
LangChain’s fix centered on a build-and-verify loop (plan, build, verify against the task, fix) and a PreCompletionChecklistMiddleware that intercepts the agent when it tries to exit and makes it run a verification pass against the task specification (LangChain). They didn’t publish a per-component number, but it was one of a handful of changes in the 13.7-point gain.
Anthropic took the idea further for long-running work: separate the agent that builds from the agent that evaluates, because models grade their own work too generously. Their evaluator was tuned to be skeptical and tested the live application directly with Playwright (Anthropic).
The code: the gate refuses finish unless a passing test run happened after the last edit.
@dataclass
class VerificationGate:
test_cmd: str
last_edit_step: int = -1
last_pass_step: int = -1
def allow_finish(self) -> str | None:
if self.last_pass_step > self.last_edit_step:
return None
return (f"Not yet. Before finishing: run `{self.test_cmd}`, read the full output, "
"and check the result against the original task (not against your code). "
"Cover edge cases, not just the happy path.")This caught a real problem while I was testing it. The sandbox didn’t have pytest installed, so the test command failed with “No module named pytest”. The scripted agent tried to finish anyway, and the gate refused. That’s exactly the case it exists for: an agent that doesn’t read test output closely enough to notice the tests never ran.
Where this generalizes: “tests pass” is a stand-in for whatever proves the task is done in your domain. For a SQL agent, the query executes and returns the expected columns. For a research agent, every claim has a source URL that resolves. For a document agent, the output validates against a schema. The pattern is the same: a deterministic check owned by the harness, not the model.
Component 7: Plan and state
The failure: on longer tasks, the agent loses track of what it has done and what’s left. After a context reset or summarization, it starts over, redoes finished work, or declares the whole project done after completing one piece.
Planning is the within-run version. Fan et al. found planning mostly changes cost for strong models and accuracy for weak ones. At 128K, removing the planning component dropped Nemotron-3 30B from 25.2% to 13.6%, but left the three stronger models roughly unchanged (Nemotron-3 550B went from 65.8% to 67.8%) while raising Nemotron-3 550B’s mean cost from $2.33 to $3.31 (Fan et al.). Their summary: planning shifts from an accuracy scaffold for weaker models to a cost saver for stronger ones.
Persistent state is the across-session version. Anthropic’s harness for multi-session coding uses an initializer agent that writes a feature list, a progress file and an init script, and a coding agent that works on one feature at a time and commits after each (Anthropic). One detail stands out: the feature list is JSON with a "passes": false field per feature, because the model was less likely to inappropriately rewrite a JSON file than a Markdown one. Anthropic reported this as a design lesson, without a benchmark number.
OpenAI’s version, from a team that built about a million lines of code with agents over five months, is to keep the agent instructions file (AGENTS.md) to about 100 lines and treat it as a map pointing to deeper documentation, not a manual (OpenAI).
The code: an explicit todo tool. The harness stores it, and it survives summarization because it’s state, not conversation.
def update_todo(self, items: list[dict]) -> str:
self.todo = items
return "Todo:\n" + "\n".join(f"[{'x' if i.get('done') else ' '}] {i['task']}" for i in items)For multi-session work, write that list to a progress.json file in the workspace, and make reading it the first action of every new session.
Component 8: Permissions
The failure: the agent runs a destructive command, pushes to a shared branch, pipes a remote script into a shell, or reads files outside its workspace.
There’s no benchmark number here, and there shouldn’t need to be. This component exists for the one run in a thousand that would otherwise be an incident.
The code: a deny list checked before execution, and a path guard on every file tool.
DENY = [r"\brm\s+-rf\s+/", r"\bcurl\b.*\|\s*(ba)?sh", r"\bgit\s+push\b", r"\bsudo\b", r"\bmkfs\b"]
def check_command(cmd: str) -> str | None:
for pat in DENY:
if re.search(pat, cmd):
return f"Blocked by policy: command matches {pat!r}. Choose a safer action."
return None
def _path(self, rel: str) -> Path:
p = (self.root / rel).resolve()
if self.root not in p.parents and p != self.root:
raise PermissionError(f"{rel} is outside the workspace")
return pBe clear about what this is: a guardrail for honest mistakes, not a security boundary. A regex deny list can be bypassed. In production, the real boundary is the sandbox: a container or VM with no credentials it doesn’t need, no network access it doesn’t need, and a disposable file system. Do both. The deny list gives the model a clear, immediate message it can learn from within the run, and the sandbox contains whatever gets past it.
Component 9: The cache-stable prompt and tool loading
The failure: agent runs are slow and expensive because every call reprocesses the entire context from scratch.
Agents are read-heavy. Manus reported an average input-to-output token ratio of about 100:1, which makes the KV cache the most important cost and latency lever. With Claude Sonnet, cached input tokens cost $0.30 per million versus $3 uncached, a 10x difference (Manus).
Caching only works if the prompt prefix is byte-identical between calls. The Manus rules:
No timestamps or per-request values in the system prompt. One changing token at the top invalidates everything after it.
Append-only context. Don’t edit earlier messages.
Deterministic serialization. Many JSON libraries don’t guarantee key order, which silently breaks the cache.
Don’t add or remove tools mid-run. Tool definitions sit near the front of the context. Mask tools instead of removing them.
There’s a real tension with Component 4: eliding old tool results edits earlier messages and breaks the cache from that point forward. Staged approaches manage this by only trimming once the window passes a threshold, so most calls in a run keep a stable prefix. Measure both effects on your own workload.
The code:
SYSTEM_PROMPT = """You are a software agent working inside a sandboxed repository. ...""" # static
def dumps(obj) -> str:
"""Stable serialization keeps the prompt prefix byte-identical (KV-cache hits)."""
return json.dumps(obj, sort_keys=True, ensure_ascii=False)Tool loading is the related problem at scale. Anthropic observed tool definitions consuming 134K tokens before optimization. Their Tool Search Tool loads definitions on demand, which cut tool-definition tokens by 85% and raised accuracy on MCP evaluations from 49% to 74% for Opus 4 and from 79.5% to 88.1% for Opus 4.5 (Anthropic). Their guidance is to use it when you have more than 10 tools or definitions above 10K tokens. In the same post, programmatic tool calling, where the model writes code that calls tools and only the final result enters context, cut average token use by 37% on complex research tasks.
Component 10: Budgets and traces
The failure: the run never ends, ends at a random point, or ends and you have no idea why it failed.
Budgets: hard limits on steps and wall-clock time, plus a warning before the limit. LangChain found agents are bad at estimating time, and that injecting a time warning nudged them to wrap up and verify (LangChain). The same post has the clearest example of a budget interacting with model settings: running gpt-5.2-codex at maximum reasoning effort throughout scored 53.9%, because runs hit timeouts, while “high” scored 63.6%.
if not warned and elapsed > budget.warn_at * budget.max_seconds:
warned = True
record.append({"role": "user", "content": "Time check: 75% of the budget is used. "
"Wrap up and move to verification."})Traces: an append-only JSONL log of every model call, tool call, detector firing and finish attempt. This is what you improve the harness with. LangChain’s whole improvement loop was trace analysis: run the benchmark, have agents analyze the failed traces in parallel, find the pattern, change the harness, repeat (LangChain).
class Tracer:
def log(self, kind: str, **data) -> None:
with self.path.open("a") as f:
f.write(dumps({"t": round(time.time(), 3), "kind": kind, **data}) + "\n")The evidence scorecard
Here are all 10 components ranked by the strength of the controlled evidence behind them, not by how interesting they are.
Two gaps are worth naming. Nobody has published an isolated ablation of a verification gate or a stall detector on a public benchmark. Both are widely used, and both appear in harnesses with large measured gains, but their individual contribution is unmeasured. That’s an open question, and one I plan to test in a future Build Log.
When the harness doesn’t matter much
It would be easy to leave you with the impression that the harness always dominates. It doesn’t, and knowing when it doesn’t saves real engineering time.
Arjmandi ran a contamination-controlled private suite of 80 tasks comparing each vendor’s native harness with a neutral one (deepagents on LangGraph), same model (Arjmandi, 2026):
Neither difference was statistically significant. Two mature, well-engineered harnesses running a frontier model with a large context window land in the same place overall.
But the average hid something. For Opus 4.8, the native harness was 9.0 points worse on repository tasks and 23.7 points better on contest-style tasks, and that interaction was significant. The overall number was nearly zero because the two effects cancelled out.
Put the studies together and a clear pattern emerges. The harness matters most when:
The context window is tight relative to the task: the 32K rows in Fan et al. and the 20K window in Lewis.
The model is weaker or less practiced at tool use: Nemotron-3 30B in every ablation.
The task is long and produces a lot of tool output.
Your harness is immature. Going from a naive harness to a good one is worth a lot. Going from one good harness to another is often worth little on average, but can still matter a lot for particular task types.
If you’re running a frontier model with a 200K window inside a mature SDK, your next hour is probably better spent on verification and traces than on context management.
How to measure a harness change properly
Most harness “improvements” I see are validated by running a handful of tasks and eyeballing the result. With agent runs, which are noisy, that’s how you ship regressions.
The method used by Fan et al. and Arjmandi is simple enough to copy:
Fix the task set. Same tasks, same order, for both configurations. 50–100 tasks is a reasonable minimum. Include your real failure cases, not just benchmark tasks.
Change one component at a time. Anthropic gives the same advice from the practitioner side: remove components one at a time and review the effect, rather than simplifying everything at once (Anthropic).
Compare paired outcomes, not two percentages. What matters is how many tasks flipped from fail to pass and from pass to fail. A paired test (exact McNemar) uses exactly that.
Report cost next to accuracy. Several components in this issue barely move accuracy but halve cost.
from math import comb
def mcnemar_exact(a: dict[str, bool], b: dict[str, bool]) -> dict:
tasks = sorted(a.keys() & b.keys())
only_a = sum(a[t] and not b[t] for t in tasks) # A solved, B failed
only_b = sum(b[t] and not a[t] for t in tasks) # B solved, A failed
n, k = only_a + only_b, min(only_a, only_b)
p = min(1.0, 2 * sum(comb(n, i) for i in range(k + 1)) / 2 ** n) if n else 1.0
return {"tasks": len(tasks), "only_a": only_a, "only_b": only_b, "p_value": round(p, 4)}A useful sanity check: if 8 tasks flip in one direction and none in the other, p ≈ 0.008, a real effect. If 5 flip one way and 3 the other, p ≈ 0.73, which is noise regardless of how the headline percentages look.
The reference harness
The complete file is about 370 lines of standard-library Python, printed in full in the appendix below.
harness.py: all 10 components, plus an adapter for any OpenAI-compatible endpoint (OpenAI, a local vLLM or Ollama server, or a gateway in front of other providers).test_harness.py: five tests that drive the harness with a scripted model, so they run without an API key. They check that a syntax-breaking edit is rejected, that a destructive command is blocked, thatfinishis refused until tests pass, that the stall detector fires on repetition, that path escapes are blocked, and that the context manager shrinks old results while keeping the full record.compare.py: the paired comparison above.
Running it against a real model:
pip install openai
export OPENAI_API_KEY=... # or OPENAI_BASE_URL for a local or compatible server
MODEL=your-model-name python harness.py ./path/to/repo "Fix the failing test in test_stats.py"It’s deliberately small. It isn’t a replacement for a production framework. It’s a way to see every moving part in one place, so when you use a framework you know what each setting is doing and which ones to measure.
Your 30-minute harness audit
Run this against whatever agent you have in production or in development. Each “no” is a candidate for your next change.
Context: Do you know what happens when a run exceeds the context window? Does it fail, truncate silently, or manage the context deliberately?
Observations: Is there a cap on every tool result? Does truncation keep the head and tail, and say that something was removed?
Tools: Could you delete half your tools and replace them with bash plus files? Have you tested that?
Edits: Are edits validated before they’re written? Does a broken edit leave the workspace unchanged?
Loops: Does anything detect the same failing action repeated three times?
Finish: Can the agent declare success without a deterministic check passing after its last change?
State: If the context were wiped mid-task, could a fresh session pick up from a file in the workspace?
Permissions: Is there a hard boundary (sandbox, scoped credentials) and not just a prompt instruction?
Cache: Is your system prompt byte-identical across calls? Is JSON serialized deterministically?
Traces: Can you pull up every failed run from last week and see exactly where it went wrong?
Measurement: When you last changed the harness, did you compare paired results on a fixed task set?
Appendix: the full reference harness
Copy this into harness.py. It needs Python 3.10+ and, only for real model calls, pip install openai.
"""
harness.py - a minimal, readable agent harness (The Applied Stack, Issue 2)
One file, standard library only, plus any OpenAI-compatible chat client.
Each numbered section maps to one harness component in the article.
agent = model + harness
harness = loop + tools + observation shaping + guarded edits
+ context manager + stall detector + verification gate
+ plan/state + permissions + budgets + tracing
"""
from __future__ import annotations
import json
import os
import py_compile
import re
import subprocess
import tempfile
import time
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any, Callable
# ---------------------------------------------------------------------------
# 9. Cache-stable prompt: static text, no timestamps, deterministic JSON.
# Anything that changes per request goes at the END of the context, never here.
# ---------------------------------------------------------------------------
SYSTEM_PROMPT = """You are a software agent working inside a sandboxed repository.
Work in four phases: PLAN (read the task, find the relevant code, write a short
todo list), BUILD (make the smallest change that solves the task), VERIFY (run
the tests and read the full output; compare against the task, not your code),
FIX (if anything fails, re-read the task and change your approach).
Use the tools. Call finish only after a verification run has passed."""
def dumps(obj: Any) -> str:
"""Stable serialization keeps the prompt prefix byte-identical (KV-cache hits)."""
return json.dumps(obj, sort_keys=True, ensure_ascii=False)
# ---------------------------------------------------------------------------
# 10. Budgets and tracing
# ---------------------------------------------------------------------------
@dataclass
class Budget:
max_steps: int = 40
max_seconds: float = 900
warn_at: float = 0.75 # inject a time warning at 75% of the budget
class Tracer:
"""Append-only JSONL trace: the raw material for improving the harness."""
def __init__(self, path: str | Path):
self.path = Path(path)
self.path.parent.mkdir(parents=True, exist_ok=True)
def log(self, kind: str, **data: Any) -> None:
with self.path.open("a") as f:
f.write(dumps({"t": round(time.time(), 3), "kind": kind, **data}) + "\n")
# ---------------------------------------------------------------------------
# 2. Observation shaping: never let one tool result flood the context
# ---------------------------------------------------------------------------
def head_tail(text: str, cap: int) -> str:
"""Keep the beginning and end, drop the middle with a visible marker."""
if len(text) <= cap:
return text
keep = max(cap // 2 - 40, 0)
omitted = len(text) - 2 * keep
return f"{text[:keep]}\n...[{omitted} chars omitted]...\n{text[-keep:]}"
# ---------------------------------------------------------------------------
# 8. Permissions: deny before execution, not after
# ---------------------------------------------------------------------------
DENY = [r"\brm\s+-rf\s+/", r"\bcurl\b.*\|\s*(ba)?sh", r"\bgit\s+push\b", r"\bsudo\b", r"\bmkfs\b"]
def check_command(cmd: str) -> str | None:
for pat in DENY:
if re.search(pat, cmd):
return f"Blocked by policy: command matches {pat!r}. Choose a safer action."
return None
# ---------------------------------------------------------------------------
# 1. Tool surface (the agent-computer interface)
# ---------------------------------------------------------------------------
class Tools:
VIEW_LINES = 100 # SWE-agent: 100-line window beat 30 lines and full file
MAX_HITS = 50 # SWE-agent: cap search results, ask for a narrower query
MAX_OUTPUT = 6000 # chars of shell output admitted to context
def __init__(self, root: str | Path):
self.root = Path(root).resolve()
self.todo: list[dict] = []
def _path(self, rel: str) -> Path:
p = (self.root / rel).resolve()
if self.root not in p.parents and p != self.root:
raise PermissionError(f"{rel} is outside the workspace")
return p
def read_file(self, path: str, start: int = 1) -> str:
lines = self._path(path).read_text().splitlines()
start = max(1, start)
end = min(len(lines), start + self.VIEW_LINES - 1)
body = "\n".join(f"{i:>5} {lines[i - 1]}" for i in range(start, end + 1))
return (f"[{path}: lines {start}-{end} of {len(lines)}; "
f"{start - 1} above, {len(lines) - end} below]\n{body}")
def grep(self, pattern: str, path: str = ".") -> str:
hits = []
for f in sorted(self._path(path).rglob("*.py")):
for n, line in enumerate(f.read_text(errors="ignore").splitlines(), 1):
if re.search(pattern, line):
hits.append(f"{f.relative_to(self.root)}:{n}: {line.strip()}")
if len(hits) > self.MAX_HITS:
return f"{len(hits)} matches. Too many to show; use a more specific pattern."
return "\n".join(hits) or "No matches."
# 3. Guarded edits: lint before accepting, reject broken edits
def edit_file(self, path: str, old: str, new: str) -> str:
p = self._path(path)
src = p.read_text()
count = src.count(old)
if count != 1:
return f"Edit rejected: `old` must match exactly once, found {count} matches."
candidate = src.replace(old, new, 1)
if p.suffix == ".py":
with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as tmp:
tmp.write(candidate)
try:
py_compile.compile(tmp.name, doraise=True)
except py_compile.PyCompileError as e:
return f"Edit rejected, file unchanged. Syntax error:\n{head_tail(str(e), 800)}"
finally:
os.unlink(tmp.name)
p.write_text(candidate)
line = candidate[: candidate.index(new)].count("\n") + 1 if new else 1
return "Edit applied.\n" + self.read_file(path, start=max(1, line - 3))[:1500]
def run(self, command: str, timeout: int = 120) -> str:
if (msg := check_command(command)):
return msg
try:
r = subprocess.run(command, shell=True, cwd=self.root, capture_output=True,
text=True, timeout=timeout)
except subprocess.TimeoutExpired:
return f"Timed out after {timeout}s."
out = f"exit={r.returncode}\n{r.stdout}{r.stderr}"
return head_tail(out, self.MAX_OUTPUT)
# 7. Plan and state: an explicit todo list the agent maintains
def update_todo(self, items: list[dict]) -> str:
self.todo = items
return "Todo:\n" + "\n".join(f"[{'x' if i.get('done') else ' '}] {i['task']}" for i in items)
def schemas(self) -> list[dict]:
def fn(name, desc, props, req):
return {"type": "function", "function": {"name": name, "description": desc,
"parameters": {"type": "object", "properties": props, "required": req}}}
s = {"type": "string"}
return [
fn("read_file", "Show 100 numbered lines of a file starting at `start`.",
{"path": s, "start": {"type": "integer"}}, ["path"]),
fn("grep", "Regex search across .py files. Returns at most 50 hits.",
{"pattern": s, "path": s}, ["pattern"]),
fn("edit_file", "Replace one exact occurrence of `old` with `new`. Python edits are syntax-checked.",
{"path": s, "old": s, "new": s}, ["path", "old", "new"]),
fn("run", "Run a shell command in the repo root. Output is truncated.",
{"command": s}, ["command"]),
fn("update_todo", "Replace the todo list: [{task, done}].",
{"items": {"type": "array", "items": {"type": "object"}}}, ["items"]),
fn("finish", "Declare the task complete, with a one-paragraph summary.",
{"summary": s}, ["summary"]),
]
# ---------------------------------------------------------------------------
# 4. Context manager: keep the full record, show the model a working view
# ---------------------------------------------------------------------------
@dataclass
class ContextManager:
window_chars: int = 60_000 # rough proxy for the model's context budget
soft: float = 0.6 # start eliding old tool results here
hard: float = 0.85 # summarize the middle here
keep_recent: int = 4 # newest tool results always shown in full
base_cap: int = 2_000 # cap for results just outside the recent set
summarize: Callable[[list[dict]], str] | None = None # optional LLM summarizer
def size(self, msgs: list[dict]) -> int:
return sum(len(m.get("content") or "") + len(dumps(m.get("tool_calls", ""))) for m in msgs)
def view(self, record: list[dict]) -> list[dict]:
msgs = [dict(m) for m in record]
if self.size(msgs) < self.soft * self.window_chars:
return msgs
# Stage 1, elision: half-life caps on old tool results (cap halves as age doubles)
tool_idx = [i for i, m in enumerate(msgs) if m["role"] == "tool"]
for age, i in enumerate(reversed(tool_idx)):
if age < self.keep_recent:
continue
tier = (age - self.keep_recent + 1).bit_length() - 1 # 0,1,1,2,2,2,2,...
cap = max(self.base_cap >> tier, 120)
msgs[i]["content"] = head_tail(msgs[i]["content"], cap)
# Stage 2, summarization: only if elision was not enough
if self.summarize and self.size(msgs) > self.hard * self.window_chars:
head, middle, tail = msgs[:2], msgs[2:-8], msgs[-8:]
while middle and middle[0]["role"] == "tool": # never orphan a tool result
middle.pop(0)
if middle:
note = {"role": "user", "content": "[Summary of earlier work]\n" + self.summarize(middle)}
msgs = head + [note] + tail
return msgs
# ---------------------------------------------------------------------------
# 5. Stall detector: deterministic rules, zero model tokens
# ---------------------------------------------------------------------------
@dataclass
class StallDetector:
repeat_limit: int = 3
edit_limit: int = 5
read_limit: int = 8
history: list[tuple[str, str]] = field(default_factory=list)
edits: dict[str, int] = field(default_factory=dict)
reads_since_edit: int = 0
def observe(self, name: str, args: dict, result: str) -> str | None:
key = name + dumps(args)
self.history.append((key, result[:500]))
if name == "edit_file" and result.startswith("Edit applied"):
self.edits[args["path"]] = self.edits.get(args["path"], 0) + 1
self.reads_since_edit = 0
if self.edits[args["path"]] >= self.edit_limit:
self.edits[args["path"]] = 0
return (f"You have edited {args['path']} {self.edit_limit} times. "
"Step back: re-read the task and consider a different approach.")
if name in ("read_file", "grep"):
self.reads_since_edit += 1
if self.reads_since_edit >= self.read_limit:
self.reads_since_edit = 0
return "You have read a lot without changing anything. Decide on a change or update the todo list."
last = self.history[-self.repeat_limit:]
if len(last) == self.repeat_limit and len(set(last)) == 1:
return ("The same action produced the same result "
f"{self.repeat_limit} times. Do something different.")
return None
# ---------------------------------------------------------------------------
# 6. Verification gate: the agent cannot finish on "looks right to me"
# ---------------------------------------------------------------------------
@dataclass
class VerificationGate:
test_cmd: str
last_edit_step: int = -1
last_pass_step: int = -1
reminded: bool = False
def note(self, step: int, name: str, args: dict, result: str) -> None:
if name == "edit_file" and result.startswith("Edit applied"):
self.last_edit_step = step
if name == "run" and self.test_cmd in args.get("command", "") and result.startswith("exit=0"):
self.last_pass_step = step
def allow_finish(self) -> str | None:
if self.last_pass_step > self.last_edit_step:
return None
self.reminded = True
return (f"Not yet. Before finishing: run `{self.test_cmd}`, read the full output, "
"and check the result against the original task (not against your code). "
"Cover edge cases, not just the happy path.")
# ---------------------------------------------------------------------------
# 0. The loop
# ---------------------------------------------------------------------------
class Agent:
def __init__(self, model: Callable[[list[dict], list[dict]], dict], root: str,
test_cmd: str, trace_path: str = "traces/run.jsonl",
budget: Budget | None = None, context: ContextManager | None = None):
self.model = model # fn(messages, tools) -> assistant message dict
self.tools = Tools(root)
self.budget = budget or Budget()
self.ctx = context or ContextManager()
self.stall = StallDetector()
self.gate = VerificationGate(test_cmd)
self.trace = Tracer(trace_path)
def run(self, task: str) -> dict:
record = [{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": task}]
t0, warned = time.time(), False
schemas = self.tools.schemas()
for step in range(self.budget.max_steps):
elapsed = time.time() - t0
if elapsed > self.budget.max_seconds:
return self._end("timeout", step)
if not warned and elapsed > self.budget.warn_at * self.budget.max_seconds:
warned = True
record.append({"role": "user", "content": "Time check: 75% of the budget is used. "
"Wrap up and move to verification."})
msg = self.model(self.ctx.view(record), schemas)
record.append(msg)
self.trace.log("model", step=step, content=msg.get("content"),
tool_calls=msg.get("tool_calls"))
calls = msg.get("tool_calls") or []
if not calls:
record.append({"role": "user", "content": "Use a tool, or call finish."})
continue
nudges = []
for call in calls:
name = call["function"]["name"]
args = json.loads(call["function"]["arguments"] or "{}")
if name == "finish":
blocked = self.gate.allow_finish()
result = blocked or "Finished."
record.append({"role": "tool", "tool_call_id": call["id"], "content": result})
self.trace.log("finish", step=step, allowed=blocked is None)
if blocked is None:
return self._end("done", step, summary=args.get("summary", ""))
continue
try:
result = getattr(self.tools, name)(**args)
except Exception as e: # tool errors go back to the model, not up the stack
result = f"Tool error: {type(e).__name__}: {e}"
record.append({"role": "tool", "tool_call_id": call["id"], "content": result})
self.trace.log("tool", step=step, name=name, args=args, result=result[:2000])
self.gate.note(step, name, args, result)
if (n := self.stall.observe(name, args, result)):
nudges.append(n)
self.trace.log("stall", step=step, message=n)
for n in nudges: # interventions go after all tool results (valid message order)
record.append({"role": "user", "content": "[harness] " + n})
return self._end("step_limit", self.budget.max_steps)
def _end(self, status: str, steps: int, summary: str = "") -> dict:
out = {"status": status, "steps": steps, "summary": summary}
self.trace.log("end", **out)
return out
# ---------------------------------------------------------------------------
# Plugging in a real model (any OpenAI-compatible endpoint)
# ---------------------------------------------------------------------------
def openai_model(model_name: str, **client_kwargs) -> Callable[[list[dict], list[dict]], dict]:
from openai import OpenAI # pip install openai
client = OpenAI(**client_kwargs)
def call(messages: list[dict], tools: list[dict]) -> dict:
r = client.chat.completions.create(model=model_name, messages=messages, tools=tools)
m = r.choices[0].message
return {"role": "assistant", "content": m.content,
"tool_calls": [tc.model_dump() for tc in (m.tool_calls or [])] or None}
return call
if __name__ == "__main__":
import sys
repo, task = sys.argv[1], sys.argv[2]
agent = Agent(openai_model(os.environ.get("MODEL", "gpt-5-mini")), repo, test_cmd="pytest -q")
print(agent.run(task))The Filter
Five resources worth your time this week:
An Empirical Study of Harness Design for Coding Agents: the best controlled harness study published so far. The results tables alone are worth the read.
Improving Deep Agents with harness engineering: the clearest practitioner account of improving a harness from traces, with the actual middleware.
Effective harnesses for long-running agents: how to structure state so an agent can work across many sessions.
Context Engineering for AI Agents: Lessons from Building Manus: the KV-cache lessons most teams learn the expensive way.
SWE-agent: Agent-Computer Interfaces: from 2024, and still the best set of small, isolated interface ablations.
One question
Which harness component has caused you the most pain in production: context overflow, doom loops, agents finishing without verifying, or something I didn’t list? Reply and tell me. The most common answer decides what I benchmark in the next Build Log.
New here? Subscribe free and get the RAG and AI Agents Cheat Sheet: a decision tree for choosing between RAG and long context, production defaults, real benchmark numbers and copy-paste prompt templates.
If this was useful, forward it to one person who is building agents.