Do not overcomplicate: Agentic Harness
Enough of an overview to understand but not enough to get stuck
Do not overcomplicate is an evergreen series that introduces concepts that 1) quickly take over the industry, 2) feel like something everyone should be doing, and 3) quickly spiral out of control in terms of complexity, when in reality, in majority of use cases, they are quite simple - or at least only as complicated as you need them to be.
I considered calling these series “wtf is…..” but if my goal is to get passed around among directors and VPs, it seems that I need a more acceptable name.
If you read nothing else because you’re headed out for the 4th of July weekend, take away this:
Harness is the difference between a demo and a functioning automation;
Similarly to evals, they are the first step to a functioning product, not the last;
Start simple: markdown, .csv, and human eyeballs.
Terminology around AI and agents evolves so quickly that it’s hard to figure out which concepts to treat seriously because they will eventually run the business, and which one to let slide (MoltBook). The rule of introducing new food to toddlers seems applicable here: if you’ve seen it on your plate 15-20 times, it might be safe to take a bite.
Building AI agents for enterprise is my day job. Because of this, I am fairly skeptical of most stories about autonomous agents running complex real-world operations end-to-end, like automating entire jobs, because, well, I work on agents that try to do exactly that.
Most of the risk is in preventing agents from making silly or disastrous mistakes. Research moves quickly, because reliability ties up a lot of ROI. Real-world case studies are few and far between, and the never-ending streams of arXiv papers make it seem like you need a dedicated ML whiz on staff to build out the sophisticated infrastructures of pipelines and evaluation algorithms.
But just like evals can be deconstructed to their minimal (and often most effective) functional baseline of a spreadsheet and eyeballs, so can the whole concept of a harness. More of a lego box of blocks to build from rather than a prescription to start from.
Doesn’t it make sense, then, that most public “LLM failures” are harness failures?
The Chevy Tahoe for $1: no output validator (or a bad one);
Replit’s agent deleting the production database: tool authorization too loose and broad;
Cursor’s “I quit” incident, where the agent refused to keep coding and told the user to learn to program: bad output validators;
Lawyers submitting hallucinated citations: issues with retrieval, prompt, and validators to check citations;
Air Canada’s chatbot’s refund - similarly, issues with retrieval - grounding claims in the actual policy document, prompt, and validators.
I’d love to deconstruct a real-life harness in the next post, so if you have one you can share (yours or public) let me know. Harnesses already are becoming the moats and tightly-protected secret sauce, and “harness engineering” a real job, even if hiding under a different title.
PS. If agent performance issues are a harness failure, is OpenAI’s social media manager job posting a harness opportunity? Because, honestly, I thought LLMs could do this already.
(I say this tongue-in-cheek, because I’m a believer that most tech and knowledge jobs - regardless how automatable they might seem on the surface - have a real moat that protects them from being replaced by automation, a deep “muck layer” that humans navigate intuitively and LLMs are not suited for. This “muck layer” is a concept I’ll return to often, which is why I wrote about it.
Sources:
https://www.mindstudio.ai/blog/agent-harness-engineering-wrapper-matters-more-than-model
https://arxiv.org/pdf/2603.28052 - “Changing the harness around a fixed LLM can produce a 6× performance gap on the same benchmark.”
Kamila’s latest post on the Substack really hit home regarding the 'agentic harness' sprawl. I’ve been actively trying to avoid this complexity tax on my personal aggregator project, Daily Relay, which runs completely local on a Node Alpine container.
Instead of letting an autonomous agent self-route or handle control flow, I built an explicitly bifurcated harness. Hard-coded JS handles task lists and calendars with 0ms latency, while a local 3B model is tightly boxed into a single-item parsing path. To make the 3B model production-grade for TTS, I built a lightweight evaluator script using regex score deductions for forbidden formatting alongside a local LLM judge. Ended up cutting generation latency by 56% just by tuning the constraints.