Why The Harness Matters More Than The Model | YC Paper Club

Why the harness matters (00:00)

The host opens by naming the dismissal head-on. Harnesses get called "just a wrapper," "just scaffolding," "just prompt/context engineering", and he shows a Reddit post from r/MachineLearning sneering that "this kind of prompt engineering" does not belong at a top-tier ML conference.

The rebuttal is empirical. On Terminal Bench 2, a "Meta-Harness" (Stanford IRIS, on Claude Opus 4.6) scores 76.4% and ranks 11th, while bare Claude Code with the same model sits at 58.0% and ranks 50th. That is an 18-point swing from the harness alone, on identical weights. He also points to the METR trend line: the length of task an agent can complete keeps climbing, and a lot of that gain has come from harnesses, not new weights.

He also flags the research angleARC-AGI measures how fast a system adapts to a brand-new problem. Claude Opus scored about 30% on the private hold-out that only Greg Kamradt and Francois Chollet can see, and "just a wrapper" pushes that to 95%, with Nvidia's AVO reaching 100%. His point: if scaffolding moves the number that much, scaffolding is research.

An auto-researcher built by accident (04:27)

The host forked Karpathy's auto-explorer in March, meaning only to build a UI to watch it work, and accidentally built a grading harness. You state a research purpose (his real example: prove an ensemble of sharded diffusion language models can out-learn an autoregressive baseline), give seed ideas, a validation metric, and baselines. A swarm of role-named agents then runs the project: a scoping agent surveys papers and GitHub, a "PI" agent (jokingly named after Christopher Re) hands work to a research agent, a council debates, and an author agent writes ablations and the paper.

He now gives eight ideas to eight 8xH100 nodes, leaves them alone, and gets back finished papers he says are genuinely good. His framing: the weights did not change, only the scaffolding that lets them run this way.

A five-minute history of harnesses (07:13)

The host walks the lineage deliberately out of chronological order, because the point is the shape of each innovation (in the action space, the active context, or the learning signal), not the dates.

• GPT-2 V0 (Feb 2019). The whole harness is a while not EOS loop with top-p sampling against an environment that scores accuracy. No tools, no skills, no memory. His GSM8K example: a system prompt ("you are a math teacher"), a context ("Suzie has $5, she spends $3"), and a graded answer.
• Few-shot learners (July 2020). Put examples back into the context so the model generalizes from them (in-context learning).
• Chain of thought. Spread the computation over more tokens instead of predicting the answer directly.
• Toolformer / WebGPT. Give the model tools it can call. A tool is a JSON object; instead of computing 5-2 in the weights, call a Python function.
• MemGPT. Give the model CRUD over its own context, not just append. Carve out a "memory" chunk it can create, read, update, and delete.
• Voyager. Chain tools into a task, learn it, and distill it back into the system prompt so the learning persists. This is where the modern idea of a skill (a named skills.md with a procedure) comes from. Demonstrated in Minecraft.
• ReAct, Self-Refine, Reflection. Multiple agents with different roles critique and improve each other's context. An internal evaluator catches a flipped answer and sends it back.
• Sub-agents and RLM. Spawn persistent sub-agents in REPLs, interact with them, and recursively expose an "RLM query" so an agent can spawn more agents to solve a broader problem, all under a main orchestrator.

That whole stack, static, is what he calls Harness V1: an agent spec with a system prompt, turn and tool-call budgets, and lists of allowed tools, skills, and sub-agents, put in a loop. It can be triggered on demand (a Slack message) or on a cron. He shows a real one built on top of Garry Tan's gstack, alongside OpenClaw, Codex, Hermes-Agent, Pi Agent Rust, and opencode.

Self-improving harnesses (13:56)

The exciting frontier, the host says, is letting the harness itself learn. Two levels: learn the system prompt, or learn the harness code.

• DSPy (demonstrate, search, predict). Take a small training set and learn the optimal system prompt. You cannot backpropagate through the process, so use genetic programming: find candidate prompts, merge them, evaluate against a fitness function on a trainset, and repeat. This gives you CRUD over the system prompt itself.
• Darwin-Godel machines. Go one level up: change not just the system prompt but the harness code that runs. Keep an archive of agents (each a harness plus a system prompt), sample from it, evaluate on a fitness function, and add the result back. A meta-harness lets an agent modify its own harness to become a different harness, and the loop produces better agents over time.
• Continual harness. Add richer memory classes and DAgger-style online learning, where you actually update the weight file at test time from a small number of examples the agent just learned. The host calls test-time training an important and under-explored direction.

Prime Agent: a self-improving RLM harness (Seth Karten, 18:35)

Seth Karten (Princeton, Prime Intellect) asks the audience to think from first principles. A raw LLM is just a sequential processor: tokens in, a next decision out, over fixed weights (L0) and a visible context (L1). The harness is the layer that adds persistent state, tools, and compute on top, everything "beyond the invocation": files and history, programs and tools, and other sessions.

His mental model for all the accumulated state is a cache hierarchy. L0 is the model weights (updated by fine-tuning). L1 is the active token-visible context (managed by compaction, the oldest and most minimal harness tool). L2 is the REPL and sub-agents (code, tools, retained values, cleaned up by "agentic garbage collection"). L3 is disk-backed state (history, artifacts, memories, skills, prompts, sub-agent specs, revised by "refinement").

The reframing he likes mosta harness moves an agent from a Turing machine (a sequential ticker tape) toward a von Neumann computer (read/write addressable external memory). And the design principle that follows is expressibility. Do not hard-code plan/act/critique loops the model can now do itself. Do give the model the primitives it cannot fake: compaction, a Python REPL, programmatic sub-agent creation, state access, and feedback mechanisms. Remove one primitive and you remove a capability.

Prime Agent's sub-agents are persistent sub-sessions: spin one up, it runs and reports back, then idles in RAM and can be messaged to resume with all its context intact, or offloaded and recalled later. Any two agents in a "nuclear family" (parent, child, siblings) can message each other directly, which he found essential for managing many parallel agents. He also argues for measuring long-horizon performance as a curve: find the practical plateau (where more test-time tokens yield only marginal gains) and compare models at a fixed expenditure rather than letting one run longer than another.

On results, Prime Agent scales to 95.5% on ARC-AGI with Opus 5 (versus a ~30% bare reference and human baseline 95.4%), well ahead of GPT-5.6 Sol (78.3%), Terra (25.7%), and GLM 5.2 (8.6%). He is candid that his first 99.9% run was cheating due to bad sandboxing, which he fixed. He also flags cost: a rival harness burned $5,000 without much performance, so cost-to-performance matters, and programmatic context work is what saves money.

Beyond ARC-AGI, he shows an emulator-bench result (reproducing a Game Boy Color, where REPL access lets the agent run out-of-loop experiments) and GPU-kernel (PMPP-Hard) results roughly at parity across models, plus an "autoresearch behavior differs across harnesses" chart where Prime Agent runs far more out-of-loop experiments per 100 training runs than Claude Code or opencode. The showcase long-horizon run: a 7-day Factorio game using 633 agents (7 max active) and 23M output tokens that keeps making tech-tree progress and does not get stuck.

OpenJarvis: personal AI on personal devices (Jon Saad-Falcon, 37:30)

Jon Saad-Falcon (Stanford, Hazy Research; advised by Christopher Re) argues personal AI is mostly cloud-bound today, which makes it costly (thousands of dollars a year), not private, rented rather than owned, and energy-hungry. His opportunity: local models are now only 6-12 months behind frontier (a Qwen3.5 27B roughly matches an August-2025 Claude Opus), and consumer accelerators run 1B-128B open-weight models with quantization. The capability exists; the surrounding system does not.

OpenJarvis defines the whole personal-AI stack as five swappable primitives: the user interfaces (CLI, browser, desktop, messaging channels), the agentic logic (composable reasoning), the intelligence (the local LLM: Qwen3.5, GPT-OSS, Gemma 3n), the inference engine and hardware (Ollama, llama.cpp, vLLM, SGLang on Apple Silicon, Nvidia, NPUs), and tools plus memory (over MCP) with learning primitives (prompt-based like GEPA/DSPy, weight-based like GRPO, SFT, LoRA).

The clever move is LLM-guided spec search: use a frontier cloud model once, at search time, to diagnose failures, propose edits across the four editable primitives, and gate to accept only non-regressing edits, so the optimized spec then runs entirely on-device at zero API cost. Cloud capability is spent once and amortized across every local query.

The resultsoptimized local stacks rival cloud on personal-AI, coding, and agentic workflows, at roughly 800x lower cost and lower latency. Opus 5 and GPT-5.6 Sol are the best optimizers, though Gemini, Kimi, and GLM also work, and every search-optimized student-teacher pair improved (the strongest, Qwen3.5-9B, hits 100% on PinchBench, 83% on LiveCodeBench, and 91% on LiveResearchBench).

QM: YC's agent harness for work (Josh France and Regan Bell, 45:58)

QM is YC's open-source work harness: one system giving every employee an OpenClaw-like assistant in Slack or a web UI, with a personal context, its own sandbox files and crons, and a multiplayer mode in Slack channels. People use it for email triage, legal and finance workflows, editing documents, pulling data from the internal database, and spinning up live internal web apps.

Josh traces the history that led herea January-2025 "general agent" (a system prompt with tools in a loop), then Claude Code and Codex in VMs wired to a Slack tag for one-off code changes, then OpenClaw for partners (each agent with its own computer), then a hard-to-manage fleet of 50-plus Hermes agents in VMs. QM is the response to that operational pain.

The design lens Regan uses is "unhobbling": there is an overhang of capability trapped inside the models, and pushing on what you offer them as capabilities is where the returns are.

The concrete moves

• Give them the ability to edit themselves. Pull the "brain" out of the per-agent sandbox and centralize everything in Postgres (sessions, memory, queue) behind a thin headless core (the agent loop, running Pi, OpenCode, or Claude Code). The sandbox stops being where the agent lives and becomes a resource it dips into. Accumulating traces lets you hill-climb an automated improvement loop, though with mixed results: an LLM-as-judge swarm develops "main character syndrome," each agent fixing only its own slice, so a human in the loop stays important.

• Wire them up to everything. Reads are permitted wherever the ACL allows, and writes go through human-reviewed bulk-upserts (the agent proposes a plan, a person eyeballs every row before it writes). They note the honest failure mode: people start rubber-stamping these, the same trust drift as early Claude Code tool approvals.

• Give them the compute they need. By default QM uses a per-channel sandbox with durable disk, but the agent can reach for a bigger machine for heavy dev work or a smaller one otherwise, and it picks its own model provider (popping to another model to dodge refusals on AI-research or security tasks). Pushing that decision into the agent, not the harness, is a contrast with the old fleet-of-Hermes model where each person got an isolated copy with no cross-org memory.

• Keep the harness thin. The core is Pi plus three basic tools that grant sandbox access: execute (shell on a persistent computer), read/write to object storage (with an ACL grant), and publish (a directory becomes an internal app, with rollback). Everything else (memory, history, cron, webhooks) is treated as removable scaffolding papering over rough edges, and the rest is plain HTTP.

They add a "grind tool" that puts budgets on goals, so an agent is not allowed to give up before a set wall-clock time or token spend, which yields much better research outputs and reports (the same technique OpenAI and Anthropic used to crack some open math problems). And QM is an "AGI-anticipating harness," so they lean thin on purpose.

The closing open problemagents do not implicitly understand social context or organizational structure. Privileged information leaks into contexts where it should not, so how much an agent can safely know is bounded by the quality of your permission system. YC has fine-grained permissioning built over years; most orgs do not.

Tools and resources

Claims to verify

• The 30% -> 95% ARC-AGI jump "from the harness alone" (host 02:52) and Prime Agent's 95.5% with Opus 5 (31:00): version-sensitive and depend on the exact ARC-AGI split; the private hold-out is not independently checkable.
• "Meta-Harness beats Claude Code by 18 points on Terminal Bench 2" (01:32): a single leaderboard snapshot on Claude Opus 4.6; leaderboards move.
• "Local models are only 6-12 months behind frontier" and "Qwen3.5 27B roughly matches August-2025 Opus" (38:26): date-sensitive and benchmark-dependent.
• "~800x lower cost" for optimized on-device stacks (43:53): headline figure; depends heavily on which cloud model and workload you compare against.
• Model names and versions throughout (Opus 5, GPT-5.6 Sol/Terra, GLM 5.2/5.3, Kimi K3, DeepSeek V4) are the speakers' near-future labels and were transcribed from slides; treat exact version numbers as approximate.

Glossary

• Harness: the software layer between an LLM and the world that adds persistent state, tools, compute, context management, sub-agents, and the loop that runs them.
• Model weights: the learned parameters of the network (Prime Agent's "L0"); updated only by training or fine-tuning.
• Context window: the tokens the model can see in a single invocation (the "active context," L1).
• Token: the unit of text a model reads and emits; costs and budgets are counted in tokens.
• EOS: end-of-sequence marker; the GPT-2 V0 harness loops "while not EOS."
• Top-p sampling: a decoding method that samples from the smallest set of tokens whose probabilities sum to p.
• GSM8K: a grade-school-math benchmark used as the running example for the V0 harness.
• Chain of thought: prompting the model to spread reasoning over intermediate tokens instead of predicting the answer directly.
• Few-shot / in-context learning (ICL): giving examples in the context so the model generalizes from them without weight updates.
• Toolformer / WebGPT: early work giving a model callable tools (a tool is a JSON object describing a function to call).
• MemGPT: gives the model create/read/update/delete control over its own context, not just append.
• CRUD: create, read, update, delete; here applied to context, system prompts, or harness code.
• Voyager: Minecraft agent that chained tools into learned skills distilled into the system prompt; origin of the modern skills.md idea.
• Skill: a named, reusable procedure (a skills.md file) the agent can look up and run.
• ReAct / Self-Refine / Reflection: patterns where an agent (or a critic agent) reflects on and improves its own output.
• Sub-agent: a spawned agent the parent can task and message; in Prime Agent, persistent sub-sessions that idle in RAM and resume.
• RLM (recursive language model): a design where an agent can recursively spawn more LLM agents to solve a broader class of problems.
• REPL / IPython shell: an interactive code environment (Prime Agent's "L2") where tools, variables, and sub-agents live in RAM.
• Compaction: summarizing the context history to keep working past the context length; the oldest, most minimal harness tool.
• Agentic garbage collection: cleaning up REPL variables and finished sub-agents so RAM does not overflow.
• Refinement: updating or deleting skills, memories, and prompts stored on disk (Prime Agent's "L3").
• DSPy (demonstrate, search, predict): a framework that learns an optimal system prompt via genetic programming against a fitness function.
• Genetic programming: an optimization that finds candidates, merges them, and evaluates, used here because you cannot backpropagate through prompt selection.
• Fitness function: the score a candidate prompt or harness is evaluated against on a trainset.
• Darwin-Godel machine: a meta-harness that can modify its own harness code, keeping an archive of agents and sampling new ones.
• Meta-harness: a harness whose job is to produce other harnesses.
• Continual harness: a harness that updates prompts, sub-agents, and even weights online during execution.
• DAgger: an online-learning method; here, updating the weight file at test time from newly learned examples.
• Test-time training / test-time compute: adapting or spending compute at inference rather than only during training.
• ARC-AGI: a benchmark measuring how fast a system adapts to a novel problem distribution; games are designed with orthogonal skills.
• Turing machine vs von Neumann computer: a sequential tape processor versus a machine with read/write addressable memory; the metaphor for what a harness adds.
• Orchestrator: the root/main agent that coordinates sub-agents.
• Daemon: a persistent background process that keeps an agent session alive after the user disconnects.
• L0/L1/L2/L3: Prime Agent's cache hierarchy (weights / active context / REPL and sub-agents / disk-backed state).
• Emulator bench: a benchmark where the agent reproduces an entire emulator (for example a Game Boy Color).
• PMPP-Hard / GPU kernel: a benchmark of writing GPU compute kernels (PMPP = "Programming Massively Parallel Processors").
• Factorio: a factory-automation game used as a long-horizon agent benchmark.
• oolong: a long-context evaluation referenced among Prime Agent's long-horizon evals.
• Verifiers: Prime Intellect's package for running standardized evals.
• Quantization: compressing model weights (for example to 4-bit) so larger models fit on consumer hardware.
• Ollama / llama.cpp / vLLM / SGLang: local LLM inference engines.
• MCP (Model Context Protocol): a standard protocol for exposing tools and memory to an agent.
• Apple Silicon / NPU: consumer accelerators (Apple's chips / neural processing units) for on-device inference.
• LoRA / SFT / GRPO / GEPA: learning methods; LoRA is low-rank fine-tuning, SFT is supervised fine-tuning, GRPO is a reinforcement-learning method, GEPA is a prompt-optimization method.
• Unhobbling: removing artificial limits so a model can use capability that is otherwise "trapped" inside it.
• Bulk upsert: inserting or updating many database rows at once; in QM, staged for human review before writing.
• ACL (access-control list): the rules that decide what an agent is allowed to read or write.
• Sandbox: an isolated compute environment; in QM, a resource the agent dips into rather than lives in.
• Cron: a scheduler that wakes an agent on a fixed interval.
• Postgres: the relational database QM uses to centralize agent state ("the brain").
• Situational awareness: the 2024 essay Regan cites for the "unhobbling" and capability-overhang framing.
• Slack tag: mentioning the agent in Slack to trigger a one-off task (YC's earlier coding-agent trigger).
• OpenClaw: an agent-with-its-own-computer product used by YC partners before QM.
• Hermes agent: the fleet of per-user VM agents YC ran before consolidating into QM.
• Grind tool: a QM tool that sets time or token budgets on a goal so the agent cannot quit early.