This year, the companies whose models Harvey builds on started selling legal AI too. Anthropic released its first legal plugin at the end of January and expanded it into Claude for Legal in May. In September OpenAI released Astra for Law, a version of its newest model configured for legal work.

So Harvey’s suppliers are becoming its competitors. Yet Harvey’s reported ARR continued to rise sharply. And in the same period, rather than replacing the labs, Harvey has been investing in more of the layer between their models and legal work: benchmarks, post-trained models, workflows, and people inside law firms.

This piece looks at how Harvey got here, where it sits, and what its valuation assumes. The question it ends on is whether owning more of that layer is enough.

How Harvey got here

For its first three years, Harvey’s product remained primarily an application layer built on other companies’ foundation models. It started as a legal assistant on GPT-4, added a document store (Vault), then agents and workflows. In 2025 it went from one model provider to several, adding Anthropic’s and Google’s models alongside OpenAI’s.

Its business grew quickly. The company reported more than $50M in ARR in February 2025, more than $100M that August, $190M in January 2026, and more than $400M in September 2026. Each new round was priced higher: $3B, $5B, $8B, $11B and now $15.5B, all within about 19 months.

2026 is where the lanes line up. The labs had no legal products before this year; by September both Anthropic and OpenAI had one. In the same months Harvey did things an application company usually doesn’t:

• it open-sourced a legal benchmark (LAB) in May;
• it ran post-training projects on open-weight models with partners from May to August;
• in August it released Tenet, “our first post-trained open-weight model”, as a research preview.

The timing is striking, but it doesn’t show Harvey was reacting to the labs. Harvey also tends to disclose its ARR around funding rounds, so the business lane is partly what the company chose to announce.

Where Harvey sits

The unusual part isn’t that frontier labs compete with Harvey. It’s that the same companies now stand in three places around it at once.

They supply its models. Harvey’s own benchmark writing describes a version of its Assistant as “built primarily on GPT-5”. OpenAI says Astra for Law will be “available soon to API customers, including Harvey and Legora”. Harvey’s newest capabilities will partly come from the same company that competes with it.

They host it. Alongside Astra, OpenAI launched legal plugins for ChatGPT, “26 from vendors such as Thomson Reuters, Harvey, Legora and iManage”. Harvey is now also something you can reach from inside a lab’s product.

They sell to its customers. Anthropic’s Claude for Legal comes with legal plugins and connectors for specific areas of law. Astra for Law is offered first to selected large firms. Freshfields has deployed Claude across the firm and is co-building legal workflows with Anthropic.

Around that centre, the pressure comes from other directions:

• Legora, another AI-native legal startup, is valued at $5.55B after its spring round, with more than $200M in ARR reported this month.
• Thomson Reuters (CoCounsel, Westlaw) and RELX/LexisNexis own the legal content and the distribution lawyers already pay for. RELX’s LexisNexis also licenses content into Harvey.
• Large firms can build their own. Freshfields is working directly with Anthropic, and Kirkland & Ellis has reportedly committed to its own platform with Palantir.

What Harvey actually owns

Judging by what Harvey sells and where it is hiring, it still looks primarily like an application and deployment company:

• Customers. The co-founder says Harvey has more than 3,000 customers, including 80% of the top 100 US law firms, 20% of the Fortune 500 and half of the Fortune 10. How many of those are firm-wide deployments isn’t disclosed.
• Workflow product. Assistant, Vault, agents and collaboration features. That’s the part lawyers use every day.
• People inside firms. Of 304 open roles on its job board, 33 are for legal engineers, people who deploy Harvey into a firm’s work. About six or seven are for research or model training.

Below the app, it owns less than the headlines suggest. It hasn’t pre-trained a model. Its post-trained models start from other companies’ open weights (Moonshot’s Kimi, Zhipu’s GLM, Alibaba’s Qwen) and are trained with partners. Tenet is a research preview, and nothing public shows it serving production traffic. Its benchmark is open, and its gains are measured on that benchmark.

Harvey does give reasons for going down the stack. On its own benchmark, “reaching the top of the closed-source leaderboard runs to roughly $50 per task and over 20 minutes of latency”, and frontier models complete “less than 10% of tasks end-to-end”. Open-weight models “can be hosted within a firm’s own secure cloud environment”. After the latest round, its co-founder said the company would invest “heavily in both its harness and its own model training”.

So the honest description today is an application company with a growing bet below the application layer. The bet is not yet a business.

What investors are pricing in

At $15.5B and more than $400M in ARR, Harvey is valued at no more than about 39 times its recurring revenue. That’s lower than at its earlier rounds, because reported ARR grew faster than the valuation, but it’s still far from the incumbents. Thomson Reuters and RELX trade at roughly 5–6 times their sales. Legora sits in the same range as Harvey.

These numbers measure different things (a private round’s valuation against reported ARR, against a public company’s enterprise value over a year of total revenue), so the comparison isn’t precise. What it shows is scale: the multiples imply expectations very different from those attached to mature legal-information businesses.

That leads to the useful question. It isn’t “is Harvey worth $15.5B?” but what has to become true for Harvey to grow into these expectations?

Three paths

The evidence supports three plausible directions. None of them is decided, because the numbers that would decide them (margins, retention, and whether Harvey’s own models carry real traffic) aren’t public.

1. A vertical intelligence layer. Harvey’s own benchmark, models and data become the reason firms stay.

• Must become true: its own models serve a meaningful share of production work, and that shows up as better margins or prices; firms pay for firm-specific models.
• Breaks it: the next frontier generation beats post-trained open models at similar cost, or firms post-train their own.

2. The deployment platform on other people’s models. Harvey stays model-agnostic and wins on workflow, integration and the people it puts inside firms.

• Must become true: retention and expansion hold as the labs sell direct, and deployment depth grows faster than the labs’ own legal offerings.
• Breaks it: firms standardise on a lab plus an internal platform, as Freshfields is doing with Anthropic, or usage pricing compresses revenue per seat.

3. Beyond law. Harvey becomes a platform for professional services more broadly. It has bought an asset-management platform and says it works with 125+ asset managers.

• Must become true: non-legal revenue becomes a material, disclosed share.
• Breaks it: incumbents or the labs own those workflows first. The evidence here is mostly announcements.

What limits every path

• The technology isn’t done. On Harvey’s own benchmark, frontier models complete less than 10% of long tasks end to end, and Tenet improves on its base model without getting close.
• The labs move up. Their legal products, and their models inside competitors’ products, reduce what an application layer adds.
• The incumbents move down. Thomson Reuters and RELX have the content and the contracts lawyers already rely on.
• Customers can build. The largest firms have the money and, increasingly, the partners to do it themselves.

The open question

We know Harvey has built distribution and revenue quickly. We know it’s experimenting below the application layer. We don’t yet know whether owning more of the model stack is necessary for its business, or even valuable to it.

The open question isn’t whether Harvey can build a legal model. It’s whether owning more of the intelligence layer makes the company harder to replace than simply owning the customer relationship.

What we know

Sources: Anthropic’s public knowledge-work-plugins repository (the legal plugin is in its first commit, 2026-01-29); Harvey’s blog (the Tenet research preview; post-training with Baseten; BigLaw Bench Arena); LawSites on the September 2026 round and on Astra for Law; TechCrunch and PointBlank on Claude for Legal; Freshfields on its Anthropic partnership; Harvey’s job board (Ashby, read 2026-09-28); public-market multiples as of 2026-09-27. Revenue figures are company-reported unless marked; where sources disagree, the research traced each figure to its date and definition rather than picking one.

When a leaderboard says a model solves 62.9% of tasks, it’s easy to read that as the model’s ability. But an AI coding agent is a model inside a harness: the tools it can call, its prompts, how it manages context, the loop it runs in, the checks around it. The score belongs to the pair.

How much does the harness matter? One benchmark ran the same models in different harnesses, and the answer is: enough to flip the result.

On Terminal-Bench 2.0, switching from a minimal harness to each model’s vendor harness changed scores by as much as +14 points for one model and −13 for another. And the direction depended on which harness it was.

What the table shows

The benchmark’s authors ran each model-and-harness pair at least five times and report the uncertainty. Nine models were run both in Terminus 2, a deliberately minimal agent, and in their vendor’s own harness:

• Codex CLI helped every OpenAI model: +14.4 points for GPT-5, +8.9 for GPT-5.2, +7.9 for GPT-5-Mini.
• Claude Code came out slightly below the minimal harness for every Claude model, from −0.8 to −5.7 points. Each of those gaps on its own is within the reported uncertainty; it’s the consistent direction that stands out.
• Gemini CLI cost Gemini 2.5 Pro 13 points, and Gemini 2.5 Flash 1.5.

The authors draw their own conclusion from it: “model selection is usually more important than agent scaffold.” That’s true on average. But for a single model, the scaffold was worth anything from −13 to +14 points.

What it doesn’t show

It isn’t a ranking of harness quality. Terminus 2 was built by the benchmark’s own authors, for this benchmark, so it plays at home. Claude Code trailing it here doesn’t mean Claude Code is badly built, and Codex CLI’s gains here don’t transfer automatically to your tasks.

It’s also one benchmark, made of terminal tasks.

What it does show is narrower and more useful: a harness has no fixed value on its own. Its effect depends on the model inside it, and can change sign.

So when the model changes, re-test the harness

If the value of a harness depends on the pairing, then upgrading the model breaks your knowledge of that pairing. The harness that helped last month’s model may be neutral, or in the way, for this month’s.

The practical rule is simple: when you switch or upgrade the model, re-run the comparison. Try your full harness against a simpler one. Remove components one at a time. Keep what still helps.

And when you read a leaderboard, ask what system the number came from.

An ablation, not a comparison

Comparing model A in harness X with model B in harness Y tells you which pair is better, not why. To learn what your harness is worth, hold the model fixed and take the harness apart:

model fixed, same tasks, several runs each

full harness                 → score, cost, time
minimal baseline             → score, cost, time
full − planning step         → ?
full − retry loop            → ?
full − custom tools          → ?
full − context rules         → ?
full − checks                → ?

Two things from the evidence above are worth building in:

• Run each configuration several times. Terminal-Bench ran every pair at least five times and still reports roughly ±3 points. A difference smaller than that is noise.
• Record cost and time, not just the score. A component that no longer changes accuracy may still pay for itself by saving tokens, as the planning study found.

Keep a component only if taking it out makes things measurably worse. Then run the same table again when the model changes.

Harnesses also age (a separate, weaker signal)

There are other signs that particular scaffolding loses its value as models improve, though none of them come from the table above, which is a snapshot, not a history:

• A study of agent design choices found that planning “shifts from an accuracy scaffold for weaker models to a cost saver for stronger models”.
• METR compared Claude Code with a simple scaffold for Opus 4.5. Claude Code came out ahead in 50.7% of their bootstrap samples: a coin flip.
• Anthropic, describing how it builds harnesses for long-running work, puts it this way: “every component in a harness encodes an assumption about what the model can’t do on its own, and those assumptions are worth stress testing.”

That fits the idea that a harness moves with the model rather than simply growing. But it’s supported by a handful of reports, not established.

What might last: a hypothesis

If some scaffolding goes stale, what doesn’t? Practitioners and vendors keep pointing at verification: tests, checks and other feedback the agent can run on its own work. It’s plausible that this ages more slowly than prompts and workflow tricks. On the evidence I found, though, it’s a hypothesis. Nobody I found has measured it across model generations.

Summary

The model isn’t the system. When you change one, re-test the other.

Sources: Terminal-Bench 2.0 (arxiv.org/abs/2601.11868, appendix Table 2); METR, “Measuring time horizon using Claude Code and Codex” (February 2026); the empirical study of agent design choices (arxiv.org/abs/2609.20804); Anthropic, “Harness design for long-running application development”. Each quote and number was read on the original page.

It launched with big numbers: “193.6x faster”, “can’t hallucinate”. I skipped those and read the independent tests instead. Two of them, by different people, point the same way:

How you ask mattered more than which model answered.

• Asked one broad question (“is this phishing?”), Jev scored 62.6% and a small ordinary LLM scored 81.3%. Given the same five narrow questions, both scored 93–95%.
• Asked “which face will this die show?”, Jev was 83% confident and right 19% of the time. Asked “is it this face?”, it said 19%, close to the true 1 in 6.

What Jev is

Jev’s documentation describes three kinds of question: a choice from a list, a score on a rubric, and a true/false question, which returns a probability. Several questions can go in one request, and each is answered “in parallel and in isolation”. Input costs $0.042 per million tokens, and output is free.

The docs also say how it’s meant to be used: “Atomic questions, composed in code.” Keep each question narrow, and combine the answers in your own code.

That advice turns out to be the whole story.

Where would I use it?

Vercel, which serves Jev through its AI Gateway, draws the line in one sentence: “Choose Jev for bounded decisions, code for fixed rules, and a generative model for prose.” In a real agent or app, that looks like this:

The first three rows are uses Vercel lists (alongside prioritizing tickets, categorizing documents, moderation and choosing which model answers). Vercel also says the guardrail doesn’t replace permissions: “Enforce access rules and required approvals in application code before executing a tool.”

The short version: use it at bounded decision points, not everywhere you’d use an LLM. The tests below show how well it holds up there.

Test 1: one question or five

An independent benchmark (jev-phishing-bench) ran Jev and Claude Haiku 4.5 on 2,000 synthetic emails, half of them phishing.

Asked the single question “is this phishing?”, Jev got 62.6% right and Haiku 81.3%. On that framing, the ordinary LLM clearly wins.

Then the author asked both models five narrow questions (for example, whether the sender looks generic), and fitted a small logistic regression on 1,000 labelled emails to combine the answers. Jev reached 95.0% and Haiku 93.2%. The benchmark calls that “statistically tied” (p = 0.063), and Haiku’s AUROC was slightly higher.

The biggest change wasn’t which model answered. It was how the task was decomposed.

What Jev kept was cost and speed. The benchmark’s own summary: “about 27 times cheaper and 5 times faster than Haiku for signals of comparable quality”. That’s the real case for a model like this: it makes asking many narrow questions cheap.

Two caveats the benchmark states itself, and which matter:

• The emails are synthetic, and a plain regex rule already gets 91.8%.
• The five questions “were written after reading its URL-evasion taxonomy”. They were designed knowing how this dataset was built.

Test 2: “which one?” or “is it this one?”

A second independent study (jev-does-not-play-dice) asked Jev about events nobody can predict, like a fair die roll or a coin flip.

Asked as a choice, “which face will it show?”, Jev was badly overconfident. Over 400 die rolls its average confidence was 82.9% and it was right 19.0% of the time, about what guessing gives. On coin flips it said 92% and was right 52%.

Asked as a yes/no, “is it this face?”, its answers were much closer to the truth: an average of 19.2% for an event with a true probability of 16.7%. It isn’t perfect. For rarer events it drifts high, saying about 15% for something that happens 5% of the time.

So the same model, facing the same uncertainty, gave very different confidence depending on how the question was framed.

On a real task the picture can be much better: a separate test on 60 hand-labelled agent tool calls got 91.7% right, and every wrong answer came with confidence below 1. Confidence depends on the task and on the question. It has to be checked, not trusted.

“Can’t hallucinate” means “can’t leave the schema”

TypeSafe says Jev “can’t hallucinate”. Its own launch post qualifies this: “Our number is not empirical. Schema matching is guaranteed.”

That’s the precise meaning. Jev can only answer with the options you gave it. It can’t invent a category or return malformed output. But it can still pick the wrong option, confidently, as the die shows.

What I’d take from this

If you use a decision model like Jev:

• Design questions, not prompts. Narrow questions combined in code did better than one broad one, for both models tested.
• Its advantage is cost and speed, not accuracy. On this evidence, it makes the decomposition affordable rather than making each answer smarter.
• Check its confidence on your own labelled data before acting on it. The form of the question changed it.

What I don’t know

• Both studies are small and single-author; one uses synthetic data that a regex handles well.
• Nothing here measures production traffic.
• TypeSafe’s speed and cost multipliers (“193.6x faster, 444.6x cheaper”) are its own, from its own workflows, and it says they are “on the higher end of real world gains”. I haven’t used them.

Sources: TypeSafe’s launch post and documentation; Vercel’s guides “When should you use Jev instead of a chat model?”, “7 practical Jev use cases” and “Where does Jev fit in an AI agent loop?”; jev-phishing-bench (github.com/anisselbd/jev-phishing-bench); jev-does-not-play-dice (github.com/KantaHayashiAI/jev-does-not-play-dice); the tool-call risk benchmark by webofmike on dev.to. Each number was read on the original page or repository.

Ollama’s git history gives an unexpected answer: both.

Over three years, Ollama moved from borrowing llama.cpp, to copying it into its own repository, to building its own GGML-based engine. Then, in May 2026, it deleted that engine and handed GGUF models to upstream llama.cpp’s server.

That sounds like a reversal. It wasn’t quite.

At the same time, Ollama was building another engine of its own, on Apple’s MLX, and it kept that one. By September 2026 the stack had split:

• GGUF models → upstream llama.cpp. Ollama borrows it.
• safetensors models → Ollama’s MLX engine. Ollama builds it.

The interesting decision wasn’t build or borrow. It was where to own the engine.

What makes this easy to miss is that users never had to notice: every command and API endpoint a 2023 user had is still there. I read Ollama’s full git history (5,796 commits, June 2023 to late September 2026) to see what happened underneath.

What users saw

The surface only grew. ollama run MODEL and the eight original endpoints (/api/generate, /api/pull, /api/tags and the rest) exist at every point I checked, including today. On top of them came /api/chat (December 2023), an OpenAI-compatible API (February 2024), /api/embed (July 2024) and account endpoints (September 2025).

Nothing a 2023 user relied on was removed. From the perspective of that original surface, Ollama remained compatible while adding more.

What moved underneath

The engine is a different story. Reading the tree and the commit messages, it went through these places:

• July 2023: Go bindings. Ollama called llama.cpp in-process (“add llama.cpp go bindings”).
• August 2023: llama.cpp’s server, as a separate process. “subprocess llama.cpp server” removed the C code; Ollama now ran llama.cpp’s own server and talked to it. From January 2024 it carried its own patches on top.
• October 2024 (v0.4.0): llama.cpp copied into the repo. “Remove submodule and shift to Go server”: llama.cpp was vendored, and driven by a Go server.
• December 2024 onwards: Ollama’s own engine. “Runner for Ollama engine”, then a GGML-based backend and model implementations written in Go, running alongside the vendored llama.cpp.

Plotted by how much of the engine Ollama owned, that’s a steady climb: from borrowing llama.cpp, to copying it in, to building its own.

It looks like build, then borrow

Then, on 29 May 2026, one commit reversed most of it:

Remove the vendored GGML and llama.cpp backend, CGO runner, Go model implementations, and sample. llama-server (built from upstream llama.cpp via FetchContent) is now the sole inference engine for GGUF-based models.

The engine Ollama had built for GGUF models was deleted. So was the copied llama.cpp. Ollama went back to running llama.cpp’s server as a separate program, this time upstream’s own, built at build time, with a small compatibility layer so it can load Ollama-format model files.

It is tempting to read that as a verdict: they tried to build their own engine, and it didn’t work out.

That’s the wrong conclusion

The same commit carries a parenthesis:

(Safetensor based models continue to run on the new MLX engine.)

Ollama had been building a second engine of its own, on Apple’s MLX, since February

1. About three and a half months after the GGUF engine was removed, the MLX one was promoted: “the only Go inference runner left and is no longer experimental.”

So in 2026 Ollama didn’t stop building. It split:

The decision wasn’t build or borrow

“Build or buy” is usually framed as one decision for a whole component. Ollama’s history shows it made twice, and the second time it wasn’t made for the whole component at all. The line ended up being drawn by model format: GGUF moved to upstream llama.cpp, while safetensors stayed on Ollama’s own MLX engine.

The more useful question isn’t “should we build our own engine?” It’s “where should we own the engine?”

The only reason the maintainers give is in that commit: “This allows us to more rapidly pick up new capabilities and fixes from llama.cpp as they come out.”

What this doesn’t tell us

Everything above comes from the repository itself: the file tree at different dates, and the commit messages. So there is a lot it can’t say:

• whether the switches made Ollama faster or slower, or changed anything users noticed;
• what the in-house GGML engine cost to maintain, and why exactly it was dropped, beyond the one sentence above;
• how the team made these decisions;
• whether the MLX engine will one day go the same way.

Commit messages are the maintainers’ own framing. I’ve quoted them, and checked each phase against the tree, but I haven’t filled the gaps with guesses.

What the history does show is a pattern worth recognising in any system you depend on: a surface that never broke, over an implementation boundary that kept moving.

Method: a full clone of github.com/ollama/ollama (HEAD 16b4376a, 2026-09-26), read from the raw history; no third-party summaries. Engine phases: 6093a88c, 42998d79, b754f5a6, ed443a03, dcfb7a10, d8cc798c, 9db4bdba, 2e036e7c. Surface additions: 7a0899d6 (/api/chat), 453f572f (/v1), b9f5e16c (/api/embed), 8b894933 (accounts).

Then I implemented those transitions in TypedMem and ran a complete scenario through them: an agent investigating a production incident. Two things became clearer than they were in that post.

First, “new information arrived” is not one memory operation.

Second, recording how memory changed is not the same as recording why an agent changed its mind. I had blurred that distinction myself.

One write, several possible meanings

An operations agent is investigating why clients can’t connect to a logging service. Information arrives from runtime telemetry, support cases, a certificate validation system, and later telemetry.

The scenario uses three kinds of memory, each with its own transition rule:

Here is what happened in the run:

• Telemetry reports that certificate errors are associated with the failures, and the agent stores it as evidence. Support cases report the same thing. TypedMem doesn’t create a second record: it keeps one record with two sources, and its confidence rises from 0.60 to 0.72.
• The agent stores its own assessment: the certificate is causing the failures. The validation system reports that the certificate is valid, stored as an assessment on the same subject. Both are kept and marked as conflicting. The conflict stays open: nothing later resolves it, and the memory layer doesn’t pretend otherwise.
• The agent decides to investigate the certificate. Later, telemetry shows that failures correlate with lost connectivity to the logging service, and the agent revises its decision. The new decision supersedes the old one: the old one stays in history, and only the new one is current.

TypedMem didn’t discover any of these relationships

The text never decided anything. I did, when I told the system which kind of memory each piece of information was:

• telemetry and support cases are evidence;
• the two judgments about the certificate are assessments about the same subject;
• the investigation path is a decision.

TypedMem then applied the rule each type declares. It doesn’t read the text: it matches records by type and subject, and the type’s rule decides what happens.

While writing this I ran the obvious mistake. I filed the validation result as another evidence record on the same subject as the agent’s assessment. There was no conflict. The store kept one record, whose text was still “The certificate is causing client failures”. The validation system became its second supporting source, and confidence rose from 0.60 to 0.72. The disagreement disappeared into agreement.

That isn’t a bug in the policy engine. It shows the boundary: the memory layer can enforce semantics, but the application has to assign the right semantics first.

Authority is a policy, not the truth

In an early outline of this post I wrote that picking the more trusted source “silently turns authority into a truth oracle”. That was too strong.

Sometimes authority is exactly the rule you want. For TypedMem’s replace-style memory types, a newer record from a weaker source can’t displace an existing one from a stronger source: it is ignored. In an earlier measurement, typed memory with that rule scored 1.00 on the authority cases.

Other kinds of memory want disagreement to stay visible. That’s what the flag rule is for. And evidence wants agreement to accumulate.

        the same incoming information
                     │
                     ▼
       depends on the memory's contract
         ┌───────────┼───────────┐
         ▼           ▼           ▼
       veto         flag      reinforce

The point isn’t that authority is bad. It’s that different kinds of memory need different transition rules, not one universal resolver of truth.

What the event log recorded

Every transition is recorded. Grouped by what it means, the run’s history is:

1. evidence added (telemetry)
2. evidence reinforced (support cases)
3. assessment added (the agent: the certificate is causing failures)
4. competing assessment stored; both records flagged (validation)
5. decision added (investigate the certificate)
6. evidence added (connectivity)
7. decision superseded (investigate connectivity)

The raw log is more detailed. Flagging and superseding touch both records, so they write paired events: nine events for these seven steps.

Replaying the whole log rebuilds the store exactly: the same six records. Replaying only the events before step 7 rebuilds the earlier state, in which the current decision was still “investigate the certificate”.

Memory history is not reasoning history

This is the part I got wrong.

The main figure of the first post had this caption:

The important object is not an isolated memory item. It is the transition structure that explains how the present emerged from the past.

I would phrase that differently now. Here is what the implementation actually knows:

The event log can tell me that the connectivity evidence was stored before the investigation decision changed. It cannot tell me that the evidence caused the change. In the run, the new decision’s only source is the agent, and nothing links it to the evidence.

Time order is not causal explanation.

It’s tempting to draw the investigation as one chain (telemetry → hypothesis → conflict → new evidence → decision revised). That drawing would claim knowledge the system never stored.

Replay has the same boundary. It answers “what did memory contain at this point?”, not “why did the agent decide that?”

Where the memory layer ends

I started with the idea that long-lived memory needs explicit state transitions. Implementing it made the boundary sharper:

• The application assigns meaning.
• The memory layer enforces the declared transition.
• The event log records what changed.
• None of these, by itself, records why the agent reasoned from A to B.

That leaves the question I don’t know the answer to yet. How much decision provenance belongs in a general memory layer? Should links from evidence to decisions be first-class memory structure, or stay application-specific?

If you’ve built long-running agents and had to answer “why did it decide that?”, I’d like to know where you ended up putting that link.

The scenario is a runnable example, examples/incident_investigation.py. It uses TypedMem’s public API, and a test pins its output, so the run described here is the run you get.

AI agents break it.

Run an agent twice on the same task, with the same code, model and prompt, and it can read different files, call tools in a different order and end somewhere else. Sometimes it succeeds both times, sometimes once. So when a pull request touches the agent and the eval score moves, what should the check believe?

A check that failed for the right reason

Here is a pull request that looks like a harmless refactor. It changes one line in a checkout handler, and the field it now reads, amount_cents, does not exist.

The check ran each of seven tasks eight times on main and eight times on the branch. Overall success fell from 0.93 to 0.68, and two things happened that ordinary CI has no vocabulary for:

• The broad reliability check passed. The evidence supported some deterioration, but not a suite-wide drop of at least the 10-point threshold declared in advance.
• The check still failed, because checkout, which had succeeded 8 of 8 times on main, succeeded 0 of 8 times on the branch.

A third detail is easy to miss: return-label went from 7/8 to 3/8 and was flagged as a warning, not a failure. Its success rate didn’t change in this PR, so that drop is pure run-to-run wobble, and a warning is the right amount of alarm for it.

A single pass/fail test cannot express any of that. Neither can a single eval score.

One run is a sample, not a replay

A deterministic test is a replay: run it again and you learn nothing new. An agent run is a sample from a distribution of possible runs. That changes what a comparison between two branches can tell you, in three ways.

1. Scores move when nothing changed

On a real coding agent (mini-swe-agent with Claude Haiku 4.5, on five SWE-bench tasks run five times each), we ran an unchanged candidate against its own baseline. Success went from 0.92 to 0.88. One task went from 3/5 to 2/5 with no code change at all.

A deterministic mindset reads that as “the PR made it worse”. It didn’t. There was no PR. A check that blocks on this trains developers to click “re-run” until it goes green, and after that nobody trusts it.

2. Averages hide breakage

The opposite failure is quieter. In the pull request above, the average fell 25 points, but not evenly: most of it was one task collapsing completely. Averaged over the suite, a total failure of one capability looks like a moderate, uncertain drop.

This isn’t hypothetical for us. The first version of our own check measured only the suite-wide average, and on a real agent it passed a change that cut success from 0.92 to 0.52, with two tasks collapsing. The statistics were computed correctly. They were answering the wrong question. That failure deserves its own post.

3. Sometimes the honest answer is “not enough evidence”

Three tasks, run twice each, going from 100% to 83% is not a regression and not a pass. It is too little data. A binary pass/fail setup has no honest way to represent that, so it gets rounded to whichever side of the threshold it falls on.

Why not just run your eval more times?

Running more repetitions is the obvious fix, and it helps, but it doesn’t answer the question on its own. More numbers still leave four decisions open:

• What counts as independent evidence? Fifty reruns of one task tell you a lot about that task and almost nothing about the others. For a claim about the whole suite, the task is the unit, not the run.
• How big a change matters? A 2-point drop can be real and still not worth blocking a merge over. You have to say how much you care about before looking.
• How do you avoid false alarms across many capabilities? Watch 50 tasks with a simple per-task rule (“block if a task that passed at least 7/8 drops to at most 2/8”) and some healthy PR will eventually trip it by chance. In our simulations of a flaky agent, that rule false-blocks about 1.4% of unchanged PRs at 5 tasks and 13.4% at 50. A rule that corrects for the number of tasks stays under 1% (computation, table C6).
• What do you do when there isn’t enough data? You need a third answer besides pass and fail.

Repeating the eval gives you the evidence. Something still has to turn that evidence into a merge decision.

Turning repeated runs into a merge decision

That is what AgentSeism does. You give it a command that runs your agent on a task and one that checks the result. It runs your base branch and your PR several times each, keeps each task’s results together, and returns one of four verdicts:

The two gates behind REGRESSION answer different questions, which is why they are separate: did the suite get broadly worse? and did something that used to work stop working? The pull request above is the case where the answers differ.

One correction to the previous note in this series. It ended on “Comparability first. Outcomes decide. Traces explain.” The first two became the product. The third, localising a regression to a stage of the agent’s trajectory, did not, and AgentSeism does not claim it.

How much should you trust it?

We froze the decision rule before testing it, then ran it on seven SWE-bench tasks it had never seen, with a pre-registered protocol: 224 real agent runs, 0 invalid, $38.49 in API cost. It passed an unchanged agent, and it caught both degradations that had been predicted in advance, a capability collapse and a broad collapse.

That evidence is narrow, and it is worth saying how:

• One agent, one model, one kind of regression. All of it is mini-swe-agent with Claude Haiku 4.5, and every degradation was made by cutting the agent’s step budget.
• Moderate regressions are hard to catch. The rule is built to avoid blocking healthy PRs, so a task falling from 100% to 75% usually won’t block a merge.
• It isn’t cheap yet. Eight runs per task on each side is a validation design, not an optimised CI budget.
• No one outside the project has used it yet.

The full protocol and results, including what the study did not establish, are in the repository.

Try it

If your agent’s eval score has ever moved on a PR and you couldn’t tell whether to believe it:

git clone https://github.com/lyr-ai/agentseism.git && cd agentseism
python3.11 -m venv .venv && source .venv/bin/activate
pip install -e .
seism demo

It takes about a second, with no API key and no Docker. Then look at the two public pull requests in agentseism-example: one docs-only change that passes, and the one-line bug above.

Or run your own eval eight times on an unchanged branch, and see how stable it really is.

If this is a problem you have too, a ⭐ on the repo tells me it’s worth continuing. An issue telling me where it breaks on your agent is worth even more.

Update, September 2026. AgentSeism’s product direction changed after this note. Comparability and outcome-based CI decisions became the product. Trace localisation, the “traces explain” step and the root-cause stage in Figure 2, remains research-era work and is not part of AgentSeism’s product today. See Method note 03.

Software regression testing assumes repeated executions are reasonably stable. If the same test passes before a change and fails afterward, the change is a plausible cause.

AI agents break this assumption.

An agent can take a different path even when its code, task, and model configuration appear unchanged. It may inspect different files, call tools in a different order, write a different patch, or recover from an error through a different route. Some of these changes are harmful. Many are not.

This creates a basic problem for agent development:

When an agent behaves differently after a change, how do we determine whether the change is a real regression, harmless stochastic variation, or the result of an incomparable execution environment?

I built AgentSeism to explore this question. The project began as a framework for detecting behavioral instability in agent trajectories. Experiments changed that direction. They showed that behavioral difference alone is not a reliable release signal.

The more useful design is an outcome-grounded regression layer for stochastic agents:

1. Check whether the two evaluations are comparable.
2. Decide whether task outcomes regressed.
3. Use traces only to diagnose a regression after it has been established.

That ordering matters.

Different trajectories can all be correct

Consider several independent runs of the same coding task under the same agent configuration.

In one frozen batch, five runs produced five different final workspace states. Four of the five solutions were accepted by the task evaluator; one was not. The four correct runs took 31, 41, 33 and 47 steps and submitted patches of 843, 1229, 978 and 1939 bytes. No two shared a tool-usage profile.

This small example exposes both sides of the problem.

Trajectory consistency would have been too strict. The four successful runs did not converge on one canonical trajectory or one identical final state. A test that required them to reproduce the same commands, intermediate files, or patch structure would have rejected valid solutions. I ran the check to see how badly: applying the kind of distribution-shift thresholds a behavioral fingerprint uses — trace length, tool mix, patch size — a composite detector fires on all six pairs of correct runs. Ground truth: none of the six is a regression.

Variation cannot simply be ignored either. The fifth run really was wrong. “Agents are stochastic” is not an excuse to treat every output as acceptable, and a design that merely tolerates difference would have passed it.

The useful distinction was not whether the trajectories matched. It was whether the resulting work satisfied the task contract.

Behavioral fingerprints can describe how an agent changed, but task outcomes must decide whether the change is a regression.

Trace similarity may still be valuable. A sudden increase in repeated tool calls, malformed actions, recovery loops, or unnecessary steps can help explain a failure. But these measurements should not silently become release gates unless the user has explicitly declared them part of the product contract.

Otherwise an observability signal becomes an accidental definition of correctness.

Sometimes the comparison itself is invalid

A second experiment exposed a different failure mode.

I attempted to continue archived agent trajectories on a new GPU host. The model identifier, model revision, vLLM version, CUDA version, prompt, and scaffold were held constant. The original trajectories had been generated on an A100 system; the new host used an H100.

Before running the full continuation experiment, I registered a serving-stack behavioral compatibility check. At each archived fork point, the new host was asked to generate the next action from the frozen prefix.

There were 24 distinct fork roots. One could not be compared because the original malformed response had not been preserved, leaving 23 archived-comparable roots.

The result was unambiguous:

• Raw responses matched on 0 of 23 roots.
• Structured actions matched on 0 of 23 roots.

In some cases both systems performed broadly similar exploration using different commands. In others the difference was categorical — the archived agent was writing a patch while the new host was still reading source code:

B_0 @ h=16   archived  find /testbed -path "*test*" -name "*.py" -exec grep -l "caplog" ...
             new host  find /testbed -path "*testing*" -name "*.py" -type f ...

A_3 @ h=24   archived  cat > /tmp/fix_clear2.py << 'EOF' ...
             new host  sed -n '685,700p' /testbed/src/_pytest/logging.py

It would have been easy to call this a behavioral regression. That would have been wrong. The agent revision had not changed. The serving environment had.

The appropriate result was:

INCOMPARABLE — serving fingerprint changed

This is more than a reproducibility footnote. Without a comparability state, an evaluation system is forced to misclassify environmental drift as agent behavior.

A regression framework for agents therefore needs at least three possibilities:

• the candidate is comparable and did not regress;
• the candidate is comparable and did regress;
• the candidate is not comparable to the baseline.

Treating the third case as a failed test hides the actual cause and encourages teams to debug the agent when the measurement instrument has moved.

One caveat I want to keep attached to that number: this is one stack change on one workload. The GPU, the driver and the kernel path moved together, so nothing here isolates a cause, and temperature 0 is not token-level determinism across execution stacks. What it supports is narrow and sufficient: a serving-stack change is not safely treated as an ordinary candidate mutation.

A better decision order

These observations led to the current AgentSeism decision flow.

1. Check comparability before running trials

The framework first compares a serving fingerprint containing the execution properties the user has declared invariant: model identifier and revision, runtime and dependency versions, GPU and driver, decoding and context parameters, prompt and scaffold versions, task image digest, evaluator version.

If a required field changes, AgentSeism returns INCOMPARABLE before spending money on candidate trials.

This check must happen first, and the reason is not only tidiness. Running a full evaluation and discovering afterward that the comparison was invalid converts an inexpensive metadata check into a wasted experiment. Detected first it costs zero trials; detected last it costs the entire budget for a comparison nobody can use.

2. Ground the verdict in outcomes

For comparable evaluations, the release verdict comes from features defined in a user contract: task success, evaluator-resolved status, safety violations, latency or cost limits, tool reliability, completion under a declared step budget.

Each feature has an explicit role. Some are release gates; others are warnings or descriptive measurements. The framework should not decide after seeing the results which metric mattered — which is why the contract is declared up front and hashed into every report.

There is a usability tension here worth naming. A developer should not need to understand risk difference or paired bootstrap to run a check, but a metric must not be chosen after the analysis. AgentSeism resolves it by expanding a three-line contract against versioned defaults into the complete contract that is actually validated, hashed, and printed. Defaults are fixed in a released version before anyone sees a number, so a default chosen in advance is not a metric chosen afterwards — and the report shows every field, including the ones the author never typed.

Repeated trials are sampled and analyzed at the task level rather than pretending that multiple runs of one scenario are independent tasks.

3. Run trace RCA only after a regression

If the outcome gate detects a regression, traces become useful diagnostic evidence. Did failures begin after a tool error? Did the agent exhaust its step budget? Did recovery behavior change? Did a particular stage accumulate extra retries? Did the candidate stop validating its work?

This is root-cause analysis, not verdict authority.

Keeping RCA off the main decision path has a practical effect: a suspicious-looking trace cannot turn a passing outcome into a regression unless the user explicitly placed that trace feature in the contract. In the implementation the renderer refuses to print RCA output under any verdict other than REGRESSION, because a reader would take it as a reason the merge is risky.

Four verdicts, not one score

The distinction between PASS_WITH_CHANGE and REGRESSION is central. It lets a report say: the agent behaved differently, but the registered product outcome did not degrade.

The distinction between INSUFFICIENT_EVIDENCE and PASS is equally important, and easier to lose. A small or invalid sample should not become approval merely because no statistically detectable failure appeared. In practice this row is the one most likely to be read as safe, so the report states it in words: neither safe nor unsafe — do not read this as a pass.

What AgentSeism is, and is not

AgentSeism sits between an agent evaluation harness and CI. The harness executes tasks and records outcomes. AgentSeism applies a predeclared contract, checks comparability, compares baseline and candidate, and produces a report suitable for a pull request.

It is not trying to replace domain-specific evaluators. A coding agent still needs tests or a task grader; a support agent may need policy and resolution checks; a security agent may need deterministic validation of its actions.

It is also not a generic “agent score”. Compressing correctness, cost, reliability, and trace behavior into one number would erase the distinction the system exists to preserve — and a single number needs weights, which is how a severe recovery regression gets averaged away by three healthy metrics.

The intended question is narrower:

Given this declared contract, is the candidate comparable to the baseline, and is there enough outcome evidence to call the change a regression?

The next test

The current evidence establishes two concrete failure modes:

1. correct agent runs can follow substantially different trajectories;
2. a serving-stack change can make apparently matched evaluations behaviorally incomparable.

It does not yet establish how well the complete workflow performs as a CI method.

To test that, I have preregistered a small pilot using controlled engineering mutations. It compares a baseline against a reduced agent step limit, and against a weakened recovery hint following a controlled malformed-action challenge. Three held-out tasks, three arms, two repetitions per cell — 18 runs. Its purpose is descriptive feasibility, not a general claim about all agents or tasks.

The experiment was registered before execution: mutation values, task-selection rules, run order and its hash, timeout handling, cost limits, and the interpretations allowed for negative or insufficient results. Two of those deserve mention because they are the parts most easily bent afterwards.

Obtaining a regression is not a success condition. If a registered mutation produces no regression, the null result stands. The mutation is not strengthened, the tasks are not changed, the threshold is not lowered, and repetitions are not added to earn one.

A timeout is not a task failure. The pilot has a per-run wall-clock cap for cost control. A run that hits it is censored, not failed — otherwise a budget rule would manufacture a regression out of long tasks, and long tasks are exactly what a reduced step limit is hypothesised to affect.

At the time of writing, the real pilot has not been run. That distinction is deliberate. The framework should not manufacture a regression merely to complete a demo, and this article should not turn an unobserved result into a product claim.

Why this matters

Stochastic systems need regression testing, but they cannot inherit deterministic testing assumptions unchanged.

Gate on exact behavior, and you reject harmless variation. Ignore behavior entirely, and you miss real failures. Compare runs produced by different serving conditions without checking compatibility, and you attribute to the agent what came from the measurement environment.

A useful Agent CI system therefore needs three separate concepts:

• Comparability — are these runs valid counterparts?
• Outcome regression — did a user-important result degrade?
• Diagnosis — where in the trajectory did the degradation likely arise?

The ordering is the method:

Comparability first. Outcomes decide. Traces explain.

AgentSeism is an attempt to make that ordering executable rather than leaving it as evaluation advice.

The project is still early. The most valuable next evidence will not be another abstract metric. It will be whether developers can declare a small contract, run a real agent change, and use the resulting report to make a merge decision.

That is the standard the project now has to meet.

What building long-term memory for agents taught me about state.

When I started building long-term memory for agents, I thought the hard problem was retrieval. Store memories, embed them, retrieve the relevant ones, rank them well enough that the agent sees the right context.

That model works — until memory starts changing.

Two years ago, a user said: “I live in San Jose.” Last week, the model inferred from a conversation: “The user lives in Seattle.” Which one should the system believe?

At first this looks like a retrieval question: retrieve both, let the model decide. But the moment an agent is expected to hold persistent state across months, it stops being about retrieval. It is a state-management problem, and it comes with the questions state-management problems always come with. Who is allowed to change this? When was it true, as opposed to when did we learn it? Can we reconstruct how we got here? Does every kind of state change the same way?

I ran into each of those while building TypedMem. Here is what each one changed.

1. Confidence is not authority

The San Jose memory came from an explicit user statement. The Seattle memory came from a model inference, with high confidence, made last week.

A policy that ranks by recency and confidence prefers Seattle. That is uncomfortable, and the discomfort is the point: the two memories differ on a dimension neither recency nor confidence measures.

Confidence answers how sure are we about this claim. Authority answers how entitled is this source to override another. An explicit statement from the user and an inference by the model are not interchangeable evidence, however sure the model is.

So authority became its own guard: a memory with weaker provenance cannot replace one with stronger provenance. Not folded into confidence, not weighted into a score — checked first, on its own.

The reverse is deliberately not true. Higher authority does not automatically win. An old high-authority statement should not overwrite a newer state just because its source is stronger; the user may well have moved. Authority is a veto for the weaker side, not a trump card for the stronger one.

The tempting alternative was a single number:

score = a·authority + b·confidence + c·recency

It is attractive because it produces one ordering. It also destroys the meaning of each term. Authority, confidence and time are different kinds of evidence, and a conflict rule that keeps them separate is slightly more explicit and much easier to reason about — you can say why a memory lost.

2. Observation time is not validity time

The original schema had one timestamp, and it was quietly doing three jobs: when the memory was observed, when its content became true, and where confidence decay starts. Those coincide often enough that one field seems fine.

Then: yesterday, the user says “I moved to Seattle a month ago.”

There are two times now. The memory was observed yesterday; its content has been true for a month. And the previous state has a natural end: San Jose stopped being true a month ago, not yesterday.

So a memory carries a validity window, half-open — [valid_from, valid_to) — separate from the observation time. Half-open so that when one state ends and the next begins at the same instant, exactly one of them is valid.

Two consequences I did not anticipate.

Future state becomes expressible. “Starting October 1, I’ll be using Postgres” is observed today and valid later. Which forces a rule: filter by validity first, then pick the latest — otherwise a declared future state shadows the current one merely because its start is later.

“Unspecified” has to stay unspecified. For old data with no validity window, the observation time is the operational fallback. But the storage keeps the distinction: a missing valid_from means the writer did not say, not the writer said it starts at observation. That sounds pedantic until you need to replay history and cannot tell which memories were declared and which were defaulted.

3. History is not replay

The system already had an event log. A replacement produced an event saying one memory replaced another, with the new version number and a fragment of the old content. Good for auditing. Useless for reconstruction.

The gap is between what happened and what the state was. Knowing that memory 42 was replaced at version 3 does not tell you what memory 42 looked like before, or exactly what it became. A log of events is a history of actions; it is not a history of state.

The fix was less clever than I expected. Every state-changing event now carries the full logical state of the memory before and after. Not a diff — the whole thing, twice. Creation has no before; deletion has no after.

That costs storage, and it makes replay boring: walk the events in order, keep the after. No patch language, no dependency on an earlier snapshot, no ambiguity about nested structure. Boring is the property I wanted.

One rule matters more than the format:

Replay must not re-run conflict resolution.

Suppose the conflict policy changes six months from now. If replay feeds the historical inputs through today’s policy, the reconstructed past differs from the past the system actually produced. Replay restores decisions; it does not reconsider them. The same log, under any current policy, replays to the same historical state.

That distinction — restore what was decided versus re-decide from the inputs — turned out to be the same one I keep meeting in agent runtimes: replaying a recorded execution and re-executing it are different products, and a system that blurs them is wrong about both.

4. There may be no universal rule for who wins

Once authority and validity were explicit, one more hard-coded assumption became visible. Under replacement, an incoming memory had to be no weaker than the existing one on both validity start and confidence.

The tempting abstraction was a lexicographic ordering: compare validity start first, then confidence as a tie-breaker. It is a natural thing to write, and it quietly changes the semantics. Timestamps rarely tie, so confidence would almost never be consulted. A newer memory with confidence 0.5 would replace a day-old one with confidence 0.9. That is newest wins with extra steps.

What the rule actually was — and what I kept — is a conjunctive guard: to replace, the incoming memory must be no weaker on every listed dimension. Newer and stronger replaces; newer but weaker is ignored; older is ignored. The order of the dimensions does not matter, because nothing is being broken in sequence — every one is checked.

And then the question that made this a per-type decision: should every kind of memory use the same guard? A deadline may care primarily about the latest effective state. A biographical fact may require both recency and confidence. Other memory types will make different trade-offs. These are different kinds of state, and there is no reason they share update semantics. So which dimensions guard replacement is declared by the memory type.

With one exception. In TypedMem, authority stays outside that list: a type may choose which temporal and confidence signals guard replacement, but it cannot silently disable provenance protection. Letting a configuration quietly turn off a safety invariant would turn the first lesson back into a suggestion.

What changed in my mental model

None of these are four features. They are consequences of one decision: treating memory as state rather than as a collection of documents.

retrieval asks       what is relevant to this query?
memory-as-state asks what do I currently believe, why, since when,
                     who was allowed to change it, and can I reconstruct
                     how I got here?

The system has more concepts now than it did — authority, validity, before and after, per-type guards. On paper that is more machinery. But the underlying problems existed before the concepts did. A single timestamp did not remove temporal semantics; it forced three kinds of time into one field. Ignoring authority did not remove provenance; it let confidence make decisions it was never designed to make. An audit log without state did not remove the need for replay; it made replay impossible. The distinction that matters is not simple versus complex. It is essential versus accidental complexity, and the test is whether each concept means exactly one thing.

What I deliberately did not build

Automatic closing of the previous validity window. Ranking by authority. Compaction and checkpoints for the event log. Migrating history under a new policy. Rebuilding a store from its log. A general rule language for resolution.

Some of these will turn out to be needed. None of them has yet, and an abstraction looks most attractive right before there is evidence for it. So the contract stops here: memories carry provenance, confidence, observation time and validity; types declare how replacement is guarded; every mutation leaves enough behind to reconstruct its result.

The next step is not another rule. It is measuring whether these semantics actually make agents more reliable. That is a question for a benchmark, not another design note.

While building replay support for AgentSeism, I tried to resume two coding-agent executions from exactly the same repository state. The source fingerprint matched byte for byte. The executions did not.

One carried a different scratch file. Their message histories were different. Restoring the repository gave me a state that looked correct according to my instrumentation, but it was not the same agent execution.

That failure changed how I think about long-running agents. Once an agent executes tools, modifies an environment, accumulates context, and runs for minutes or hours — on one fixed task I measured runs from three minutes to over six hours — recovery stops looking like “reload the prompt and workspace” and starts looking like a distributed-systems problem.

I had started with a simple question: “How do I save the workspace?” The experiment forced a harder one: “What does it actually mean to resume an agent?” Following it led quickly into checkpoint consistency, execution ownership, fencing, idempotency, replay semantics, and external side effects.

1. Repository state is only part of agent state

For a coding agent, the repository is an obvious piece of execution state and one of the easiest pieces to measure objectively. In AgentSeism, tracked source state is represented using a canonical Git diff, so two executions can be compared without interpreting model reasoning.

That is what made the pair in the opening measurable in the first place: the same canonical diff, byte for byte. Everything the diff does not cover — scratch files, message history — was different. The source code was identical; the complete execution state was not.

The repository is therefore one component of agent state rather than the agent state itself. Restoring repository state is not the same as resuming the agent execution.

Restore the transcript and the tracked source but drop the scratch files, and you get an agent whose memory contradicts what it can see: it remembers writing repro.py, reads it back, and is told the file does not exist. What follows is not a resumed job. It is a confused one.

2. A checkpoint is a recoverable boundary

A more useful definition is: a checkpoint is a recoverable boundary in the execution state machine.

For a long-running coding agent, that boundary may include conversation context, tracked and untracked workspace, tool state, execution step, remaining token/time/cost budgets, model configuration, and progress of externally visible operations. The exact schema is application-dependent. The important property is that the runtime can give a precise meaning to recovery from that checkpoint.

Restarting a fresh agent from an old Git diff may be useful, but it is a restart from source state, not a resume of the original execution. The two are different products, and a runtime should say which one it offers.

3. Partial checkpoints are more dangerous than missing checkpoints

Suppose an agent begins creating a checkpoint at step 100. Messages and workspace are stored, but the worker crashes before tool state is durable. Restoring the pieces that happen to exist would create a mixed state that never existed in the original execution.

This is worse than having no checkpoint at all, because it looks legitimate. A missing checkpoint costs work; a half-written one costs trust, and the failure surfaces much later as agent behaviour nobody can explain.

A safer design writes checkpoint components as immutable objects first and publishes a small manifest only after every required component is durable.

Recovery recognizes only a committed manifest. Partially uploaded objects are harmless orphans until garbage collection or later deduplication. The goal is to make invalid mixed states impossible to express, rather than merely detectable after the fact.

This is not two-phase commit. It is the much cheaper pattern — data first, pointer last — that Git uses for objects and refs, and that filesystem journals and object-store table formats use for the same reason: make the only mutable thing a single small atomic write.

4. Immutable data and execution authority are different problems

Suppose Worker A owns an execution under generation 7. A network partition causes its lease to expire, and Worker B takes ownership under generation 8. Worker A may still be alive.

If A finishes uploading an immutable, content-addressed workspace blob, little harm occurs. What A must not be allowed to do is declare a new recovery point after it has lost ownership.

This leads to a useful principle: separate immutable data-plane writes from ownership-sensitive metadata publication. Blob writes can be independent; publishing a checkpoint manifest or advancing the recovery pointer requires the current fencing generation.

5. The architecture naturally splits into a control plane and a data plane

As the recovery requirements accumulated, the control-plane/data-plane boundary became the architectural abstraction I found most useful.

The control plane owns execution intent and authority: job lifecycle, scheduling, leases, fencing generations, recovery lineage, quota, policy, and checkpoint publication authority.

The data plane performs the work: workers, sandboxes, the agent loop, model and tool calls, workspace mutation, immutable checkpoint production, and trace emission.

Two edges cross the boundary upward, and they are different in kind. The blob write is unconditional: content-addressed, immutable, harmless from any generation. The manifest publish is a request that the control plane may refuse, and refuses whenever the generation is stale.

The distinction is not merely organizational. The data plane is where risky and failure-prone execution happens. A buggy agent or crashed worker should not be able to redefine execution ownership or declare an arbitrary checkpoint authoritative.

The data plane may produce candidate state; the control plane decides which state is authoritative.

6. Lease, fencing, and recovery preserve authority

A lease defines time-bounded execution ownership, but it cannot prove that an old worker has stopped. During a network partition, a stale worker may continue running even after the control plane has reassigned the job. For a while, two workers are physically executing the same job.

Fencing therefore protects ownership-sensitive writes. Each assignment receives a monotonically increasing generation, and durable services reject mutations from stale generations. The store does the rejecting, not the worker, because a partitioned worker cannot be trusted to know it has been fenced.

Worker-side self-termination is still useful because it reduces wasted computation, but it is not the correctness boundary. Self-fencing is an optimization; server-side fencing is the correctness boundary.

The four mechanisms solve different problems, and each catches what the one above it cannot:

The bottom row is the trap: self-fencing is the only one of the four that is not a correctness mechanism, and it is the one that looks most like a fix.

7. “Latest” is not one thing

Suppose an agent reaches step 120 but its newest durable checkpoint is step

1. The worker crashes, and a replacement resumes from step 100 under a higher generation. Ownership moved forward while logical execution progress moved backward. Both statements are correct, and a design with one sequence number has to lie about one of them.

A runtime should therefore distinguish three orderings, and use each for what it is for:

• generation — who owns execution now. Strictly monotonic. Recovery selection uses this first.
• checkpoint sequence — which checkpoint within this attempt. Monotonic within a generation. Recovery selection uses this second.
• agent step — how far this branch has got. Not monotonic across recovery. Billing and budgets use it; recovery selection never does.

Choosing a recovery point by agent step is how a stale branch gets adopted because it happened to get further. I prefer recovery_head over latest_checkpoint: it means the newest durable recovery point on the currently authoritative execution lineage, not the largest historical step number.

The user-facing consequence is small but real: a progress indicator derived from the recovery head goes 120 → 100 after a recovery, which reads as data loss for a system that behaved correctly. Report the high-water mark and the current branch position as two numbers, and do not invent a percentage — an agent’s step count is not distance-to-completion.

8. Checkpointing does not make external effects transactional

The control plane can protect state it owns. The external world is harder.

Suppose an agent checkpoints, calls send_email(), the provider accepts the email, and the worker crashes before recording success. A replacement restores the previous checkpoint. The runtime cannot know whether retrying will duplicate the message.

No better workspace snapshot removes this ambiguity. A durable runtime therefore needs a separate effect protocol: write-ahead intent, durable logical operation identity, effect execution, result commitment, and reconciliation when the outcome is uncertain.

The write-ahead intent is what turns an unknown into a known unknown. On recovery, pending means this may or may not have happened — not a solution, but the precondition for every one that follows. The cheapest discharge is often to trade readability for idempotency: embed the operation id in the message itself, and on recovery search the provider’s sent items before deciding whether to send. Where no read path exists, the tool declares whether it prefers at-least-once or at-most-once, and for the latter the platform parks the job for a human rather than guessing.

Checkpointing recovers computation state; it does not make external side effects transactional.

9. Recovery and replay can want different observations

Read-only tools avoid duplicate effects but introduce another problem. Suppose an expensive search returns observation O1 after the last checkpoint, and the worker later crashes. Re-running the search may cost money and may now return O2.

Replay asks, “What did the agent observe then?” and usually wants O1. Recovery asks, “What should the live job observe now?” and may prefer O2.

For expensive observations, persistence can be decoupled from checkpoint cadence: record the result at the moment of the call, keyed by its semantic input, so that a crash does not force a high checkpoint frequency on a system that did not otherwise need one. Semantic cache identity must include all result-changing context, including tool version, canonical arguments, repository revision, and tenant or authorization scope. Omitting authorization scope can turn a performance optimization into a cross-tenant data leak.

Retention is also part of the contract. Recovery-critical payloads cannot be sampled away while recovery promises to use them. Observability payloads may use sampling or tiered retention. Digests can be kept broadly to detect that an observation changed even when the payload is no longer available.

If recovery is expected to provide a payload, its retention period is a correctness property, not merely a storage optimization.

10. Resume is a continuation, not a replay

Everything above restores the starting point. None of it restores the future, because the next model call is a fresh sample from a system that is not deterministic.

I measured this directly. Forking continuations from an identically reconstructed state — same tracked source, same scratch files, same transcript, temperature 0:

14 of 16 continuations reproduced the donor's next three actions exactly
 0 of 15 reached the donor's final state

Short-horizon behaviour is near-deterministic; long-horizon behaviour is not. The checkpoint guarantees the agent restarts from the same state, not that it does the same thing.

The same applies to a retried model call. A response lost in transit and re-requested is a new sample, not a repeat — even at temperature 0 the serving stack does not reproduce itself, and in one batch of 333 model calls, 12 were sampled more than once because of transport failures alone, one of them six times. A silent retry does not redo the same work slightly wastefully; it branches the trajectory, and the trajectory has to record that it did.

Two consequences follow. Idempotency keys cannot be derived from trajectory position, because the resumed trajectory diverges from the original. And if a product genuinely needs replay — audit, regression testing — it needs recorded outputs, every tool result and every model response, replayed from the log instead of re-executed. Replay from recorded outputs, yes; reproduce by re-execution, no.

11. Long-running agents start looking like durable workflows

A short agent can often be retried from the beginning. A two-hour agent cannot. As runtime increases, so does accumulated context, completed tool work, modified environment state, external effects, and compute already invested. The probability of an infrastructure failure during the run also increases.

At that point, durability stops being optional. The system begins to resemble a durable distributed workflow: a job has an owner, runs in an isolated worker, advances an agent state machine, produces checkpoints and effects, and may transfer ownership after failure.

This is why I expect agent infrastructure to borrow increasingly from workflow engines, distributed schedulers, transactional systems, and sandboxed compute platforms rather than only from model-serving APIs.

12. What I would build next

AgentSeism’s current replay and fork infrastructure is experimental rather than a production runtime. Its purpose is to reconstruct execution states precisely enough to study agent variation and intervention. But the work suggests a concrete production direction.

I would make the control-plane/data-plane boundary explicit; make checkpoints crash-consistent through immutable objects and atomic manifest publication; route consequential external operations through a guarded effect service; separate live recovery semantics from deterministic replay; and add stronger sandbox isolation plus a shared inference layer.

I would then test the design with deliberate failure injection: worker crashes, network partitions, stale owners, interrupted checkpoint publication, model timeouts, and ambiguous tool effects.

The key metric would not be whether a resumed agent follows exactly the same future trajectory. Section 10 says it will not. The stronger durability question is whether the system can resume from a well-defined state without corrupting execution, duplicating consequential effects, or losing information required for correctness.

Conclusion

I started with what looked like a small implementation task: reconstruct an agent at a previous step so I could replay and fork its execution. The first surprise was that restoring the repository was not enough.

Following that observation led naturally into checkpoint consistency, execution ownership, fencing, idempotency, observation persistence, and recovery semantics. None of these problems are unique to AI, but agents combine them in an unusual way because they mix long-running computation, stochastic model calls, mutable environments, and potentially irreversible tools.

The mental model I now use is simple:

An agent checkpoint is not a saved workspace. It is a durable boundary in an execution state machine.

Once an agent becomes long-running enough to deserve recovery, that distinction matters.

Abstract

I used to think evaluating an AI agent was conceptually straightforward: freeze a task and configuration, run the agent, record the outcome, compare scores.

Repeated executions changed that view.

The same agent, task, model and temperature=0 configuration can produce materially different executions. More importantly, the evaluation system itself can create artifacts that look like model behavior: a context-window limit can selectively remove long trajectories, a reasonable-looking state definition can create false reconvergence, an infrastructure timeout can masquerade as agent failure, and an obviously “cold” benchmark run can already have been warmed by an earlier failed measurement.

These are not implementation annoyances. They change what an evaluation result means.

My current view is that agent evaluation needs at least five separate layers: a frozen execution protocol, repeated runs, explicit termination and censoring semantics, validated behavioral measurements, and an external task-level outcome. Only after those are separated does it make sense to compare agents, or to claim that a behavioral difference matters.

This post records the evaluation assumptions that broke while I was building AgentSeism, and the protocol I use now.

1. “One run represents the agent”

Suppose Agent A solves a benchmark task and Agent B fails it. Which agent is better? With one execution each, we barely know.

A long-running agent is not a single model call. It repeatedly generates actions, modifies an environment, observes the result, and folds those observations into subsequent context, so small differences propagate through the execution.

In my coding-agent experiments, repeated executions used the same task, model, agent configuration and temperature=0, and still did not reliably follow the same trajectory. Some differences disappeared, some persisted, and some executions diverged, reached the exact same repository state, and diverged again.

A single execution therefore measures something closer to

— the outcome of run \(r\) on task \(i\) — than an intrinsic property of the agent. What we usually care about is a distribution:

Repeated execution is not simply a way to make an average more precise. It is how you find out whether the thing being evaluated has meaningful run-to-run variance at all.

2. “A failed run is an agent failure”

Repeated runs immediately raise another question: what counts as an outcome?

In one experiment I configured the serving system with a 32k context limit. Several trajectories terminated with ContextWindowExceeded. One task — Seaborn — lost all three of its runs that way.

It would have been easy to record success: false and fold those into the agent’s accuracy. But the model itself supported a much longer context. The 32k boundary was an infrastructure choice I had made. Those executions answered “can this agent complete the task under my 32k serving constraint?” — not “can this agent solve the task?”

The distinction got sharper when I noticed which executions were being removed. Long trajectories involve more exploration and accumulate more context, so context-window censoring was not random. The experiment was preferentially deleting a particular kind of execution: a missing-not-at-random pattern.

I now separate at least four termination classes.

The exact taxonomy depends on the system. Collapsing these into success=0 can distort an evaluation badly.

                       EXECUTION ENDS
                             │
             ┌───────────────┼───────────────┐
             │               │               │
             ▼               ▼               ▼
         COMPLETED      AGENT FAILURE    NOT AN AGENT FAILURE
                                         ┌──────┴──────┐
                                         ▼             ▼
                                  INFRA FAILURE     CENSORED
                                  endpoint died     context limit
                                  container crash   step limit
                                  transport error   time boundary

Figure 3. Failure is not one bucket. The question is not “did this run finish?” but “what process caused observation to stop?”

3. “Measurement code cannot manufacture a finding”

The most uncomfortable evaluation bugs are not crashes. They are measurements that produce plausible numbers. I hit three.

A “cold” run that was already warm

While measuring prefix caching, a script assumed repeat == 0 → cold and repeat == 1 → warm. Reasonable enough. But earlier executions had already sent the request and populated the cache before the script crashed while printing its results. On rerun, the nominally first measurement was already warm.

The eventual data made it visible: a genuinely cold ~28.5k-token prefill took about 12.1 seconds, while a cached repetition took about 0.43 seconds. Earlier ~14k measurements that had looked inexplicably fast were not evidence of exceptional cold-prefill performance — the cache was already populated.

The bug was not in vLLM. It was in the assumption that the first recorded measurement is the first system exposure. Experimental state can survive a failed measurement.

A reconvergence that never happened

AgentSeism compares trajectories by repository source state. The first implementation treated two runs as reconverged if they shared a source-state hash after diverging, and the analysis produced a lot of apparently anomalous pairs.

The reason was embarrassing. The hash e3b0c442… is SHA-256 of the empty diff. Every execution occupies that state before it modifies anything, so two trajectories could “reconverge” purely by both having changed nothing yet. Once the empty state was excluded, the anomalies largely disappeared.

Nothing about the agent changed. Only the measurement did.

Two “different outcomes” that were the same source state

Another pair appeared to finish with different patches even though their final tracked source hashes were identical. One submission contained roughly 18 KB of extra untracked scratch content; the actual tracked modification was the same ~720-byte code change in both.

Two different objects had been compared: a submission string and a canonical tracked repository state. Both are legitimate measurements of different questions. If the question is whether the agent produced the same source repair, the tracked state is the right outcome; if it is whether the agent submitted the same artifact, the string may be. The bug was silently treating the two as interchangeable.

4. “Fixing a measurement is a neutral act”

There is a dangerous moment in any exploratory experiment: the data look strange, you find a flaw in the analysis, you fix it, the results look cleaner — and then another strange result appears, and you fix that too. At some point “debugging the measurement” becomes indistinguishable from adjusting the analysis until the data tell a satisfying story.

I ran into exactly this.

The empty-state reconvergence issue was clearly a measurement bug: an unmodified repository should not count as meaningful reconvergence. But another pattern survived the fix — some trajectories reached the same non-empty source state and later diverged again.

My original classification treated that as anomalous, because it implicitly assumed that once trajectories reconverge they stay reconverged. The data showed the assumption was false. Changing the instrumentation to make those cases disappear would no longer have been fixing a measurement; it would have been deleting a real topology my conceptual model failed to represent.

So I kept the observation and expanded the representation to a triple \((D, R, F)\) — divergence, non-empty reconvergence, final convergence — which describes the three observed cases directly:

absorbed        1 1 1
persistent      1 0 0
re-divergence   1 1 0

Fix instrumentation when it fails to measure the construct you defined. Change the construct only when you are willing to say the original hypothesis or representation was incomplete.

Those are different scientific operations and should leave different records.

5. “Writing the design down makes it reproducible”

Preregistration sounds excessive for an engineering project. I found it useful for a blunt practical reason: it stopped me from making dozens of individually reasonable decisions after seeing the data.

For one experiment I froze, in advance: task-selection rules, run count, model and serving configuration, state representation, divergence and reconvergence definitions, treatment arms, outcome metric, stop conditions, and the handling of infrastructure failures.

It did not prevent mistakes. One donor-selection rule later produced two experimental arms with identical terminal targets, which made the planned treatment contrast unidentifiable.

What mattered was what happened next. There were other runs in the same dataset that would have produced different terminal targets, and picking one of those would have been easy. The gate returned UNIDENTIFIABLE, Phase B did not run, the rule was amended and documented, and the amendment was applied only to a new sample.

Preregistration did not make the original design correct. It made the design failure visible, which turned out to be the more useful property.

6. Repetition does not rescue a bad metric

Suppose I run an agent 100 times and precisely estimate how often it produces different patches. If all of those patches are correct, I have precisely measured a behavioral difference that may not matter.

This was the most important conceptual correction in the AgentSeism work. Initially I focused on trajectory variation: do executions diverge, do they reconverge, do they finish in different source states? Those are useful behavioral measurements. They are not task outcomes.

For coding agents the stronger outcome is externally measurable — FAIL_TO_PASS and PASS_TO_PASS give a correctness label independent of the agent’s own reasoning. That creates two layers: behavioral variation and consequential variation. Different trajectories with the same correctness may be harmless diversity; similar trajectories that repeatedly fail may be stable unreliability.

             REPEATED EXECUTIONS
                     │
                     ▼
         How much does behavior vary?
                     │
              trajectory metrics
                     ▼
        ┌─────────────────────────┐
        │ Behavioral variation    │
        └────────────┬────────────┘
                     │  + external labels
                     ▼
        ┌─────────────────────────┐
        │ Consequential variation │
        │ Did the variation       │
        │ change correctness?     │
        └─────────────────────────┘

Figure 5. More repetitions reduce uncertainty about the metric you chose. They do not make the metric more meaningful.

This is the direction of the next experiments: attach correctness labels to repeated trajectories, and find out which execution differences actually correlate with — or cause — failure. It is the same question Experiment 01 ended on.

7. “Evaluation measures the model”

An agent evaluation result belongs to an entire protocol. “Agent X achieves 72% accuracy” is incomplete without knowing what execution process produced the number: model and exact revision, sampling configuration, tool definitions, agent implementation, context-window limit, step limit, retry policy, container version, serving engine, concurrency policy, termination semantics, scoring implementation.

Some of those look like infrastructure trivia. They still change the distribution being measured.

I considered running several AgentSeism trajectories concurrently to cut GPU rental time. From a serving perspective that is obviously right — continuous batching improves utilization. But the experiment was studying run-to-run variation under temperature-zero inference, and changing concurrency changes batch composition and potentially the numerical execution path. A serving optimization would have altered the stochastic process under study.

In production I would enable concurrency. For that experiment I kept runs sequential. The correct configuration depends on the question.

8. The protocol I use now

The pipeline I want looks less like run → score:

In practice, six questions before I trust an agent-evaluation result:

1. Is the protocol frozen? Can I state exactly what system produced these executions?
2. Do I have repeated runs? Have I measured run-to-run variability rather than assuming it away?
3. Do termination classes have explicit semantics? Can I separate agent failure from infrastructure failure and censoring?
4. Has the measurement been validated? Do the hashes, state representations or judge scores correspond to the construct I claim to measure?
5. Is there an external outcome? Am I measuring task success, or only behavioral difference?
6. Is uncertainty visible? Does the reported result reflect the distribution I actually observed?

If I cannot answer these, adding more benchmark tasks is usually not the first thing I need.

9. What I still don’t know

This protocol solves part of the problem. Repeated executions get expensive, and the right number of repetitions depends on the agent’s variance and on the decision being made. Some evaluations have deterministic external labels; others need human or model judges, which introduce another layer of measurement noise. Production agents also operate on non-stationary task distributions, which makes any fixed benchmark an incomplete proxy for deployed behavior.

The question I care about next is more specific: once repeated executions carry external correctness labels, can we identify where successful and failed trajectories begin to become distinguishable, before termination?

If so, evaluation stops being only retrospective. Instead of “which agent scored higher?”, the question becomes “when did this execution become more likely to fail, and could an intervention at that point have changed the outcome?” That is the bridge between agent evaluation and agent reliability.

Conclusion

The biggest change is that I no longer treat the benchmark score as the beginning of the analysis. It is the end of a measurement pipeline.

Before trusting the score I want to know what protocol generated it, how much executions vary, why each run terminated, whether the instrumentation measures what I think it measures, and whether behavioral differences are grounded in an external outcome.

So the principle is not simply run the agent more than once. It is:

Repeated runs reveal variance. Valid measurement tells you what varied. External outcomes tell you whether the variation mattered.

Only then does it seem fair to call the result an evaluation.