How execution variation propagates, when it affects outcomes, and whether agents can be steered before failure.
Research notes written alongside working systems and reproducible experiments.
AI companies
When your AI supplier becomes your competitorHarvey is growing fast between frontier labs moving up into legal software and legal incumbents moving down. Its answer so far isn't to replace its model suppliers: it's to own more of the layer between models and legal work. Is that enough?
New tech, explained
The model isn't the systemOn one coding benchmark, switching from a minimal harness to each model's vendor harness changed scores by as much as +14 points for one model and −13 for another. Benchmark numbers are partly system numbers, and changing the model means re-testing the harness.
New tech, explained
The question mattered more than the modelJev, a new "decision model", launched with big claims. Two independent tests point somewhere more useful: how you ask changed the result more than which model answered.
Systems, Seen · No. 2
The engine kept moving underneath OllamaOllama didn't choose between building and borrowing its inference engine. By 2026 it borrowed upstream llama.cpp for GGUF models and built its own MLX engine for safetensors.
Agent memory · Note 03
What happened when I made agent memory transitions explicitNew information isn't one memory operation, and memory history isn't reasoning history. What running a complete scenario through explicit memory transitions taught me about where the memory layer ends.
Agent reliability · Method note 03
Why agent CI can't be treated like deterministic testsRun the same agent twice and you get two different results. So what should a pull request check believe? A 25-point drop that wasn't enough evidence on its own, a capability that failed every run, and why running your eval more times doesn't settle it.
Agent reliability · Method note 02
Not every behavioral change is a regressionRegression testing assumes repeated runs are stable. Agents break that assumption. Two experiments — four correct runs that shared no trajectory, and 23 of 23 fork points diverging when only the GPU changed — pushed AgentSeism to a different order: comparability first, outcomes decide, traces explain.
Agent memory · Note 02
Memory is not a list of factsFour things building long-term agent memory taught me about state: confidence is not authority, observation time is not validity time, an audit log is not a replayable history, and there may be no universal rule for which memory wins.
Agent runtime · Field note 02
Restoring the code is not resuming the agentWhat agent replay taught me about durable execution: checkpoint consistency, execution ownership, fencing, and why resume is a continuation rather than a replay.
Agent reliability · Method note 01
One run is not an evaluationWhat repeated agent experiments taught me about variance, censoring, measurement bugs, and knowing whether a result is real — five evaluation assumptions that broke, and the protocol that replaced them.
AI systems · Inference notes 01
27B parameters, 48 GB of VRAM, and the things I got wrong about LLM inferenceA practical mental model for weights, context memory, prefill, decode, batching and prefix caching — from deploying a 27B FP8 model for an agent workload with vLLM, and finding out which of my intuitions were wrong.
Agent runtime · Field note 01
What does it take to actually replay an AI agent?Saving the prompt and the code is not enough. Reliable replay requires reconstructing the execution state the agent actually carried — tracked source, workspace artifacts, carried history, and a coherent resume point.
Agent memory · Note 01
From retrieval to state: rethinking long-term memory for AI agentsLong-term memory is usually treated as a retrieval problem. The harder problem is deciding what remains true after a history of changes, contradictions, decisions, and evolving goals — an agent can retrieve the right memory and still be wrong about the present.
Agent reliability · Experiment 01
12 of 13 runs reached the exact same code state, then ended 11 different ways — even at temperature 0Twelve of thirteen executions of the same coding agent, same model, temperature 0, reached an identical intermediate repository state — and then ended in eleven distinct final states. The useful question is not whether runs differ, but which differences reach the outcome.
No posts in this topic yet.
Measuring when execution variation becomes consequential, and testing whether agents can be steered before failure. Built around state-level instrumentation, replay, fork, and controlled intervention.
A reproducible reliability benchmark for memory-enabled agents — the harness whose failures drove the memory work.
Schema-aware typed memory for AI agents — what an agent carries between steps, and what that does to its behavior.
A layered knowledge engine.
