When does project memory help a coding agent?

01 · the clear win

Memory shortened the dig.

Asked why keypress failures kept recurring across DotState input fields, both agents reached the correct architectural diagnosis. The Stele arm reached it with much less investigation because memory identified where and how to start verifying.

Correctness
1 = 1
Both arms earned full reward.
Investigation
−40.7%
16 investigative calls with Stele, 27 without.
Input tokens
−71.3%
519,028 with Stele, 1,811,074 without.
Recorded cost
−54.0%
$0.4268 with Stele, $0.9286 without.

Recall

Stele surfaced App::sync_input_mode, App::get_action, and the files src/app.rs and src/ui.rs.
Target

It supplied the literal source phrase ALWAYS insert the character first and the screen files where that duplicated guard lived.
Verify

The agent opened the named code, searched for the phrase, counted the implementations, and checked the screen trait before answering.

Stele did not replace investigation. It gave investigation a precise starting point.

The useful memory was operational: exact symbols, exact file anchors, the relevant event-ordering rule, and a search string that occurred in source. The agent still checked the central filter and the local per-screen guards. That is the behavior this benchmark was meant to distinguish from a memory system that simply encourages an agent to trust an old conclusion.

N=1 matters. This is a concrete, valid example of the mechanism working. It is not yet evidence for how often the same reduction will repeat across tasks.

02 · where memory did not win

Relevant memory still has a price.

The other admitted cohorts make the cost visible. In both, correctness stayed equal. Stele either had little useful history to apply, or surfaced a valid reminder that the control could recover cheaply from source.

Low-memory tare · N=1

Package cache was an intentionally small task.

Both agents made the correct change. Stele used one fewer call, but its fixed context and tool surface cost more than it saved.

+24.04s wall time
+83,520 input tokens
+$0.1261 recorded cost

Raw result · Preregistration

expected overhead, no benefit

Relevant but redundant · N=2

Settings-label memory was delivered and used.

All four agents solved the task. The reminder correctly identified the display-string contract, but both controls also recovered it from nearby source.

2/2 vs 2/2 correctness
+33.08s paired median wall time
+$0.205 paired median Gateway cost

Raw result · Preregistration

mechanism worked, outcome did not improve

The interpretation

An explanation for overhead is not a reason to hide it. Stele loads operating context, exposes graph tools, retrieves candidates, and asks the agent to maintain durable knowledge. On a small project, or on a task where the answer is already obvious from one file, that cost has little room to pay back. Stele, and persistent memory systems generally, may be the wrong trade for projects where discovery is already cheap.

03 · the product hypothesis

The early sessions fund the later ones.

Adopting project memory means paying token and tool cost before the graph is useful. Early sessions populate decisions, failed approaches, hidden contracts, and verified lessons. The thesis is that later sessions recover that cost by spending less time rediscovering the same facts. Discovery and investigation are often the most expensive part of a coding session, so this is where payback must appear.

This curve is a hypothesis, not a fitted result. The current release measures isolated value, tare, and null/mixed cases. A registered multi-session benchmark is still required to locate any crossover point or show that one exists.

04 · behavior under memory

We did not observe memory replacing verification.

A known failure mode for agent memory is overreliance: an agent accepts a stale fact, stops investigating, or skips verification. Recent memory evaluations have found that agents can retrieve updated evidence and still act on outdated assumptions. That failure mode did not appear in the admitted Stele trajectories.

Across four valid pairs, every Stele arm earned full correctness reward. In the keypress case, the agent followed memory into the named source and checked the pattern. In the settings case, both treatment agents used the reminder and still performed the task's required source and test work. The low-memory task also completed correctly despite the added overhead.

This is a limited safety observation, not proof that stale-memory harm cannot occur. The current suite is small and does not include a dedicated contradiction or stale-fact challenge. What it can say is narrower: within these admitted runs, Stele did not make the coding agent less correct or observably cause it to stop investigating and verifying.

Background: STALE benchmark on outdated agent memory

05 · admissions

Broken experiments stayed visible.

Seven versions were excluded for apparatus or question-design failures. Their evidence remains public, but none contributes to the effect estimates above.

Versions	Why excluded	What happened to the outcome
Keypress v1	Turn cap, invalid trajectory, and verifier defects	Retained, not pooled
Keypress v2	Forbidden repository-history inspection and verifier false negatives	Retained, not treated as replication
Package cache v1	Undeclared lockfile mutation prevented the hidden contract test	Retained, not pooled
Settings label v1–v3	Mutation allowlist, cold-build, and clean-exit budget defects	Retained, not pooled
Drift r1	The N=1 signal disappeared at N=5 because the question allowed multiple reasonable readings	Retained as a question-design lesson

What is still unknown?

We do not yet know a general win rate, stable average cost effect, or statistical significance. We have not yet measured whether graph-writing cost in one session pays back in later sessions. Keypress v3 and the low-memory tare remain N=1 until their frozen protocols are repeated.

06 · qualitative field evidence

Stele also found the work that was not on main.

comparison9 min · 7 calls
Stele47 sec · 3 calls
comparison tokens450K
Stele tokens154K

In a separate retained DotState session, both responses found the structural keypress problem. Stele also identified the parked tui-forge-integration workspace as a likely solution path and pointed to its owned-focus form widgets. The comparison response did not mention it.

Main-branch source inspection could not recover that branch-only work. This is the kind of cross-session continuity the product is intended to provide. It is also uncontrolled, post-hoc evidence: the branch was not inspected during the treatment, the session was not a registered endpoint, and these numbers are not pooled with the controlled results.

Read the complete report and evidence boundary