The cheapest model call is the one you avoid
How deterministic retrieval, reusable semantic context, and smaller models can reserve frontier-model attention for the work that needs it.
- Most repository lookups should not require a model.
- Smaller models are useful for bounded summaries, metadata, and candidate ranking when they meet the contract for that role.
- Generating semantic context once lets many later sessions reuse the same work.
- Local inference still consumes hardware capacity, electricity, memory, and time.
- Frontier agents should receive focused context, solve the task, and verify the result against current source and runtime evidence.
On this page
In A code graph is not repository memory, I wrote about why structural relationships are not enough for most of the questions I ask a coding agent.
Summaries and metadata add some of the missing meaning. They can describe what a file is for, which responsibility it owns, and how it relates to the task in front of the agent.
That creates another question. If a frontier model has to interpret every file and rank every lookup, the system has only moved the cost.
Most lookups should not need a model
Exact paths, identifiers, direct text matches, and strong symbol matches should take a fast deterministic path. A model should not be asked to rediscover an answer that the repository has already made explicit.
Semantic work becomes useful when the request is about purpose, ownership, or an ambiguous set of candidates. Even then, the job is usually bounded. A model may need to summarize one changed file, extract structured metadata, or reorder a limited list of plausible files.
Use a model only when deterministic evidence cannot settle the lookup.
Reuse avoids repeated semantic work
I generate semantic context once and reuse it across later sessions.
Without repository memory, each new frontier-agent session may rediscover what the same files do. The agent searches, opens several candidates, follows nearby code, and rebuilds a working picture that disappears when the session ends.
With RepoMemory, a smaller local or inexpensive model can build that semantic context during indexing, update it when the file changes, and make it available to many later sessions.
deterministic tools find clear evidence
cheaper models build reusable semantic context
frontier agents apply that context to the current problem
source and runtime checks verify the resultRepoMemory uses smaller models to prepare reusable context so the frontier agent does not have to rediscover the repository during every session.
Local inference still has a cost
I prefer local models for much of this work because the source stays on hardware I control. Local inference also gives me more control over availability, model selection, and when background indexing runs.
Local inference replaces the API bill with electricity, hardware depreciation, memory use, heat, and latency. Keeping a large model loaded so it can rank every lookup may cost more than using a small hosted model. Running any model for a question that a text search already answered wastes resources regardless of where inference occurs.
For this workload, placement matters less than whether inference is necessary and correctly sized.
For RepoMemory, that means updating summaries only when files change, reusing the results, skipping model calls when deterministic evidence is strong, and choosing the least expensive model that has proved reliable for each role.
Cheap output still needs a quality floor
A model that writes convincing summaries may still fail as a ranker. Ranking requires it to preserve the supplied candidates, follow a strict response contract, and keep the right file in the result.
A cheap model that returns malformed metadata, drops the correct candidate, or confidently describes the wrong responsibility creates more work for the frontier agent. The recovery can cost more than the original lookup would have.
I test summaries, metadata, and ranking as separate jobs. For each one, I choose the least expensive model that has demonstrated reliable performance.
Where the frontier model belongs
The frontier model should receive a focused set of relevant files and useful context. It can then apply the current task, inspect the evidence, and decide what needs to happen next.
That is the work I want its attention spent on: resolving ambiguity, tracing a behavior across boundaries, reviewing risk, and checking whether the proposed change is actually correct.
Repository memory prepares the starting point. Current source, tests, and runtime behavior decide whether its context was correct.
In practice, avoiding repeated retrieval work leaves the frontier model for the cases that still require source inspection and judgment.