The benchmark, in the open. Same agent, same model.
Claude Code + Opus 4.8, with and without the Code Lexica MCP server, measured against the full SWE-Bench Pro task suite. Model, harness, turn limit, and seeds held constant. The only variable: whether your agent has a persistent context layer.
The full benchmark run — cost-per-task, token, and pass-rate results, the reproducible harness, and per-task data — publishes shortly. The methodology we're holding ourselves to is below.
What we hold constant. What we change.
Benchmark credibility comes from isolating one variable. Here's exactly what changes between the baseline and the Code Lexica run.
| Model | Claude Opus 4.8 (claude-opus-4-8) |
| Agent harness | Claude Code · latest GA release |
| Task suite | SWE-Bench Pro (1,865 tasks, multi-language) |
| Trials per task | 3 attempts, best-of-N reported |
| Turn limit | 250 turns per task |
| Extended thinking | Off (matches typical headless usage) |
| Variables held constant | Model, harness version, turn limit, attempt count, hardware, seed strategy |
| Variable changed | Code Lexica MCP server connected vs. disconnected |
| Metrics captured | Pass rate · input tokens · output tokens · cache reads · cache writes · wall clock · cost (list API price) |
Why token reduction is suddenly load-bearing
June 2026's pricing shifts ended the agentic free lunch. Engineering teams that don't measure token spend are about to be surprised.
Headless agent runs start blind.
No memory of yesterday's run, no map of your repo. Every claude -p invocation re-discovers the architecture from scratch.
~70% of agent tokens are waste.
Stanford research found agents burn the majority of their tokens re-reading files, exploring dead ends, and snowballing context.
Pricing is shifting under your feet.
Anthropic moved programmatic Claude usage off subscription pools onto full API rates. GitHub Copilot moved agent mode to a metered pool. Subsidies are gone.
One bad cron can hit $1,500 in a day.
Without persistent context, there's no upper bound on how much an agent will spend rediscovering your codebase per task.
Why first-pass pass rates improve
Cheaper tokens are only half the story. Persistent context also makes the agent's first attempt more likely to be correct.
Relevance beats volume.
Stuffing the context window with grep'd file contents is a worse signal than a structured, scoped answer to the specific question the agent asked.
Persistent context catches load-bearing patterns.
Code Lexica knows which modules are critical-path, which patterns are deprecated, and which interfaces are stable. The agent gets architectural facts, not guesses.
Standards land before generation, not after PR review.
Org conventions, preferred libraries, and error-handling style ship into the prompt so the agent writes idiomatic code on the first try.
Evidence reduces hallucination.
Every Code Lexica response references exact files and functions. The agent stops inventing APIs and modules that don't exist.
How the context layer fits in
Your agent doesn't change. Code Lexica sits between the agent and your codebase, returning scoped context per request via MCP.
The agent asks for scoped context; the MCP layer returns structured, relevant answers. No bulk file dumps, no firehose.
Verify what we publish
A benchmark you can't reproduce isn't a benchmark — it's a marketing post. Here's what we ship alongside the numbers.
Per-task CSV
Raw data: every task, every trial, full token breakdown. Downloadable on this page.
Coming with final benchmarkRun it yourself
Public repo with the exact harness, seeds, and config we used. Reproduce or audit the numbers.
Coming with final benchmarkMethodology questions
What engineering leaders ask before trusting a benchmark.
See what Code Lexica saves on your codebase.
Two-week proof-of-value with instrumented token spend and first-pass quality. Most teams see measurable cost reduction in week one.