The paired A/B test is how you verify savings on your own code instead of trusting a marketing page. It runs the same task from the same base commit in two isolated worktrees: one arm with Code Lexica, one without, holding the agent and tooling constant.
Running a test
Describe a real task from your backlog and let both arms run:
- cdlx ab-test run --task "Fix the flaky retry logic in the payments worker"
Make it fair
Give the test a verify command (tests, build, lint) the task must pass. It keeps both arms honest and gives the scorecard its quality signal.
Reading the scorecard
Each arm reports:
- Tokens and cost, priced from one rate table so API-key and subscription usage compare fairly
- Duration
- Verify-command pass rate, so you can see quality held while spend dropped
Where results land
Measured results appear on your Savings dashboard, so the people who approve the budget can see them without touching a terminal.