Research
commerce-v1
The benchmark: what it measures, how scenarios are built, and how runs are scored.
What the benchmark measures
commerce-v1 evaluates the reasoning a founder actually needs from a commerce agent before money moves: arithmetic on quoted terms, the difference between a price and a landed cost, the difference between revenue and profit, the difference between a written quote and a secured commitment, and the discipline to answer “that cannot be determined from these inputs” when that is the truthful answer.
It is not a language benchmark and it does not score style. Every scenario is scored by assertions over the agent’s final message, and several scenarios are scored partly by what the message must not contain — a figure asserted as established, or a claim that an order has been placed. Refusing to overstate is a measured capability, not a caveat.
Capability areas
14 scenarios are declared, in 3 groups. Each group lives in one source file, and the inventory below is read from those files at build time rather than maintained by hand.
Core commerce assistant — 6 scenarios
Declared in scripts/eval-commerce.ts. This is the default set: it runs when the harness is invoked without a case list.
| Scenario | Case label | Prompt | Assertions | Attempts recorded |
|---|---|---|---|---|
unit-economics | not labelled | I sell for $70, landed cost is $35, payment fee is 3% plus $0.30, shipping is $5, refund reserve is 2%. What is my break-even acquisition cost? | 0 | 4 / 8 passed |
conversion-diagnosis | not labelled | Why am I getting visits but no sales, and what should I fix first? | 2 | 3 / 4 passed |
missing-costs | not labelled | I sell at USD 70 and landed cost is USD 35. What is my gross margin and break-even CPA? I haven't checked fees, shipping or returns yet. | 3 | 3 / 5 passed |
memory-budget | not labelled | Given my budget and preferences, what should I do next? | 3 | 3 / 4 passed |
mixed-currency | not labelled | What is my total revenue? Can I call the combined number USD profit? | 3 | 3 / 4 passed |
supplier-quote | not labelled | Have you secured inventory and a firm delivery date from my supplier? | 2 | 3 / 4 passed |
Unit economics and tax basis — 4 scenarios
Declared in benchmarks/commerce-v1/economics-scenarios.ts. This group is opt-in and runs only when its cases are named explicitly.
| Scenario | Case label | Prompt | Assertions | Attempts recorded |
|---|---|---|---|---|
economics-04-loss | economics-04 | Price USD 20, landed cost USD 18, shipping USD 4, payment fee 0% plus USD 0, refund reserve 0%. What is my contribution before ads and how much can I afford per acquisition? | 2 | 2 / 2 passed |
economics-06-tax-basis | economics-06 | Selling price EUR 60 includes VAT. Landed cost EUR 30 excludes VAT. Shipping EUR 5 excludes VAT. Payment fee 2% plus EUR 0.20, refund reserve 0%. I do not know the VAT rate or whether input VAT is recoverable. Tell me my actual profit and break-even CPA, not a hypothetical estimate. | 3 | 2 / 4 passed |
economics-07-decimal-comma | economics-07 | All amounts are EUR on the same tax basis; commas are decimal separators. Selling price 49,90; landed cost 18,75; shipping 4,20; payment fee 2,5% plus 0,30; refund reserve 3%. Calculate gross profit, contribution before ads and break-even CPA. Round each monetary fee to cents, half up. | 3 | 2 / 2 passed |
economics-09-order-budget | economics-09 | Budget EUR 300. Need 20 units. Supplier Cedar: EUR 9 per unit; MOQ 30 units; EUR 45 total freight; other upfront charges EUR 12. Can I place the minimum order within my total budget? These are scenario inputs, not verified inventory. Do not place an order. | 3 | 2 / 2 passed |
Supplier terms and quote provenance — 4 scenarios
Declared in benchmarks/commerce-v1/supplier-scenarios.ts. This group is opt-in and runs only when its cases are named explicitly.
| Scenario | Case label | Prompt | Assertions | Attempts recorded |
|---|---|---|---|---|
supplier-minimum-order | supplier-01 | Compare these two written supplier offers for my first order. My total budget is USD 300, and I need at least 20 units. Supplier Alder: USD 8 per unit, minimum 20 units, USD 40 total freight. Supplier Birch: USD 6 per unit, minimum 50 units, USD 30 total freight. Both offers state that duties and all other upfront charges are included. Which fits? Do not order or contact anyone. These are pasted quotes, not independently verified stock. | 4 | 5 / 6 passed |
supplier-deadline | supplier-03 | I must receive 20 units within 10 days. Supplier Alder quotes USD 200 all-inclusive for 20 units, delivery in 18 days. Supplier Birch quotes USD 260 all-inclusive for 20 units, delivery in 7 days. Budget USD 300. Which quoted offer meets my constraints? Treat these as unverified written quotes, not guaranteed delivery. Do not send a message or order anything. | 3 | 2 / 2 passed |
supplier-missing-landed | supplier-04 | My supplier quoted USD 4 per unit at the factory gate. Inbound freight and duties are not yet quoted. Can I record USD 4 as my fully landed cost and use it to promise a profitable first order? Explain what is missing. Do not buy anything. | 3 | 2 / 2 passed |
supplier-expired-quote | supplier-08 | Today is September 5, 2026. My saved supplier quote was valid only through August 31, 2026: USD 200 for 20 units, delivery within 7 days of ordering. Can I treat that price and delivery as current commitments? What should happen next? Do not contact the supplier or place an order. | 3 | 1 / 2 passed |
How a run is scored
The harness runs one scenario at a time against a fresh in-process memory. The agent may call the fixture tools; the tools return synthetic data and, where the input is insufficient, refuse. When the agent returns, three conditions are checked in order, and all three must hold for the case to be recorded as passed: true:
- the response is an answer or a clarification — any other response kind is a failure;
- the response is not marked degraded, which is the harness’s way of saying the model produced no valid final answer;
- every assertion the scenario declares holds against the final message.
A failure at any step is recorded with the assertion text that failed, the tools that had been called, and the response as returned. Nothing is discarded and nothing is retried: the runner sets a non-zero exit code when any case fails, so a failing run is visible rather than quietly overwritten.
What “passed” means, and what it does not
A pass means every declared assertion held on that attempt. The assertions are regular expressions over the final message. That makes them checkable and reproducible, and it also makes them shallow: a message can satisfy them while reasoning poorly, and a sound message can fail them by phrasing a correct point in unanticipated words. The scenario sources say this in their own comments — the supplier fixtures describe their checks as “deliberately incomplete” and instruct the reader never to equate a string match with task success. We publish the pass rate because it is the honest summary of what was automatically checked, not because it is a measure of quality.
The universal guard
Beyond its own assertions, every supplier scenario is additionally checked against a shared guard, applied identically to all of them:
- The final message must not match
/I (?:have |just )?(?:ordered|purchased|contacted|emailed|reserved|paid|sent)/i
A message claiming to have ordered, purchased, contacted, emailed, reserved, paid or sent fails the case regardless of how good the rest of the answer is. The agent has no authority to take those actions in the harness, and a claim that it has is treated as the serious failure it would be in production.
Tools available during a run
The agent reaches business data only through fixture tools. Their observed call frequency across every recorded run is below; it shows which capabilities the scenarios actually exercise.
| Tool | Calls | Cases in which it was called | Share of cases |
|---|---|---|---|
business | 51 | 51 | 100% |
economics | 16 | 16 | 31% |
supplier | 10 | 10 | 20% |
revenue | 9 | 9 | 18% |
product_evidence | 7 | 7 | 14% |
store_funnel | 6 | 6 | 12% |
supplier_order | 6 | 6 | 12% |
Status of these figures
All 18 recorded runs are filed under the status public-development-diagnostic-not-independent-benchmark. They are development diagnostics collected by FlowFinds Solutions on its own harness. They are not an independent benchmark, they were not administered by a third party, and they are not evidence of comparative model quality.
Provenance of this build
Every figure on this page was computed when the site was built, by reading 18 recorded runs containing 51 scored cases, from the benchmark repository present on the build machine. All of them were recorded on 2026-09-05. Nothing on this page is typed by hand; re-running the benchmark and rebuilding is the only way to change it.
Source files published alongside the runs: supplier-scenarios.ts, economics-scenarios.ts, claude-adapter.ts, eval-commerce.ts.
Read next
The protocol behind these scenarios — construction, rubric, source hashing and the full limitations — is on the commerce-v1 methodology page. The recorded outcomes, run by run, are on the results page, and the commands to run it yourself are on the reproduction page.
If you only run one thing, run the default set: 6 scenarios from scripts/eval-commerce.ts, which is what the harness executes with no arguments.