Research
Research
How we measure a commerce agent, and everything we have measured so far.
What we measure
A commerce agent is asked to do things that are easy to describe and hard to verify: read a supplier’s written terms, work out whether an order fits a budget, say what a margin actually is once fees and freight are counted, and decline to assert a number the inputs do not support. Fluent prose about any of these is worthless. We therefore measure the agent the way a model is measured, on fixed scenarios with fixed pass conditions, and we publish the runs it fails alongside the runs it passes.
The benchmark is called commerce-v1. It currently declares 14 scenarios across 3 capability areas — core commerce assistant (6), unit economics and tax basis (4) and supplier terms and quote provenance (4). Each scenario is a single prompt with a recorded reference answer and a set of assertions applied to the agent’s final message. A case passes only when every assertion holds.
What has been recorded so far
As of this build there are 18 recorded runs, holding 51 scored cases across 4 provider and model pairings, all measured on 2026-09-05. Of those cases 37 passed, giving an aggregate pass rate of 72.5%; 12 produced no valid final answer at all and are counted as failures. The aggregate mixes models, scenario subsets and harness revisions, so it is a description of what has been run, not a score for any one system. The results page breaks it apart.
Status of these figures
All 18 recorded runs are filed under the status public-development-diagnostic-not-independent-benchmark. They are development diagnostics collected by FlowFinds Solutions on its own harness. They are not an independent benchmark, they were not administered by a third party, and they are not evidence of comparative model quality.
Provenance of this build
Every figure on this page was computed when the site was built, by reading 18 recorded runs containing 51 scored cases, from the benchmark repository present on the build machine. All of them were recorded on 2026-09-05. Nothing on this page is typed by hand; re-running the benchmark and rebuilding is the only way to change it.
Source files published alongside the runs: supplier-scenarios.ts, economics-scenarios.ts, claude-adapter.ts, eval-commerce.ts.
Why the runs are published
Every figure in this section is computed at build time from the recorded artifacts. Nothing is transcribed, so nothing can drift from the evidence, and a run that goes badly appears here as promptly as one that goes well. The raw artifacts are downloadable in full from the results page, the protocol is written out on the methodology page, and the commands that produced them are on the reproduction page. If our numbers are wrong, those three pages are enough to prove it.
What we do not claim
- These runs are not an independent evaluation. FlowFinds Solutions wrote the scenarios, wrote the assertions, and ran the harness.
- The scenarios are synthetic. No customer data, no real supplier, no real spend and no real order is involved in any recorded run.
- A pass means the assertions held on one attempt. It is not a statement about the model in general, and the scenario definitions say so themselves: the automatic checks are described in source as deliberately incomplete.
- Sample sizes are small and unequal between models. Differences of a few cases are not evidence of a difference in capability.
Everything in this section
The benchmark, its protocol, its results, how to run it yourself, and the system cards built from what the runs revealed.