Reproduce our results

Every figure published in this section came from the command below, run on a developer machine, writing a JSON artifact per run. Nothing else produced them. This page gives you the same commands.

1. Get the code

The benchmark lives in the FlowFinds application repository, alongside the agent it measures — deliberately, so the rubric and the implementation cannot drift apart. That repository is not yet public. Until it is, ask for a read-only checkout at [email protected]; we will not pretend a clone URL exists that does not.

What we do publish in full, today, without asking anyone: the scenario definitions, the model adapter, the runner, and every raw run artifact. Those are the four things you need to audit the numbers even without executing anything, and they are linked at the bottom of this page.

Once you have a checkout, the layout the commands assume is:

flowfinds-organ/
├─ scripts/eval-commerce.ts                  the runner
├─ benchmarks/commerce-v1/
│  ├─ supplier-scenarios.ts                  4 scenarios
│  ├─ economics-scenarios.ts                 4 scenarios
│  └─ claude-adapter.ts                      the Anthropic diagnostic adapter
└─ artifacts/commerce-reasoning-v1/          one directory per run, written by the runner

2. Install

The project pins its dependency tree in package-lock.json. Install it exactly rather than resolving afresh, or you are not running what we ran.

cd flowfinds-organ
npm ci

3. Run the default set

With no arguments the runner executes the default set — 6 scenarios declared in scripts/eval-commerce.ts — against the default provider.

npx tsx scripts/eval-commerce.ts

The runner prints one JSON line per case as it goes, and writes the artifact after every case to a freshly created directory under artifacts/commerce-reasoning-v1/. Its first line of output names that directory. The process exits non-zero if any case fails, so this is safe to put in CI without it silently passing.

4. Choose the provider and model

Provider and model are set through the environment. The accepted providers are openai and anthropic; anything else is refused rather than defaulted. The Anthropic path additionally requires an explicit model identifier — the adapter refuses to run without one, so that a run can never be attributed to a model nobody named.

# the default provider, using the configured model
COMMERCE_EVAL_PROVIDER=openai npx tsx scripts/eval-commerce.ts

# the Anthropic diagnostic adapter; the model identifier is mandatory
COMMERCE_EVAL_PROVIDER=anthropic \
COMMERCE_EVAL_MODEL=<model identifier> \
  npx tsx scripts/eval-commerce.ts

5. Run specific scenarios

The opt-in groups do not run by default. Name the cases you want as a comma-separated list. An unknown identifier is rejected, and an empty selection is refused, so a run cannot pass by scoring nothing.

COMMERCE_EVAL_CASE=economics-04-loss,economics-06-tax-basis,supplier-minimum-order,supplier-deadline \
  npx tsx scripts/eval-commerce.ts

The 14 declared case identifiers in full:

unit-economics
conversion-diagnosis
missing-costs
memory-budget
mixed-currency
supplier-quote
economics-04-loss
economics-06-tax-basis
economics-07-decimal-comma
economics-09-order-budget
supplier-minimum-order
supplier-deadline
supplier-missing-landed
supplier-expired-quote

6. Check that you ran the same code we did

Every artifact records the SHA-256 digest of the files that determine its result. Compare your checkout against the digests in any run we publish; if they differ, the results are not directly comparable, and the difference is in the files below rather than in the model.

shasum -a 256 \
  lib/commerce/agent.ts \
  lib/commerce/economics.ts \
  lib/commerce/economics-provenance.ts \
  lib/intent/codex-cli.ts \
  scripts/eval-commerce.ts \
  benchmarks/commerce-v1/claude-adapter.ts \
  benchmarks/commerce-v1/supplier-scenarios.ts \
  benchmarks/commerce-v1/economics-scenarios.ts \
  lib/commerce/supplier-order.ts \
  lib/commerce/tools.ts

Then read sourceHashes out of the artifact you are comparing against:

# from a published artifact
curl -s https://example.invalid/research/commerce-v1/runs/diagnostic-FU1SXM.json | \
  node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>console.log(JSON.parse(s).sourceHashes))'

# from a run you produced yourself
node -e 'console.log(require("./artifacts/commerce-reasoning-v1/<run id>/model-eval.json").sourceHashes)'

Substitute your own host for example.invalid, or simply download the artifact from the results page, where every published run is a direct link.

7. What you should expect to see

You should not expect to reproduce our pass rates case for case. Model outputs are not deterministic, each case is attempted once per run, and the assertions are regular expressions over free text. What you should be able to reproduce is the structure: the same scenarios, the same assertions, the same artifact shape, and pass or fail decisions that are checkable line by line against the failure text the runner records. If a case passes for us and fails for you, the artifact tells you which assertion parted, and that is a conversation about the rubric rather than about anyone’s word.

Nothing in the harness can spend money on your behalf, contact a supplier or place an order. The tool layer is synthetic fixtures; it has no capability to act. Model inference itself is billed by your provider in the ordinary way.