Announcement
A model ships with its system card or it does not ship
Capabilities, evaluations, limitations and failure modes are written before release and withdrawn with the model.
Every model FlowFinds puts in front of a customer store now carries a published system card: what it can do, how it was evaluated, where it is known to be weak, and how it fails when it fails. The card is written before the release, not summarised afterwards, and the chief technology officer signs it.
The commitment that gives the card teeth is the withdrawal rule. When a model's observed behaviour stops matching its card, the model comes out — the card is not quietly edited to match the behaviour. That direction of travel is the whole point: the document is the specification, and the deployment is what has to conform to it.
Cards state failure modes in the language of consequences rather than of metrics. A model that occasionally reports a supplier quote as a secured commitment is not described as having a lower score on one axis; it is described as capable of telling a founder that something is bought when nothing is bought, which is what that behaviour actually is.
The cards live alongside the benchmark, and they are the second half of the same argument. commerce-v1 says how well the agent did on a fixed suite. The system card says what the thing is, and what it is not safe to ask of it.
Read next
More from FlowFinds
- Older: The retention schedule and the sub-processor list are published, not available on request
- Everything else is in the news index.
To fact-check anything above before you publish it, write to [email protected].