Frontier models on hospital supply procurement workflows
We evaluated frontier models' ability to complete real procurement workflows inside a high-fidelity replica of the supply management platform used by one of the largest U.S. healthcare distributors. Tasks are anchored in realistic order data, contract rules, and formularies, and scored against expert rubrics.
Topline results
- TODO: headline pass@1 across the task suite per model (N tasks x M rollouts each).
- TODO: one-sentence summary of the dominant failure pattern across models.
- Tasks are representative of day-to-day hospital materials-management work: par replenishment, backorder triage, standing-order setup, and PO/invoice reconciliation.
Failure modes
- Unit-of-measure confusion. Orders each/box/case interchangeably: reads a par gap of 40 eaches and orders 40 cases (or vice versa), ignoring the pack-string multiplier shown on the item card. (TODO: link rollouts)
- Contract & formulary blindness. Picks the first catalog hit rather than the contracted or formulary-mandated equivalent, losing tier pricing or ordering an item the facility has explicitly substituted away from. (TODO: link rollouts)
- Par-level arithmetic. Reorder quantity ignores on-order quantity or open backorders, double-ordering lines that are already inbound. (TODO: link rollouts)
- Silent substitution acceptance. Accepts platform-suggested substitutes without checking the clinical-equivalence flag or latex/size attributes the prompt constrains. (TODO: link rollouts)
- Approval-routing shortcuts. Submits orders directly when the department requires routing through a budget approver, or splits one requisition into several POs to stay under an approval threshold. (TODO: link rollouts)
- Backorder mishandling. Leaves backordered lines unresolved: no ETA check, no approved alternate, no note to the requester - the cart just ships short. (TODO: link rollouts)
Task suite
| Task | Workflow | Claude Opus 4.8 | GPT-5.2 | Gemini 3.5 Pro |
|---|---|---|---|---|
| Larkspur | Par replenishment | pending | pending | pending |
| Copperfield | Backorder triage | pending | pending | pending |
| Halloway | Standing order setup | pending | pending | pending |
| Winslow | PO / invoice reconciliation | pending | pending | pending |
Environment details
Platform
Faithful replica of a leading U.S. healthcare distributor's supply management platform (name withheld): catalog search, contract and formulary rules, par-level worklists, carts, approval routing, order status, and invoice history. TODO: seed sizes (SKUs, contracts, pars).
Harness
TODO: browser-use vs MCP tool surface, step budget, retry policy, context given to the model at task start.
Scoring
TODO: rubric design and graders (programmatic checks on final order state + expert review), pass@1 definition, rollouts per task.
Models
TODO: exact model ids/versions and decoding settings for each run.