Worked example · anonymised and illustrative · the shape of an engagement, not a past result
Claims triage: moving motor claims to straight-through processing, with the eval set built first.
A claims operation where every first notification of loss is read by a handler before anything happens. The question is not whether a model can classify claims; it is which claims it may be trusted with, and how that trust is earned and measured.
Measurement plan · agreed before build, measured weekly
| Metric | Baseline | Target | How measured |
|---|---|---|---|
| Straight-through processing rate | Measured in discovery | Agreed at gate 2 | Claims closed without handler touch, 30-day trailing |
| Median time to first decision | Measured in discovery | Under one day for straight-through claims | FNOL timestamp to decision timestamp |
| Misrouted claims | Measured in discovery | No worse than baseline at every gate | Reassignment events per 1,000 claims |
| Fraud-flag precision and recall | Published at gate 2 (week 4) | Threshold fixed at week 6; changes need an eval run | Held-out eval set labelled by the insurer |
| Inference cost per claim | — | Ceiling in the SOW; reviewed fortnightly | Token spend ÷ claims processed, incl. retries |
| Customer complaint rate | Measured in discovery | No worse than baseline | FCA-reportable complaints ÷ claims |
Every number above is agreed in writing before build and measured weekly. Eval harness, prompts and cost model live in the insurer's repository; threshold changes after week 6 require a documented eval run.
The situation this is written for
A motor book producing thousands of first notifications of loss a month, each read by a handler before anything happens. Often a previous vendor pilot has classified claims impressively on a demo set and been switched off after a few weeks in production because nobody could explain which claims it got wrong.
How we would run it
The first two weeks are discovery. Weeks three and four build the evaluation set, not the model: historical claims labelled by the insurer’s own senior handlers, with disagreements recorded rather than resolved. Baseline metrics and the cost model are published to the claims director before a model is chosen. That baseline is gate 2, at week four.
The triage layer is a classifier with an in-house secondary model for fraud signals. Every decision carries the evidence it used, in the handler’s language, and a confidence band. Claims below the band go to a handler with the model’s reasoning attached; claims above it are processed straight through. The band is tightened at fixed gates, each time with an eval run attached to the pull request, and never between them.
What you would be left with
The eval harness, prompts and cost model in your repository, a runbook for re-labelling, and a drift alert that fires when the live distribution moves away from the eval set. Your engineers run the system alone for the last fortnight while we shadow on call, and the exit review says in writing what we would do differently.
Start with a two-week discovery sprint.
Same people scope and build. Numbers agreed up front, measured weekly, published at exit.