Does repairing fewer fields help?
A typed extraction can contain valid fields alongside unresolved ones. We compared regenerating the whole frame with regenerating only the failed fields, using the same initial outputs and a maximum of one repair call.
No accuracy advantage established.
Selective retry matched 29 of 80 expected actions; full-frame retry matched 31. Selective retry remains an optional mechanism.
| Method | Correct | Wrong proposals | Blocked | Calls* |
|---|---|---|---|---|
| Simple extraction | 28 | 10 | 30 | 68 |
| Typed extraction | 25 | 5 | 38 | 68 |
| Full-frame retry | 31 | 10 | 27 | 106 |
| Selective retry | 29 | 11 | 28 | 106 |
Twelve cases had unavailable source histories and count as unsuccessful for every method. *Calls are attributed per method; the three typed methods share their initial outputs. The study made 212 actual calls.
What was measured
The scorer requires an exact match of the tool name and all JSON arguments, including value types. It scores proposed actions; no business tool executes. Every method receives the target tool schema and the referenced source sessions.
The primary comparison was selective minus full-frame retry accuracy: −2.5 percentage points, with a source-group bootstrap 95% percentile interval of −6.85 to 0.00. This interval describes one captured model run, not variation across repeated generations.
What the failures revealed
Initial validation rejected 65 fields across 38 cases: 34 unresolved values, 28 invalid uses of defaults, and three unsupported source quotes. Only two cases differed in correctness between the repair methods. One concerned an optional date range. The other was a citation-format failure from independent generations of an identical request.
Selective retry used 22.2% fewer output tokens, with essentially unchanged input tokens and call count. This does not establish lower total cost at matched accuracy.
Limits of the comparison
- The 80 cases form a development partition. The 320 held-out cases were not used.
- Source sessions and the target tool were supplied. This is not full-corpus retrieval or an official leaderboard reproduction.
- The public dataset is synthetic. It does not annotate dependency graphs, interactive clarification quality, or executed outcomes.
- Quote checks establish source occurrence, not semantic entailment.
- Offline replay reproduced every prediction and summary metric. This verifies replay, not an independent model replication.
Sources and scope
Data comes from Mem2ActBench at its pinned revision. The downloadable summary includes aggregate results, dataset hashes, and the comparison settings. Reports use the optional Multivon evaluation integration.
These results do not establish state-of-the-art performance or production readiness. The evaluated recovery strategy is experimental.