Reasoning configuration comparison
- Task and known baseline failure:
- Approved data, tool and action boundary:
- Baseline model/configuration:
- Candidate model/configuration:
- Provider documentation checked and date:
- Prompt version shared by both:
- Required evidence and deterministic checks:
- Pass criteria, latency target and cost limit chosen before the run:
| Case | Expected answer or evidence | Baseline result / latency / cost | Candidate result / latency / cost |
|---|---|---|---|
| Normal | |||
| Boundary | |||
| Missing input | |||
| Contradictory input |
- Timeouts and retries included:
- Repeated trials and observed variability:
- Remaining failures:
- Selected route and observable escalation trigger:
- Fallback when no candidate passes:
- Change that requires retesting:
Model confidence and a long explanation are not validation. Keep author-written expected answers separate from observed model results.