AI Workflow Eval Scorecard
Use before a material prompt, model or workflow change, and at a monitoring cadence justified by risk and change rate.
Workflow
- Workflow name:
- Owner:
- Current production version:
- Candidate version:
- Eval date:
- Reviewer:
Dataset
- Number of examples:
- Easy cases:
- Hard cases:
- Edge cases:
- Known previous failures:
- Dataset/rubric versions and approved data location:
- Tuning cases separated from held-out cases:
- Important classes missing or underrepresented:
- Last dataset review:
Scorecard
| Dimension | Threshold | Current | Candidate | Pass? | Notes |
|---|---|---|---|---|---|
| Correctness | Define before run | ||||
| Blocking policy failures | Define prohibited observed failures | ||||
| Format/schema validity | Define before run | ||||
| Usefulness | Define rubric and threshold | ||||
| Regression cases | Specify mandatory cases |
Failure Review
| Case ID | Failure type | Current behavior | Candidate behavior | Fix or decision |
|---|---|---|---|---|
Release Rule
- Blocking dimensions:
- Allowed tolerance:
- Required approver:
- Post-release check:
Decision
- Ship:
- Hold:
- Revise:
- Roll back:
- Reason:
Slice Review And Limits
- Critical slice results shown separately from the overall average:
- False accepts and false rejects:
- Repeated-run variability or uncertainty:
- Judge/reference disagreements investigated:
- What the sample does not establish:
A higher overall score does not override a failed blocking requirement. Zero failures observed does not establish zero risk.