Checklist
Integration evaluation checklist
Fix the task set, answer key and scorer first; run the candidate and a baseline under the same budget; record cost, latency, failures and losses.
For teams running a scoped evaluation. Written by the Encevra engine team. Reviewed 2026-10-12.
Before running
- 20 to 50 representative tasks from real work
- Answer key written by a qualified person
- Scoring rules fixed in writing
- A competent single model baseline prompt
- Budget per task agreed
While running
- Record engine and configuration version
- Record cost and latency per task
- Keep every failed and incomplete run
- Do not tune on the held out tasks
Reporting
- Wins and losses against the baseline
- Unsupported findings per task
- Cases needing human review
- What would change your decision
How to use this checklist
Copy it into your evaluation document, assign an owner to each line, and do not start runs until every 'Before running' line is done. Changing the tasks or scoring after seeing results makes the comparison unreliable.
Frequently asked questions
- How many tasks should an evaluation have?
- Twenty to fifty representative tasks is a practical start.
- Should failed runs count?
- Yes. Keep every failed and incomplete run in the results.
- Can Encevra run the evaluation with us?
- Yes. You can request a scoped evaluation.