Guide
How to evaluate an AI contract review API against a supplied playbook
Write down the playbook rules, plant known deviations in real agreements, and score each tool on whether it finds them, quotes the right clause, and avoids invented issues. Count misses and false findings separately.
For legal operations and in house counsel. Written by the Encevra engine team. Reviewed 2026-10-12.
Build the case set
Take 20 to 50 agreements you already reviewed. For each, record the deviations a lawyer found and the clause they sit in. That record is your answer key, and it is more useful than any vendor benchmark because it reflects your playbook.
What a single test case looks like
A test case is one playbook plus one agreement, sent as the question and material in a single run. The example below is an authored sample from Encevra's own docs, shown so you can see the shape of a request; it is not a recorded output and should not be read as a result.
When you build your own answer key, write down the deviations and the clause they sit in before you run anything, the way this playbook names a 3 year term cap, mutual obligations, a liability cap and a governing law.
Authored input example, not a recorded engine run
Question sent to the engine
Review this NDA against our playbook.
Material sent with it
PLAYBOOK: Term max 3 years. Mutual obligations only. Liability cap required. Governing law: New York. NDA (Acme and Brightline, v2): 1. Confidential Information means all information disclosed by Acme. 2. Recipient (Brightline) shall protect Confidential Information for 7 years. 3. Brightline shall indemnify Acme for any losses without limit. 4. This Agreement is governed by the laws of Delaware. 5. Section 6 governs return of materials. (No Section 6 is included.) 6a. Term: this Agreement ends after 2 years.
Review tasks a run like this is expected to cover
- Does the confidentiality term exceed the playbook's maximum?
- Are obligations mutual, as the playbook requires, or one-sided?
- Is there an uncapped indemnity where the playbook requires a liability cap?
- Does the governing law clause match the playbook's required jurisdiction?
- Is a clause the agreement references actually included, or missing?
Score what matters
Score separately: deviations found, clause references correct, deviations invented, and rules the tool says it cannot check. A tool that misses less but invents more is not automatically better.
- Recall against the answer key
- Clause location accuracy
- Unsupported findings per agreement
- Time and cost per agreement
Evaluation criteria checklist
- Does every finding quote a specific passage, and does that passage actually exist at the stated location?
- Does the tool say what it could not check, rather than staying silent on gaps?
- What happens on a run that cannot be completed cleanly: is that distinguished from a finished, clean result?
- How long does a run take end to end, and does that fit how your team works?
- What does a run cost, and is that cost only charged when a usable result comes back?
Include a simple baseline
Give a strong single model the same agreement, the same playbook and a competent prompt. If the specialised workflow does not beat it on your set at a similar budget, you have learned something valuable.
Limits to account for while testing
- Runs are asynchronous and can take minutes; a synchronous call will wait, but budget for roughly 1.5 to 3 minutes on the full panel per agreement during evaluation
- A single run accepts up to 150,000 characters of material on the full Noetis engine, or 40,000 characters on the lite engine; split larger agreements before submitting
- Under self-serve billing only released runs are charged; a run that finishes incomplete is not charged, so an evaluation of mostly incomplete runs costs time, not money, but still tells you something about the tool
- Live access to run real evaluation cases is approval-gated, not available by default on a new account
- If you plan to show results from this API to your own clients or customers rather than use it only internally, that use needs a separate signed licensing agreement
Turning a playbook into test cases
Take each playbook position, such as a liability cap or a notice period, and write one contract excerpt that follows it and one that breaks it. Score whether the engine flags the broken one and quotes the right clause.
Frequently asked questions
- How many contracts do I need?
- Twenty to fifty representative tasks is a practical starting set for a scoped evaluation.
- Should a lawyer write the answer key?
- Yes. The answer key needs a qualified reviewer, written before any system is run.
- Does a located quote mean the legal reading is right?
- No. It proves the passage exists. Whether it supports the conclusion is a separate check.