Pharos Production / Review brief
AI Software Brief: Define Evaluation Before Integration
An AI feature needs a useful task and a way to recognize failure. Write both before selecting a model or connecting an agent to business systems.
Original editorial worksheet · Updated
- Use it for
- Scope AI agents, retrieval-augmented generation, document processing or an applied machine-learning feature.
- Output
- A blank, editable record with evidence fields and ownership.
- Tool behavior
- The CSV is a static download. No project details are requested or submitted.
Set the boundary before comparing
Describe the input, the intended output and the action a person takes next. A retrieval system needs a defined source collection and access rules. A tool-using agent also needs a list of permitted effects. Computer vision, language processing and document extraction need task-specific examples; a polished demonstration is not an evaluation set.
Keep retrieval quality, generated-answer quality and action execution as separate review items. A correct source passage does not establish that the answer uses it correctly. A plausible answer does not establish that a requested action was authorized or completed.
The evidence record
Use one row per review area. The role is a starting point; assign a named owner in your project. Attach a sanitized reference and retain unresolved questions until a reviewer has a supported answer.
| Review area | Evidence to attach | Suggested role |
|---|---|---|
| Task | One workflow and its current baseline | Product owner |
| Data | Source rights, access boundary and refresh rule | Data owner |
| Evaluation | Representative cases and labeled failure conditions | Evaluation owner |
| Actions | Allowed tools, approval boundary and duplicate-effect rule | System owner |
| Operations | Latency, usage limits, cost records and failure response | Operations owner |
| Change control | Model, prompt, retrieval and evaluation versions | Release owner |
The CSV adds assigned owner, evidence reference, status, unresolved question and review date columns. Every row starts as draft.
A fictional example
Suppose a fictional support assistant can draft a refund recommendation. Evaluation should distinguish an unsupported policy answer from an unauthorized refund action. If the project includes retrieval but excludes payment execution, write that exclusion into the brief and the acceptance cases.
Use the record in a review
- Collect representative examples, including cases that should be refused or escalated.
- Specify which data each workflow may access and which effects it may trigger.
- Agree how failures will be labeled and who reviews changed behavior.
- Record operational limits before comparing a managed API with a self-hosted deployment.
Download the blank CSV and keep the completed version with your project records.
Source and limits
The blank record is a scoping aid. It contains no model benchmark, security test or claim of accuracy. Budget limits and quality thresholds must come from the project.
NIST AI Risk Management Framework. Background for organizing AI risk management. The fields and sequence on this page are this site's own planning method.
Source reviewed 9 November 2026. The suggested fields are this site's editorial method, not an external certification standard.