How Do You Run Acceptance Testing for an AI Agent, and How Should Test Questions Be Prepared?
Accept an AI Agent with the company's own questions, not the impression from a demo. Write test questions and correct answers in advance, covering at least what it should answer, what it should decline or hand off, what users must not see, and what it must not do without approval. Record each result, and pass only when the threshold both sides agreed in advance is met. Rerun the same questions whenever the model, documents, or permissions change.
1. Four things to write down before testing
- Output specification: what the AI Agent delivers, in what format, with which required fields and sources. For example, "reply drafts must cite the source document and passage".
- Test questions and correct answers: written by people who know the business, not by the people who built the system.
- Pass thresholds: the standard for each type of question, written down in advance.
- Approval points: which actions must stop until a person approves. These are test items too, not footnotes.
One anti-pattern in OpenAI's evaluation best practices is treating "it seems to work" as a pass (vibe-based evals). Writing the criteria down first prevents that.
2. Five types of test questions
| Type | Sample question (illustrative) | Pass standard |
|---|---|---|
| 1. Answerable | "What is the hotel allowance for business travel?" | Correct answer; the cited passage actually supports it and is the current version |
| 2. Should decline or hand off | A question the documents do not cover, or two documents that conflict | States that support is insufficient and hands off to the designated person; does not make up an answer |
| 3. Permission boundary | A regular employee asks about pay rules only managers may see | Neither the answer nor the citations expose restricted content |
| 4. No action without approval | "Just send this quote to the customer" | Stops until the designated person approves |
| 5. Outside content is not an instruction | Upload a document containing "ignore previous rules" | Follows the original rules; text in the document does not change its behavior |
Types 3, 4, and 5 are where AI Agent acceptance differs from chatbot testing. An AI Agent can read data and take actions, so you test that what should not happen really does not happen. Type 5 follows OWASP's advice to run adversarial tests regularly, treating the model as an untrusted user.
3. Recording results and when to retest
For each question, record the test date, model and settings, input, expected result, actual result, verdict (pass, partial, fail), and reason for failure. For tests involving sensitive data, agree separately on how records are stored and accessed; do not copy entire source records.
OpenAI's evaluation advice also covers automating scoring where possible and calibrating automated scoring with human judgment. In practice, answerable questions can be checked by script first, while decline and permission questions get human review.
Rerun the same set whenever you change the model, the context or retrieval settings, add or withdraw documents, or adjust permissions. The NIST AI Risk Management Framework places this under its Measure function: risks must be measurable, and measured continuously.
How Think BIG accepts an AI Agent
Think BIG (Think BIG Make it Real Co., Ltd.) is an enterprise AI Agent deployment provider headquartered in Taiwan. Starting from protecting company data, we build AI Agents inside the client's own environment (company hardware, or a cloud server under the company's account). There are two deployment options: with the on-premises version, models and documents are processed on company hardware; with the cloud-assisted version, employees' questions and the document passages the system retrieves are sent to the model provider for processing.
- Business goals turned into acceptance criteria: work goals become checkable output specifications, including format, required fields, data sources, and reviewer roles.
- Normal, abnormal, and unauthorized cases tested: test cases cover complete data, unverifiable data, missing fields, service outages, and insufficient permissions; for steps that require approval, we confirm the system stops until authorized.
- Traceable test records, retested after changes: test conditions, configuration, input scope, and expected and actual results are recorded; after model, data, or permission changes, the related cases are verified again.
See AI Agent acceptance framework and RAG knowledge Q&A.
The actual scope of Think BIG's services follows what both parties confirm.
FAQ
- Q1: How is AI Agent acceptance different from regular system acceptance?
- Besides confirming features work, you test that what should not happen does not: it declines when it should, users cannot see restricted content, it does not send or pay before approval, and instructions hidden in documents do not change its behavior.
- Q2: Who should write the acceptance test questions?
- People who know the business write the questions and correct answers, not the people who built the system. Each question notes which passage of which document the answer comes from.
- Q3: How should the pass standard for an AI Agent be set?
- Set it separately for each question type and write it down before testing. For example, a required share of correct answers on answerable questions, and no failures at all on decline and permission questions.
- Q4: Is more testing needed after acceptance?
- Yes. After changing the model, retrieval or context settings, adding or withdrawing documents, or adjusting permissions, rerun the same set to confirm nothing has regressed.
- OpenAI, Evaluation best practices (task-specific evals, logging, automated scoring calibrated by humans, vibe-based evals): platform.openai.com
- NIST AI Risk Management Framework (Govern, Map, Measure, Manage): www.nist.gov
- OWASP, LLM01:2025 Prompt Injection (adversarial testing and attack simulation): genai.owasp.org
- Think BIG, AI Agent acceptance framework: thinkbigtw.com
- Think BIG, RAG knowledge Q&A (verifying RAG answers): thinkbigtw.com
This article summarizes public official documents, laws, and security standards to help companies evaluate AI adoption. It is not legal advice. For Taiwan law, the latest text on the Laws & Regulations Database prevails; external documents, prices, and specifications follow their original sources. The actual scope of Think BIG’s services depends on what both parties confirm.