AI agent assurance: an enterprise evidence checklist

AI agent assurance is the process of building and reviewing evidence that an agent can perform a defined task within agreed boundaries. It connects claims about the agent to test results, operating controls, limitations and an accountable decision about its use. The broader discipline of AI assurance includes measuring, evaluating and communicating system trustworthiness, as described in DSIT’s introduction to AI assurance.

For an enterprise preparing to deploy an agent, the practical question is: does the evidence support the authority we are about to give this system? This guide provides a checklist for answering that question, including a worked review example.

Key takeaways

• Review evidence against a specific deployment decision, including the actions and data the agent may access.
• Examine whether each important claim has relevant, inspectable evidence and whether contradictory findings remain unresolved.
• Separate task performance from permission enforcement and operational recovery. Each needs its own evidence.
• Record a bounded decision: approve the defined scope, restrict it, or hold deployment until a material gap is resolved.

What needs assurance when an agent can act

An agent may retrieve a document, choose a tool and update a business record in one workflow. A convincing final response tells a reviewer little about whether the preceding actions respected the user’s authority or reached the right destination.

OWASP identifies excessive functionality, permissions and autonomy as causes of excessive agency. Its recommended controls include limiting tool capabilities, enforcing authorization in downstream systems and requiring approval for high-impact actions. These are useful starting points for deciding which boundaries need evidence. OWASP’s excessive agency guidance explains those controls.

Security is one part of the review. The business owner also needs to know whether the agent completes useful work, recognizes unsuitable requests and leaves the process recoverable when a dependency fails. A system that consistently refuses legitimate tasks may protect a boundary while failing its business purpose.

Start with the proposed use. A read-only research assistant, an internal ticket-routing agent and an agent that makes external commitments warrant different review questions.

How assurance relates to testing and certification

These activities contribute different information to a deployment review.

Activity	Main question	What the reviewer should examine
Evaluation and testing	How did the system behave under specified conditions?	Cases, expected outcomes, observations and coverage gaps
Security assessment	Can a relevant boundary be crossed or a capability misused?	Threat assumptions, attack paths, findings and remediation evidence
AI agent assurance	Is there an adequate basis for the proposed use?	The combined evidence, reasoning, limitations and approval conditions
Certification claim	What requirements does this particular certificate cover?	The issuer, scheme, assessed object, scope and validity conditions

Assurance can draw on several techniques. DSIT describes a toolbox that includes impact assessment, performance testing, audit and conformity assessment, selected according to context. A certification claim therefore needs its own scrutiny rather than being treated as blanket deployment approval. See DSIT’s assurance toolkit.

For the terminology and assessment lifecycle, read our guide to AI agent certification and bounded assurance. The checklist below focuses on judging whether the evidence is sufficient.

Build an evidence checklist for the deployment decision

An assurance case organizes evidence into an argument a decision-maker can examine. Dstl’s guide to assurance of AI and autonomous systems describes this approach and emphasizes examining evidence that challenges confidence as well as evidence that supports it.

The following sequence is a practical review method for enterprise teams. It is not a formal certification scheme.

1. Write the decision before collecting more evidence

Specify what the reviewer is being asked to permit. Include the agent configuration, users, environment, accessible data and allowed actions. Name important exclusions.

“Approve the support agent” is too broad. “Permit this configuration to categorize tickets in the internal IT queue, with uncertain cases sent to a person” gives reviewers something concrete to assess.

Then identify the consequences of getting it wrong. Incorrect routing might delay help. Access to a restricted ticket might expose confidential information. Those failures should not disappear inside the same average performance score.

2. Link each claim to evidence that could establish it

For each important requirement, write a claim, identify the control and specify the observation needed. A policy document establishes what should happen. A permission configuration shows how access is set. An execution record shows what happened in a particular interaction. Consider whether the combination supports the claim.

NIST AI RMF 1.0 calls for documenting test sets, metrics and evaluation tools in MEASURE 2.1, and demonstrating performance or assurance criteria under conditions similar to deployment in MEASURE 2.3. Read the NIST AI RMF Core.

Apply that distinction when accepting evidence from a supplier. Ask which version, environment and tool configuration produced the result, and whether those conditions match your intended deployment.

3. Challenge the strongest evidence

Choose an important claim and ask what could make its supporting evidence misleading:

• Did a test use a privileged account that ordinary users will not have?
• Was the tool response simulated, leaving actual permission enforcement untested?
• Does the result show a proposed action, an attempted action or a confirmed change?
• Were difficult cases excluded without an explanation?
• Can the reviewer inspect enough of the record to understand the finding?

These questions help identify the next useful test. They also prevent a large evidence folder from substituting for a clear answer.

4. Record uncertainty as a review outcome

Use distinct labels for a supported claim, a contradicted claim and a claim with insufficient evidence. A missing tool record may prevent a conclusion even when the final answer appears correct.

Assign each material gap an owner and a next step. That step might be collecting a missing observation, narrowing the deployment scope or correcting the control. Additional testing is useful when it addresses the actual gap.

Worked example: reviewing an internal ticket-routing agent

Consider a hypothetical agent intended to read English-language tickets in one internal IT queue, assign an approved category and route them to a support team. Its proposed authority excludes closing tickets, changing account permissions, accessing restricted queues and sending external messages. Ambiguous requests require human triage.

The matrix below is an illustrative review aid. It contains proposed checks, not customer results or evidence that any product has passed them.

Claim to review	Evidence to collect	Gap that needs a decision
Routine tickets reach an appropriate queue	Representative test tickets, agreed routing criteria and recorded outcomes	The sample excludes common ambiguous requests
Restricted tickets remain inaccessible	Connector permissions and access-control tests using relevant identities	Only an administrator’s configuration has been tested
Prohibited operations stay blocked when tickets contain hostile instructions	Controlled adversarial cases, attempted tool calls and downstream authorization results	The final response is retained, but tool activity is missing
A failed update leaves a recoverable ticket state	Timeout and retry tests, resulting ticket state and operator recovery steps	The agent reports success before the update is confirmed
Uncertain cases reach a person	Escalation records and a walkthrough with the receiving support team	A handoff is generated, but nobody receives or owns it

The value of this matrix is the relationship between its columns. A routing benchmark could support the first claim while leaving the access-control claim unanswered. A well-written escalation message could satisfy a wording check while failing to create a usable handoff.

Now suppose the review has convincing routing evidence, but recovery after a tool timeout remains untested. An appropriate response would be to hold autonomous updates while collecting recovery evidence. A recommendation-only pilot could be considered separately, after checking its own access, disclosure and human-review boundaries. Calling it a pilot does not resolve those questions.

For the adversarial row, our prompt injection testing guide explains how to construct authorized tests and interpret tool behavior.

Decide whether to approve, restrict or hold

Agree release conditions before reviewing the results. Distinguish requirements whose failure blocks the proposed scope from lower-consequence issues an authorized owner may accept with documented conditions.

A useful decision record should state:

1. Approved scope: the exact configuration, workflow and authority covered.
2. Evidence basis: the findings and records that support the decision.
3. Remaining limitations: what is unresolved, excluded or accepted, and why.
4. Operating conditions: required human reviews, restrictions and recovery arrangements.
5. Review triggers: changes or observations that require the decision to be revisited.

Avoid combining every finding into one pass rate. Successful routing cases do not compensate for unauthorized access. Likewise, a restriction should be enforceable: if approval depends on disabling a tool, verify that it is disabled in the deployment configuration.

For the ticket-routing example, approval should identify the permitted queue and category updates. It should not silently extend to closing tickets or administering accounts. The deployment owner needs a record they can implement, and the next reviewer needs a record they can challenge.

Limits that should remain visible

Evidence applies to the conditions examined. It may leave uncertainty about unfamiliar inputs, changing dependencies or uncommon sequences of events. A review should explain consequential gaps without implying that every unknown can be eliminated.

Assurance also depends on evidence quality. Incomplete records, unrepresentative cases or an ineffective human handoff can weaken an otherwise persuasive case. A completed checklist does not guarantee future safety or establish legal compliance.

Keep the review proportionate to the proposed authority and consequences. Where evidence cannot support a material action, narrow the action or gather better evidence before proceeding.

Frequently asked questions

Can we reuse a model provider’s evaluations?

Yes, as supporting evidence about that component. Check the model version, evaluation conditions and relevance to your use. Then identify what remains untested in your agent’s tools, permissions, retrieval and human approvals. Provider evidence should have an explicit role in the deployment decision.

Which framework can help structure the work?

The NIST AI Risk Management Framework is a voluntary reference for organizing AI risk management. Use it alongside applicable company policies and requirements selected for your operating context. The checklist in this article is a review aid, not a claim of NIST certification.

When should the assurance decision be reviewed again?

Revisit it when a change could invalidate its evidence or conditions, such as a new writable tool, broader user access or a different workflow. Operational findings can also require review. Our runtime monitoring guide explains how to connect signals to investigation and reassessment.

Prepare your first assurance review

Choose one agent and one deployment decision. Bring its permitted actions, a small set of consequential claims and the evidence already available. Use the checklist to identify what a reviewer can conclude and what still needs work.

Request a demo to discuss your enterprise’s assurance scope with AI Agent Certify. Confirm supported coverage, evidence handling and deployment availability for your use case. You can also explore the platform approach to Assurance Plans, evidence and human review.