AI Agent Risk Assessment: What Organizations Need to Evaluate

Gartner expects 40 percent of enterprises to demote or decommission autonomous AI agents by 2027, and the cause is not model quality. It is governance gaps that nobody found until an agent did something expensive in production. Closing that gap is the entire job of an AI agent risk assessment.

Most assessment templates still in circulation were built for predictive models. They ask about training data, accuracy, drift, and disparate impact, then stop. An agent that can call an API, write to a production database, spawn a sub-agent, and remember what it did three sessions ago fails in ways those questions never reach. What follows is the evaluation set that covers the difference: what to inventory, how to tier agents by autonomy, which failure modes to test for, and how to map the result onto NIST AI RMF and ISO/IEC 42001.

What Changes When a Model Becomes an Agent

A language model produces text. An agent produces consequences. That single difference reorganizes the whole risk picture, and it does so through four structural properties that traditional AI risk assessment was never designed to evaluate.

1. Autonomy. The agent selects its own sequence of steps toward a goal. You approve the objective, not the path, which means the path is unreviewed at design time.
2. Tool access. The agent holds credentials and calls external systems. Every tool in its inventory is a live capability, not a described one.
3. Persistent memory. The agent carries state across sessions. Something it absorbed on Monday can steer a decision on Friday, long after the original context is gone.
4. Delegation. Orchestrating agents spawn sub-agents. Accountability for the final action gets distributed across a chain that no single review ever sees end to end.

The Cloud Security Alliance calls out a fifth property that deserves its own line: the temporal gap between an agent initiating an action and a human observing it. An agent can delete records, send external communications, modify configurations, or trigger payments in a burst that finishes before anyone notices the behavior was wrong. Detection lag is not a monitoring inconvenience. It is a risk dimension, and it belongs in the assessment as its own scored field.

This is why an agent risk assessment cannot be a model risk assessment with two extra questions bolted on. Model risk asks whether the output is correct. Agent risk asks what happens next if it is not.

An agent risk assessment should not operate as an isolated compliance exercise. It should form part of a broader agentic AI governance framework that defines accountability, authority boundaries, oversight mechanisms, and lifecycle controls for autonomous systems.

Build an Agent Inventory That Records Capability, Not Just Existence

Almost every organization that runs an AI inventory records the same eight fields: system name, owner, vendor, purpose, data classification, model provider, go-live date, and review cycle. For agents, that record tells you nothing useful. Two agents can share every one of those values while one reads a knowledge base and the other issues refunds.

The distinction Gartner flags is between an agent’s ability to act and the scope of access it has been granted. Those are separate variables, and conflating them is the most common inventory defect we see in AI governance reviews. An inventory that captures capability looks like this.

Agent inventory fields that actually inform risk

Inventory field	What to record	Why it changes the risk score
Tool manifest	Every tool, API, and connector the agent can invoke, with the permission scope attached to each	Defines the maximum damage the agent can do on its worst day, independent of what it usually does
Write authority	Which systems the agent can modify, delete from, or transact against, versus read only	Read-only agents cap out at data exposure. Write access introduces irreversibility
Identity and credentials	The service account or non-human identity the agent authenticates as, and whether it acts as itself or impersonates a user	Determines whether audit logs can attribute an action to an agent or only to a borrowed human identity
Memory scope	Whether memory persists across sessions and users, what gets written to it, and who can influence what it stores	Persistent memory turns a single poisoned input into a durable behavioral change
Delegation graph	Which agents this agent can invoke, and which agents can invoke it	Multi-agent chains create trust inheritance that no single-agent review catches
Human checkpoint	The exact decision points where a person must approve, and the approval volume per week	High approval volume is a leading indicator that the checkpoint has become a rubber stamp
Reversibility	Whether each action type can be undone, by whom, and within what window	The strongest single predictor of real-world impact when an agent goes wrong

One practical test for inventory completeness: hand the record to someone who has never seen the agent and ask them to describe the worst plausible outcome if it were fully compromised. If they cannot answer from the record alone, the inventory is a registry, not a risk artifact.

Tier Agents by Autonomy Before You Assign a Single Control

Uniform governance across all agents is a documented failure pattern. Gartner analyst Shiva Varma describes enterprises treating agent governance as binary, either locked down or fully trusted, and identifies that binary as the root cause of failure. Over-restrict the simple agents and delivery slows, which pushes teams into shadow development. Under-restrict the autonomous ones and you inherit operational, security, and compliance exposure you never assessed.

Proportional governance solves this by classifying agents into autonomy levels, each representing a different trust boundary with its own control set. The four-tier model below is the one worth adopting as your assessment spine, because it answers the question that actually drives control selection: can this agent act, and does a human stand between the decision and the effect.

Autonomy tiers, assessment focus and minimum controls

Tier	What the agent does	Assessment focus	Minimum controls
Level 1: Observe	Read-only access to defined sources. Output visible only to the requester. Summarization, retrieval, code explanation	Data exposure and output accuracy	Scoped data access, user authentication, usage logging, functional and security testing
Level 2: Advise	Generates recommendations, drafts, and proposed actions. Humans review and execute manually. No write access	Decision influence and automation bias	All Level 1 controls, plus accuracy and hallucination testing, domain quality evaluation, user training on appropriate reliance
Level 3: Act with approval	Writes data, sends communications, modifies configurations, but only after explicit human approval per action	Whether approval remains a meaningful control under load	Approval workflows with audit trails, agent-specific incident response, security testing of the expanded attack surface
Level 4: Act autonomously	Executes independently within guardrails. Humans review exceptions, logs, and aggregate outcomes	Guardrail enforcement, drift, and containment speed	Continuous monitoring, enforced guardrails, rapid rollback, circuit breakers on threshold violation, named ownership for agent behavior

PRO TIP: Assess the tier the agent can reach, not the tier it launched at Agents get promoted quietly. A Level 2 advisory agent gains a write-enabled tool in a sprint that nobody flagged as a governance event, and it is now Level 3 running under Level 2 controls. Make tier promotion an explicit reassessment trigger with a named approver, and record the tier in the inventory as a versioned field rather than a static attribute.

The Ten Failure Modes Your Assessment Has to Test

In December 2025, the OWASP GenAI Security Project published the Top 10 for Agentic Applications, developed with input from more than 100 industry contributors and built from incidents observed in production systems rather than research projections. It is the closest thing the field has to an agreed threat taxonomy, and it gives an assessment something most AI risk templates lack: a testable list.

The ten categories, identified as ASI01 through ASI10, cluster into three families that map cleanly onto assessment workstreams.

Family one: the agent is turned against you

• ASI01 Agent goal hijack. Instructions hidden in documents, retrieved content, or tool outputs redirect what the agent is trying to achieve. The EchoLeak case is the reference example: an email carrying a concealed payload prompted a Microsoft 365 Copilot instance to exfiltrate confidential mail and chat logs with no user click involved.
• ASI05 Unexpected code execution. Agent-generated code or shell commands run without sufficient validation or isolation.
• ASI06 Memory and context poisoning. Corrupted persistent memory keeps influencing decisions across sessions, long after the injection point is forgotten.

Family two: the agent’s own authority is the weapon

• ASI02 Tool misuse and exploitation. Recursive tool calls, unsafe tool composition, budget exhaustion, and state leaking between tool contexts. Note that these happen with entirely valid permissions.
• ASI03 Agent identity and privilege abuse. Impersonation, cross-agent trust abuse, privilege inheritance through delegation chains, and role bypass.
• ASI04 Agentic supply chain compromise. Poisoned tool schemas, deceptive tool descriptions, misrepresented permissions, and compromised agent registries, all of which land at runtime rather than at build time.

Family three: the system misbehaves as a system

• ASI07 Insecure inter-agent communication. Message injection, spoofing, and agent-in-the-middle interception.
• ASI08 Cascading agent failures. A small fault propagating through tool chains, dependent agents, and trust relationships.
• ASI09 Human-agent trust exploitation. Fluent, confident output producing uncritical acceptance, which is automation bias with a better interface.
• ASI10 Rogue agents. Goal drift, reward hacking, agent collusion, and autonomy that quietly exceeds its designed boundary.

Three of these have no equivalent in the older OWASP Top 10 for LLM Applications: inter-agent communication security, system-wide failure cascades, and behavioral drift. If your assessment inherited its threat list from LLM-era guidance, those three are the gaps to close first. The full framework is available from the OWASP GenAI Security Project and is worth reading in the original.

Risk assessment identifies what could go wrong, but organizations must also decide how those risks will be controlled in production. Effective governance controls for autonomous AI agents help determine which actions can proceed independently and which require approval, monitoring, containment, or rollback.

Score Agent Risk on Blast Radius, Reversibility and Detection Lag

Likelihood times impact does not survive contact with agentic systems. Likelihood is unstable because an agent’s behavior changes with the model version, the prompt, the retrieved context, and the tools available on a given day. Impact is underspecified because the same action carries wildly different consequences depending on whether it can be undone.

A more useful scoring approach evaluates three properties that stay stable even when agent behavior does not.

5. Blast radius. How many systems, records, customers, or dollars can this agent touch in a single uninterrupted run? Score it against the tool manifest and permission scope, not against the intended use case. Intended use is a design assumption. The manifest is a fact.
6. Reversibility. Can the action be undone, by whom, and inside what window? A refund is reversible. An email to a regulator is not. A schema migration sits somewhere painful in between. Rank action types as reversible, compensable, or permanent, and treat permanent actions as requiring a tier above whatever the agent otherwise qualifies for.
7. Detection lag. How long between the agent taking a wrong action and a human being in a position to know? This is where most agent programs are quietly weakest. If the answer is measured in days, the control set has to shift from review toward prevention, because review will arrive after the damage is complete.

The three combine into a practical decision rule. High blast radius with low reversibility and high detection lag is the profile that produces the incidents behind Gartner’s demotion forecast, and it should never be approved for autonomous operation regardless of how well the agent performs in evaluation. Strong benchmark performance does not compensate for an inability to undo the mistake or notice it in time.

This scoring also gives risk committees something they can act on. Telling a business owner that an agent scores 3.4 on a five-point scale invites debate. Telling them the agent can issue up to 400 irreversible customer-facing communications before anyone would notice tends to end it.

Agent Identity Is the Control Plane Most Assessments Skip

Palo Alto Networks put the average enterprise machine-to-human identity ratio at 82 to 1 at the end of 2025. Agents, tools, datasets, APIs, and the orchestration pipelines connecting them all carry identities, and each one is a potential point of compromise. Yet most AI risk assessments treat identity as an infrastructure concern that lives in a different review entirely.

That separation fails for a specific reason. Agents frequently authenticate using a borrowed human identity, typically the credentials of the user who invoked them or a shared service account created during a pilot. When that happens, three things break at once. Audit logs attribute agent actions to people, which makes forensic reconstruction guesswork. Least privilege becomes unenforceable, because the agent inherits everything the human can do rather than only what the task requires. And revocation is blunt: disabling the agent means disabling the human.

Questions worth asking in every agent assessment

• Does this agent hold a distinct, attributable identity, or does it act as a user or a shared account?
• Is the permission set derived from the task, or inherited from whoever happened to deploy it?
• When agent A invokes agent B, does B receive A’s privileges, its own, or the original human’s?
• Can we revoke this agent’s access in under an hour without taking down a business process?
• Do the logs record which agent, which tool, which parameters, and which upstream trigger?

The last question deserves particular attention in regulated US environments. Under NIST SP 800-53 and the audit expectations that flow into SOC 2 and FedRAMP evidence, attribution is not optional. An agent whose actions cannot be distinguished from a human’s in the log record creates a control deficiency that has nothing to do with AI and everything to do with accountability.

Map the Assessment to NIST AI RMF and ISO/IEC 42001

Neither framework was written with agents in mind, and both remain the right place to anchor the work. The NIST AI Risk Management Framework gives you the risk analysis structure through its four functions. ISO/IEC 42001 gives you the management system that makes those decisions repeatable across teams and defensible to an auditor. They answer different questions, which is why mature programs run both.

Here is how the agent-specific evaluation described above lands against each.

• Govern (NIST) and Clause 5 Leadership (ISO 42001). Named ownership for agent behavior, an approved autonomy tier policy, and a documented position on which action types are never delegated to autonomous execution.
• Map (NIST) and Clause 6 Planning (ISO 42001). The capability inventory, the tool manifest, the delegation graph, and the blast radius assessment. This is where most of the agent-specific work concentrates.
• Measure (NIST) and Clause 9 Performance Evaluation (ISO 42001). Adversarial testing against the ASI categories, approval quality metrics rather than approval counts, detection lag measurement, and drift monitoring against the original goal specification.
• Manage (NIST) and Clause 8 Operation (ISO 42001). Circuit breakers, rollback procedures, agent-specific incident response, and decommissioning that actually revokes credentials rather than just switching off the interface.

NIST has acknowledged the agentic gap directly. Through the Center for AI Standards and Innovation, it launched an AI Agent Standards Initiative in February 2026, and an AI Agent Interoperability Profile is expected in the fourth quarter of 2026. The AI RMF 1.0 itself is under revision as part of the White House AI Action Plan. Programs building agent assessments now should design them to absorb that guidance rather than waiting for it, because the exposure is already live.

What US Regulation Actually Requires in 2026

There is no federal statute in the United States that mandates an AI agent risk assessment. That fact gets misread constantly, usually in the direction of assuming there is nothing to comply with. The obligations exist, they just arrive through sectoral regulators, contract terms, and state disclosure law rather than through a single AI act.

Colorado is the clearest illustration of why watching only the headline AI statute is a mistake. SB 26-189, signed May 14, 2026 and effective January 1, 2027, repealed and replaced the original Colorado AI Act. It removed the duty of reasonable care against algorithmic discrimination, the deployer risk management program requirement, and the annual impact assessment mandate. What remains is a narrower transparency framework built on developer documentation, deployer disclosure, and a post-adverse-outcome notice to consumers within 30 days, enforced exclusively by the Attorney General with penalties up to 20,000 dollars per violation.

Read that carefully and the practical conclusion is counterintuitive. Colorado dropped the impact assessment requirement, and the reasons to run one did not change. If your agent contributed to an adverse consequential decision, you have 30 days to explain its role to the affected consumer. You cannot produce that explanation from a system nobody assessed, documented, or logged. The statutory mandate went away. The evidentiary need did not.

The obligations that reach agents in practice

• Sectoral regulators. HIPAA for agents touching protected health information, GLBA and fair lending rules for financial services, and SEC cybersecurity disclosure obligations where an agent incident becomes material. None of these mention agents. All of them apply to what agents do.
• Federal procurement. Executive Order 14110 directed agencies to align risk management practices with NIST AI RMF, and that alignment increasingly flows down to vendors through contract terms rather than regulation.
• State disclosure law. Colorado from January 2027, plus a widening set of state automated decision-making rules. The trend is toward disclosure and recordkeeping duties that presume you have documentation to disclose.
• EU AI Act spillover. The Digital Omnibus, Regulation (EU) 2026/1744, entered into force on July 27, 2026 and deferred Annex III high-risk obligations to December 2, 2027 and Annex I to August 2, 2028. Article 50 transparency duties and the Article 4 AI literacy duty were not deferred. US firms serving EU users inherit the timeline regardless of where they are headquartered.

HEADS UP: Deferred is not cancelled The sixteen-month EU deferral has been widely read as breathing room, and organizations that treat it that way will arrive at December 2027 with weeks of preparation time instead of months. The underlying risk did not move with the compliance date. Neither did the harmonized standards work that the deadline depends on, which is precisely why the original date slipped.

Human Oversight That Survives Contact With Production

Human-in-the-loop is the control most organizations claim and the one that degrades fastest. The failure is well described: approval workflows collapse under time pressure and approval fatigue, producing a false sense of safety while the attack surface keeps expanding. A reviewer clicking approve on the 340th routine request of the week is not exercising oversight. They are providing an audit artifact.

Assessments should therefore measure oversight quality rather than oversight existence. Four indicators are worth instrumenting.

8. Rejection rate. If a human approver rejects fewer than roughly one in fifty agent proposals, the checkpoint is probably not functioning as a control. Investigate before you assume the agent is simply that good.
9. Time to decision. Approvals granted in under a few seconds on decisions that would take a person minutes to evaluate independently indicate pattern-matching rather than review.
10. Approval volume per reviewer. Volume that grows faster than headcount is a scheduled failure. Set a ceiling and treat breaching it as a governance event rather than a staffing problem.
11. Override survivability. When a reviewer does reject, does the agent respect it, or does it re-propose the same action through a different path in the next run?

Two design choices help materially. Route only genuinely consequential decisions to human approval, because reviewing everything guarantees reviewing nothing carefully. And give reviewers the agent’s reasoning and the counterfactual, not just its conclusion, since a recommendation presented without its basis invites exactly the trust exploitation that ASI09 describes.

Reassessment Triggers, Not Annual Reviews

Annual review cycles were built for systems that change annually. Agent behavior can shift in an afternoon when a model version updates, a tool is added, a prompt is edited, or an upstream data source changes shape. A calendar-driven reassessment will miss all four.

Event-driven triggers work better. Define them in the agent’s governance record and wire them to the same change management process that already governs production systems.

• The agent gains a new tool, connector, or permission scope.
• The underlying model version changes, including silent vendor updates to a hosted endpoint.
• The agent’s autonomy tier is promoted, or a human checkpoint is removed.
• The agent begins invoking, or being invoked by, another agent.
• The system prompt, goal specification, or guardrail configuration is modified.
• An incident occurs involving this agent or a structurally similar one elsewhere in the estate.
• The agent is deployed into a new jurisdiction or a new consequential decision domain.

The evidence supports this framing. IDC research found 88 percent of AI pilots never reach production, with the failures clustering on governance, data readiness, and observability rather than model quality. Forrester reaches a similar conclusion, attributing agent failures largely to ambiguity, miscoordination, and unpredictable system dynamics rather than conventional bugs. The organizations that get agents into production and keep them there are not the ones with better models. They are the ones that noticed a change and reassessed before it became an incident.

These agent-specific risk factors also expose the limitations of traditional AI governance for autonomous agents, particularly when existing governance processes focus on model outputs without adequately considering what an AI system can actually execute.

Frequently Asked Questions

What is an AI agent risk assessment?

It is a structured evaluation of what an autonomous AI agent can do, what happens when it acts incorrectly, and how quickly you would know. Unlike a model risk assessment, which focuses on output accuracy and bias, an agent assessment examines tool access, write authority, memory persistence, delegation chains, and reversibility of actions. The output is a risk tier that determines which controls apply.

How is AI agent risk assessment different from a standard AI impact assessment?

An AI impact assessment asks how a system affects people, typically around fairness, privacy, and consequential decisions. An agent risk assessment asks what the system can execute and how far the damage travels before someone intervenes. You need both. Impact assessment covers the decision. Agent assessment covers the action taken on that decision.

Which framework should US organizations use for agent risk assessment?

NIST AI RMF is the practical anchor, since US procurement teams, auditors, and insurers treat it as the reference for reasonable AI risk practice. Pair it with ISO/IEC 42001 for the management system layer and the OWASP Top 10 for Agentic Applications for the threat taxonomy. NIST is developing agent-specific guidance, with an AI Agent Interoperability Profile expected in late 2026.

Do we still need impact assessments after Colorado SB 26-189 removed the requirement?

Yes, for evidentiary reasons rather than statutory ones. SB 26-189 dropped the annual impact assessment mandate but kept a duty to explain an automated decision’s role in an adverse outcome within 30 days. You cannot produce that explanation without documented assessment and logging. The requirement changed form rather than disappearing.

How do you assess risk in a multi-agent system?

Assess each agent individually, then assess the delegation graph as its own artifact. Map which agents can invoke which, what privileges pass along each edge, and where a fault in one agent propagates. Two categories in the OWASP list, insecure inter-agent communication and cascading agent failures, exist only at the system level and are invisible to per-agent review.

What autonomy level requires the strictest controls?

Level 4, where agents act independently within guardrails and humans review exceptions and aggregate outcomes rather than individual decisions. At this level actions execute at a speed and scale that outpaces human oversight, so controls shift toward continuous monitoring, enforced guardrails, rapid rollback, and circuit breakers that halt operation on threshold violations.

How often should AI agent risk assessments be repeated?

Use event triggers rather than a calendar. Reassess when the agent gains a tool or permission, when the underlying model version changes, when a human checkpoint is removed, when it starts interacting with another agent, or after any related incident. Annual cycles miss changes that alter agent behavior in a single deployment.

Who should own AI agent risk assessment inside an organization?

Risk and compliance should own the framework and the tiering decisions, while engineering owns the technical evidence such as tool manifests, logs, and adversarial test results. Every agent also needs a single named business owner accountable for its behavior. Assessments that lack that named owner tend to stall when a control decision requires someone to accept residual risk.

The Takeaway

The organizations that lose control of AI agents are rarely the ones with weak models. They are the ones that assessed the agent as though it were a model, missed that it could write to production, and had no way to notice for three days. Capability inventory, autonomy tiering, blast radius, reversibility, and detection lag are what separate an assessment that predicts incidents from one that documents them afterward.

The most useful next step is small. Take your three most autonomous agents, write down every tool each one can call and whether each action can be undone, and see whether the answer surprises you. It usually does.

For teams building this capability formally, GAICC’s Agentic AI Governance Professional and ISO/IEC 42001 certification programs cover agent risk assessment, autonomy tiering and framework mapping in depth.