01

Governance begins with the job definition

An agent is easier to govern when its role is narrow enough to describe and test. Document who uses it, what event starts the task, which information it may use, which actions it may take and what counts as a successful result. Also write down the actions it must never take.

NIST's AI Risk Management Framework organizes risk work around Govern, Map, Measure and Manage. Applied to an agent, this means assigning accountability, understanding the operating context, evaluating behavior and responding to observed risk throughout the lifecycle rather than performing a single pre-launch review.

02

Seven controls to define before production

The control set should match the impact of the task. A knowledge assistant and an agent that changes customer or financial records should not share the same release standard.

Purpose

Define the task, user, success condition and prohibited outcomes.

Knowledge

Approve sources, permissions, freshness rules and citation behavior.

Tools

Use least-privilege access and separate read, draft, submit and irreversible actions.

Validation

Constrain outputs and verify required fields, formats and business rules before action.

Escalation

Send uncertainty, sensitive decisions and failed validation to a named human owner.

Evaluation

Test representative, edge, adversarial and failure scenarios against defined acceptance criteria.

Monitoring

Log relevant activity, measure corrections and tool failures, and maintain a response procedure.

03

Treat prompt injection as an operating risk

OWASP lists prompt injection among the critical risks for applications built with large language models. Instructions can appear directly in a user request or indirectly inside content the model retrieves. The practical response is layered: restrict sources, separate untrusted content from system instructions, limit tool permissions, validate outputs and require approval for high-impact actions.

A model's refusal behavior is not a complete security boundary. The surrounding application must assume that generated text can be wrong or manipulated and prevent that text from becoming an unrestricted command.

04

Evaluate behavior with real work

Create a test set from representative tasks, common mistakes, ambiguous requests, missing documents and adversarial inputs. Score grounding, task completion, corrections, escalation and tool errors. Preserve failed examples so later versions can be compared against the same standard.

Release gradually. Start with internal users or draft-only actions, review logs and expand permissions only when evidence supports the change. Governance is strongest when technical controls, operating ownership and measured behavior reinforce one another.

Record the model, prompt, knowledge version and tool permissions used for each evaluation cycle. Without that configuration context, a later score cannot explain whether the system improved or merely changed.

05

Keep people where consequences matter

Human review is appropriate for financial commitments, regulated advice, sensitive external communication, irreversible changes and decisions with legal, clinical or reputational consequences. The reviewer needs the source context, proposed action and reason for escalation, not simply an approve button.

The objective is not to add approval to every step. It is to place accountable judgment at the points where an incorrect action would matter and let automation handle predictable work around those points.