Governing the Rise of AI Agents: Understanding IMDA’s Model AI Governance Framework for Agentic AI

The four pillars of responsible agentic AI

These are the four core pillars and what they mean to organisations.

1. Assess and bound risks upfront

Before deploying AI agents, organisations must first evaluate whether a use case is suitable for agentic AI. Risk assessment should consider factors such as

• The domain and tolerance for error, such as financial transactions versus meeting summaries
• Access to sensitive or confidential data
• Interaction with external systems
• The scope of actions the agent can perform, such as read versus write permissions
• The reversibility of the agent’s actions.

For example, an agent capable of executing payments or changing databases carries far higher risks than one summarising internal documents. To reduce potential harm, organisations should design clear boundaries for agents. These may include:

• Limiting tools and system access
• Applying identity management and permission frameworks
• Running agents in controlled environments

The objective is to limit what agents can do before problems arise.

2. Make humans meaningfully accountable

Despite their autonomy, AI agents must operate within systems of human oversight. The framework emphasises clear accountability across the AI value chain. This may include

• Model developers 
• Agent platform providers
• Deploying organisations
• End users

Within organisations, responsibilities should be distributed across teams such as leadership, product teams, cybersecurity specialists and operational users. For instance, Protiviti's Guide to AI Governance - Frequently Asked Questions suggests that chief information security officers, when assessing AI agents, ought to concentrate on the operational environment where these agents will be implemented and how they might affect business operations. The evaluation needs to cover how each agent connects with current systems, what sensitive data it can access, and the security protocols safeguarding against unauthorised entry or improper use.

Human oversight should also be built into workflows through meaningful checkpoints, especially when agents perform high risk or irreversible actions. Examples include:

• Editing sensitive data
• Making payments
• Sending external communications

These checkpoints ensure humans retain control over critical decisions while allowing agents to automate routine tasks.

3. Implement technical controls across the agent lifecycle

Responsible deployment of AI agents requires robust technical safeguards throughout development, testing and operation.

• Development stage: Organisations should implement guardrails for agent planning and reasoning components, apply strict controls to tool usage, and ensure least privilege access to data and systems.
• Predeployment testing: Agents should be thoroughly tested to evaluate task execution accuracy, policy compliance, correct tool usage and robustness in edge cases.
• Post-deployment monitoring: Because agents interact dynamically with proper environments, organisations should gradually roll out deployments; monitor agent behaviour continuously; log actions and interactions; and establish escalation procedures for anomalies.

In complex systems, agents may also monitor other agents to detect abnormal behaviour.

4. Enable end-user responsibility

Responsible AI deployment does not end with developers. Users who interact with AI agents also play an important role. Organisations should ensure transparency and training for users by providing information on:

• The agent’s capabilities
• Data access and usage
• Potential limitations or failure modes
• Escalation channels if issues arise

The framework distinguishes between these two types of users:
 

1. Users interacting with agents: These include customers interacting with support chatbots or service agents. These users primarily require transparency about the system.
2. Users integrating agents into workflows:  These include employees using coding assistants or automation tools. These users require training and governance policies to ensure responsible usage.

Understanding automation bias, the over-reliance of humans on automated systems, is of great importance.

Why the IMDA framework matters

Agentic AI systems are expected to become more deeply embedded across enterprise operations. According to Protiviti’s AI Pulse Survey Vol 3, 77% of the most mature organisations are using or expect AI agents to manage repetitive tasks. The use of AI agents is also expanding from operational and customer-facing functions to strategic planning, fraud detection and compliance-driven functions, according to the survey. 

Without governance, risks may include:

• Incorrect or unauthorised actions
• Data breaches
• Biased decisions
• Disruption to connected systems

The IMDA framework provides a structured approach for organisations to balance innovation with responsibility. It also reinforces an important principle: AI autonomy should expand organisational capabilities while maintaining human accountability.

Agent technology is developing at a rapid pace, which is why the Model AI Governance Framework for Agentic AI is designed to be a living document. As organisations increasingly adopt AI agents to automate workflows and decision-making, governance frameworks like this will play an important role in ensuring that innovation proceeds responsibly.