Beyond governance: Who answers for AI outcomes?

From governance to accountability

By the end of 2026, every in-scope Commonwealth AI use case must have a designated accountable owner. Agencies should have already met the requirement to appoint Chief AI Officers, and under the Commonwealth’s updated AI policy they must now identify who is accountable for each AI use case. Launching the APS AI Plan, Public Service Minister Katy Gallagher said the plan “puts strong governance and transparency at its core, ensuring human oversight is always present.”

The policy represents an important shift in how the Commonwealth thinks about AI governance. Until recently, the conversation has focused on frameworks, ethics principles, risk assessments and oversight committees. The Commonwealth is now moving beyond governance structures and demanding that a named individual answers for an AI system’s use and outcomes.

This shifts the conversation from governance to accountability. When asked who’s accountable, most agencies can point to a steering committee or an executive sponsor for AI. Many have appointed Chief AI Officers and Accountable Officials as required under the plan. These committees or roles own adoption, capability building and governance. However, they are not automatically accountable for every individual AI use case operating within the agency: the chatbot advising citizens, the model supporting compliance decisions, or the tool summarising case files.

What the auditors found

Recent audit findings show that governance arrangements alone do not always produce clear accountability. When the Australian National Audit Office (ANAO) examined AI governance at the Australian Taxation Office in 2025, it found an organisation with extensive experience using AI and mature technology capabilities. Yet the audit still identified weaknesses in accountability arrangements and the integration of governance processes into project lifecycles. It found that 74 per cent of ATO’s in-production models did not have a completed data ethics assessment. ANAO found this undermines the ATO’s ability to deliver and to assure the delivery of AI that aligns with ethical principles.

Aviation has solved the accountability problem

Aviation offers an apt example. Modern aircraft rely heavily on automation, but flying an airliner with automated controls does not absolve the captain of accountability. Regardless of how much automation is involved, one person remains accountable for the flight – the pilot in command. One person, every flight.

Government AI has the automation. It wants the same level of accountability. The question is whether AI “captains” are empowered, or captains in name only.

Government AI has the automation. It wants the same level of accountability. The question is whether AI “captains” are empowered, or captains in name only.“A solution should be built with accountability at the centre, meaning governance, compliance and the embedding of ethical principles into each solution roll up to one identified person,” says Hanneke Catts, a Director at Protiviti.

Where it breaks

“We have narrowed this to one question” says Will Hunt, a Director at Protiviti, “Name the person who can switch the system off today, if needed.”

At most agencies today, the answer is a list of functions rather than a name. Technology points to the business; the business points to the platform owner; the platform owner points to the vendor.

When a name does comes up, often it’s the person who signed the impact assessment. Typically, that person has no monitoring budget, no contractual right to the model vendor’s performance data, and no written authority to switch anything off.

Accountability only works when it is matched with authority. Naming an accountable owner is not enough if that person lacks the visibility, decision-making powers or resources needed to oversee the use case effectively.

Without that authority, an accountable owner becomes a scapegoat. If the point is to address AI risks meaningfully by assigning an accountable owner, that owner must be involved in the procurement and contract management of the model, understand what disclosures vendors must provide, how model changes are communicated, and what rights the agency has when issues arise. They should receive performance and monitoring information after deployment rather than only participating during initial approval stages. Most importantly, they should have authority to intervene when necessary.

Enabling accountability: Five questions for audit and risk committees

The agencies that implement the Commonwealth’s new requirements most successfully are likely to be those that enable accountability instead of just assigning it. Audit and risk committees can test this with five straightforward questions:

• Is there an inventory that identifies every in-scope AI use case and its accountable owner?
• For the highest-impact use case, can the owner suspend the system today?
• Do vendor contracts provide rights around disclosure, model changes and remediation?
• What monitoring occurs after deployment, and who reviews the results?
• If a citizen challenges an AI-influenced outcome tomorrow, who signs the response?

Accountability is not a brake

Announcing the new AI Review Committee in June 2026, the DTA’s Lucy Poole said “Agencies remain responsible for identifying, assessing, managing and monitoring the risks of their AI use cases.”

The mandate is clear. The next step is ensuring that owners have the authority, information and organisational backing to answer when the outcome is challenged — by an ombudsman, an auditor-general or a parliamentary committee. Accountability with empowerment is how agencies move beyond governance, and how public sector AI will ultimately earn trust.