When AI oversteps: Model Risk Management lessons from the Meta, Anthropic, and OpenAI security incidents

The recent security breaches involving Meta, Anthropic, and OpenAI has highlighted an important challenge for organisations deploying AI. As AI systems become more capable of accessing information, using tools and taking action, risk frameworks need to evolve beyond traditional approaches to model oversight.

Traditional Model Risk Management (MRM) focuses on data, methodology, performance and outputs. However, incidents involving GenAI systems often arise from the interaction between the model, its objectives, connected tools and the wider control environment. Risk functions increasingly need to assess the entire system, including how components interact and how unexpected behaviour could create operational, regulatory or customer harm.

Do you govern the model or the system?

The increasing adoption of generative AI is challenging some of the assumptions underlying established MRM approaches.

While model performance remains important, organisations also need to understand how AI systems interact with data sources, business processes and connected tools. In practice, risk often emerges not from the model itself, but from the wider ecosystem in which it operates.

This raises an important question for firms deploying AI: are you governing the model, or the system?

Traditional MRM centres on data, methodology, performance and outputs. By contrast, risks in GenAI systems often emerge from interactions between the model, its objectives, connected tools and the wider control environment. This may require risk frameworks to take a more holistic view of the end to end system rather than focusing primarily on model outputs.

Who controls access?

Every AI system should have a clearly defined purpose and approved boundaries. Firms need to understand what the system can access, which actions it can take and how those permissions are controlled.

Critical controls should operate independently of the model itself. Permissions, approval gates and escalation mechanisms should not rely solely on instructions that the system may fail to follow. Effective governance depends on establishing clear limits around capability, authority and autonomy, particularly as AI becomes increasingly embedded within business processes.

Would you spot a breach?

Controls are only effective if firms can identify when they fail.

Monitoring should detect when a system moves beyond its approved use, while reliable records and audit trails should support investigation and response. As AI systems become more complex, firms need confidence that they can identify unexpected behaviours, assess potential impacts and intervene where necessary.

Ownership is equally important. Responsibility for approving changes, responding to incidents and accepting residual risk should be clearly defined before deployment. Clear accountability helps ensure that governance arrangements remain effective throughout the lifecycle of the system.

Adapting MRM for GenAI and agentic systems

As organisations continue to explore the potential of GenAI and agentic AI, MRM frameworks will need to evolve to address new forms of risk.

At 4most, we help organisations:

• Adapt MRM for GenAI and agentic systems
• Assess end-to-end risk for use cases
• Strength governance over capability, access and autonomy
• Define practical MRM controls for complex AI-enabled workflows
• Build MRM-compliant monitoring and intervention frameworks that work in practice

These AI security incidents serves as a timely reminder that effective AI governance extends beyond model performance. As AI systems become more capable of accessing information, interacting with tools and taking action, organisations must ensure that oversight frameworks evolve accordingly. The focus is no longer solely on how a model performs, but on how the wider system behaves.