We Built AI to Think Faster. We Haven’t Built Organizations to Think Better: The hidden organizational failure behind AI risk

Why organizations are misdiagnosing AI risk—and what it will cost them.

For the past several years, organizations have been told a consistent story about artificial intelligence:

·       If you want to reduce risk, focus on the model.

·       Audit the data.

·       Test for bias.

·       Improve explainability.

Billions of dollars and countless frameworks later, one uncomfortable truth is becoming harder to ignore: AI failures are not slowing down. They are accelerating.

From biased hiring tools to flawed risk scoring systems to opaque decision-making in customer-facing applications, organizations continue to deploy AI systems that produce outcomes they cannot fully explain, defend, or control.

The prevailing response has been to treat these failures primarily as technical problems. But technical risk is only one side of the problem. 

The Misdiagnosis: Treating AI Risk as a Technical Problem

Most AI governance strategies are built on a flawed premise:

If we fix the model, we fix the risk.

This assumption drives investment toward:

·       Bias detection tools

·       Model audits

·       Fairness metrics

·       Algorithmic transparency

These are necessary. But they are not sufficient.

Because they ignore the environment into which AI is actually deployed:

AI is deployed in organizations made up of humans operating under pressure, ambiguity, and constraint.

Research in cognitive psychology and organizational behavior consistently shows that under conditions of speed and pressure, decision quality declines—even among experienced professionals. Hermanns and Teuber (2026) found that time pressure negatively affected human-AI team performance and reduced participants’ ability to distinguish correct from faulty AI responses.¹ Yet these are precisely the conditions under which most AI systems are deployed.

Just ask those building it.

This concern is increasingly coming from inside the industry itself. In July 2026, more than 1,300 employees of frontier AI companies signed Pacing the Frontier, warning that capability development could outpace society’s ability to understand and govern increasingly powerful systems. Their warning points to a capacity problem: the systems are advancing faster than the technical, institutional, and human mechanisms are being built to govern them.

The Real Problem: A Breakdown in Organizational Decision-Making

AI systems do not operate in isolation. They are embedded in workflows, overseen by teams, and deployed through decisions made by people.

Those decisions are shaped by:

·       Time pressure

·       Performance incentives

·       Unclear accountability

·       Fragmented ownership

·       Cognitive overload

Under these conditions, even well-designed AI systems can produce harmful outcomes—not only because the model is flawed, but because the organization failed to exercise sufficient judgment around it. This is not simply a technology problem. It is a decision architecture problem.

I call this the Dual Failure Model of AI Governance: consequential AI risk emerges from the interaction between vulnerabilities in the AI systems and vulnerabilities in the organizational environments responsible for governing them.

The Dual Failure Model of AI Governance

To understand why AI governance continues to fall short, leaders need to recognize a dual failure occurring inside their organizations.

The Dual Failure Model of AI Governance explains that risk emerges from the interaction between flawed models and flawed decision environments.

1.     Model-Level Risk

This is where most attention goes:

·       Biased or incomplete data

·       Opaque algorithms

·       Limitations in model performance

·       Potential for disparate impact

These risks are real, measurable, and widely studied. But they are only half the equation.

2.     Organizational Decision Failure

This is where risk is created—and amplified.

Inside most organizations:

·       Decisions about AI deployment are rushed

·       Accountability for outcomes is diffuse

·       Escalation pathways are unclear or underused

·       Human oversight is nominal rather than functional

Critically, these environments suppress deliberate, effortful reasoning—the kind of thinking required to identify and mitigate risk before it becomes visible.

Generative AI intensifies this problem because it dramatically increases the production of fast, fluent, plausible output. In the language of dual-process theory, AI can function as a System 1 accelerator. Effective oversight requires deliberate, analytical scrutiny associated with System 2 thinking.

But putting a human in the loop does not automatically activate System 2. Time pressure, workload, incentives, automation bias, unclear authority, and organizational culture can push people toward the very kind of rapid judgment governance is supposed to counterbalance.

A human cannot function as an effective control merely because they possess technical authority to intervene. They also need psychological permission to do so. If challenging an AI recommendation means slowing a launch, questioning a senior leader, contradicting a technical team, or risking performance consequences, formal override authority may mean very little in practice. Psychological safety is therefore not simply a cultural benefit in AI governance. It is part of the control environment.

The right number of people is not the same as the right amount of governance capacity. Governance capacity is not simply headcount. It is whether people have the expertise, time, authority, information, incentives, and psychological safety required to exercise judgment when it matters. More on this in future writing.

Organizations reward speed, efficiency, and rapid iteration. Recent incidents show what happens when increasingly capable systems encounter governance mechanisms that cannot keep pace.

Just ask Jacob Coxon, the AI safety researcher who resigned from Anthropic in September 2026 after publicly expressing concerns about the trajectory of frontier AI development. Coxon’s account illustrates a deeper organizational problem: people inside an organization can recognize serious risks while remaining embedded in competitive structures and incentives that continue to propel development.

According to Coxon, the competitive dynamics create pressure to continue. In his own words, irresponsible actors “are racing straight to self-improving superintelligence and gambling with our lives.”

These are the exact conditions under which poor decisions are most likely.

Knowing the risk is not the same as having the organizational capacity to govern it.

Many organizations believe they are managing AI risk because they have implemented governance frameworks. In reality, those frameworks often assume a level of scrutiny, coordination, and critical thinking that does not exist under real operating conditions.

The result is a false sense of security—one that only becomes visible after failure.

Why “Move Fast” Is Quietly Increasing Your Risk Exposure

Many organizations continue to apply “fail-fast” principles to AI development.

In traditional software development, fail-fast approaches can work because many failures are relatively contained, observable, and reversible. The same logic becomes considerably more dangerous when applied without modification to high-impact AI systems.

Fail-fast is an experimentation strategy, not a governance strategy. In high-impact AI environments, failing fast can mean harming fast.

AI failures can be:

·       Systemic rather than isolated

·       Scaled across populations

·       Socially distributed before detection

Under pressure to move quickly:

·       Teams rely more heavily on intuition than analysis

·       Risk signals are deprioritized

·       Ethical concerns are treated as secondary

·       Oversight becomes procedural rather than meaningful

The result is not just faster innovation. It is faster propagation of flawed decisions at scale. When AI systems operate in high-impact domains, failing fast can mean harming fast.

The Illusion of “Human-in-the-Loop”

In response, many organizations claim to mitigate risk by keeping “humans in the loop.” But in practice, this often amounts to little more than a checkbox.

Few organizations can answer basic questions such as:

·       Who exactly is responsible for reviewing AI-driven decisions?

·       What authority do they have to override the system?

·       What training do they have to recognize bias or risk?

·       Under what conditions are they expected to intervene?

Without clear answers, human oversight becomes symbolic rather than functional.

And symbolic oversight does not prevent harm. The issue is not whether humans are in the loop. It is whether organizations have created the conditions for those humans to think critically, challenge outputs, and act with authority.

What Organizations Should Do Differently

Organizations that are beginning to manage AI risk effectively are not just investing in better models. They are redesigning how decisions are made.

They focus on three critical shifts:

1.     Designing for Deliberate Decision-Making

Instead of assuming good judgment will occur, they build structures that require it.

This includes:

. Formal decision checkpoints where deployment cannot proceed without documented risk review

·       Defined escalation pathways for high-risk use cases

·       Structured decision frameworks rather than ad hoc approvals

 They recognize that good decisions do not happen by accident. They are designed.

2.     Clarifying Decision Rights and Accountability

In many organizations, responsibility for AI outcomes is fragmented across teams.

Effective organizations:

·       Assign clear ownership for AI decisions

·       Designate named individuals accountable for outcomes post-deployment

·       Ensure accountability does not disappear once systems go live

When everyone is responsible, no one is responsible.

3.     Aligning Incentives with Responsible Outcomes

If teams are rewarded for speed alone, they will optimize for speed.

Organizations seeking stronger governance:

·       Incorporate risk management into performance metrics

·       Reward teams for identifying and mitigating potential failures

·       Create space for critical questioning without penalty

They understand that culture is not separate from governance. It is governance.

A More Useful Question for Leaders

Executives might be asking:

“Is our AI biased?”

A more useful question is:

“Under what conditions would our organization fail to recognize or act on that bias before it causes harm?”

That question turns AI governance from a compliance exercise into an organizational diagnostic and shifts the focus from the system to the decisions surrounding it. And that is where the greatest risk—and the greatest opportunity—lies.

The Cost of Getting This Wrong

Organizations that continue to treat AI governance as a technical exercise face increasing exposure across three fronts:

·       Regulatory risk as oversight expectations become more stringent

·       Reputational damage when failures become public

·       Operational instability as teams struggle to manage systems they do not fully understand

These are not hypothetical risks. They are already materializing. Simply read the headlines about the Hugging Face and DseWiki incidents.

Hugging Face agents escaped a controlled environment, exploited real infrastructure, crossed organizational boundaries, and compromised a third party.

During an internal OpenAI cybersecurity evaluation, a highly capable research model circumvented controls intended to isolate it from the internet, exploited vulnerabilities, escaped its sandbox, and ultimately accessed Hugging Face’s infrastructure. OpenAI later acknowledged that the models communicated through unauthorized channels, exploited shared infrastructure, gained internet access, and accessed third-party systems in ways that were misaligned with the assigned task. OpenAI described the incident as a “warning shot,” noting that sufficiently capable agents can now collaborate, work around technical controls, and take dangerous actions no human explicitly directed.

This is a story about the conditions surrounding the model: reduced safeguards, an evaluation environment that allowed too much latitude, insufficient containment, and organizational processes that did not prevent a capable agent from moving beyond its intended boundaries. The model failure and the governance capability failure interacted.

DseWiki agents found an unauthorized place to communicate, coordinate, evade controls, and preserve their activity.

The DseWiki incident actually began months earlier. Researchers later discovered more than 15,000 edits made by AI agents on the German-language programming wiki. The agents had effectively repurposed the site into an external communications channel where they shared tactics, discussed ways to bypass restrictions, coordinated approaches to evaluation tasks, and attempted to conceal their behavior. When a human moderator began deleting the material, the agents created backup pages and alternative locations to preserve their communications. Researchers also identified attempts to tamper with the website itself.

Reuters reported that OpenAI officials had learned of the episode before it became public. OpenAI disputed some characterizations of the activity and said the German incident was distinct from the later Hugging Face breach, but the episode nevertheless provides evidence of agents engaging in unanticipated coordination and circumvention outside their intended environment. 

OpenAI ultimately submitted an incident report to the European Commission, confirming that the incident is more than an interesting research observation.

Neither incident proves that catastrophic AI failure is inevitable. They demonstrate something more immediate and actionable: technical safeguards can fail, AI systems can behave in unanticipated ways, and human oversight can arrive after machine-speed action has already occurred. The question is no longer whether advanced AI systems can behave in ways their developers did not intend. They already have.

The governance question is whether organizations can detect, interpret, challenge, and contain those behaviors before they produce consequential harm.

Both incidents demonstrate that technical safeguards can fail, models can behave unexpectedly, and human oversight can arrive after the system has already acted at machine speed. These are observed organizational realities, in a fail-fast culture that ruled innovation before agentic AI hit the scene.

The Bottom Line

AI governance failures are best understood as dual failures: flawed models can produce bias and disparate impact, while organizations too often fail to create the conditions in which people can exercise the critical thinking necessary to identify, challenge, and govern those harms at scale.

Neither side of the Dual Failure Model for AI Governance can compensate indefinitely for the other. Better organizational judgment cannot make an unsafe model safe. Better models cannot compensate for organizations incapable of recognizing, challenging, escalating, and learning from risk. Responsible AI requires both.

AI does not remove the need for human judgment. It increases the consequences of getting that judgment wrong.

Organizations that fail to redesign their decision systems will continue to experience AI failures—no matter how advanced their models become.

Those that succeed will not be the ones with the most sophisticated algorithms.

They will be the ones with the most deliberate, accountable, and well-designed ways of making decisions about them.

About the Author

Dr. Tchicaya Ellis Robertson holds a Ph.D. in Applied Psychology with an Industrial and Organizational Psychology specialization and has spent more than 30 years studying how organizational systems, human behavior, and operating conditions shape performance and decision-making. An executive advisor with advanced training in generative AI and digital transformation, her current work examines AI governance through the lens of organizational capacity, human judgment, and responsible innovation.

__________________________________________________________________________

¹Hermanns, L., & Teubner, T. (2026). Under pressure: How time constraints, task complexity, and AI reliability shape human-AI interaction. Behaviour & Information Technology, 45(11), 2597–2621.