The New Risk Physics

Autonomy Changes Exposure

A useful design heuristic is:

Autonomous risk ≈ error probability × authority × reach × velocity × irreversibility

This is not a regulatory formula, an actuarial model, or a claim that these variables multiply cleanly in practice. It is a way to expose what model-centric thinking can miss: a system with a low per-decision error rate may still create unacceptable institutional risk when it possesses broad authority, can reach many consequential systems, operates at machine speed, and produces effects that are difficult to reverse.

Why Velocity Matters

Human organizations have historically contained an accidental but important control: throughput. One employee can make only so many payments, approvals, trades, customer communications, access changes, or case dispositions in an hour. Human latency is inefficient, but it also limits the rate at which one mistaken interpretation can propagate.

Autonomous systems remove much of that friction. A flawed instruction, compromised identity, bad context, misaligned objective, stale policy, or unanticipated interaction can be repeated across thousands of actions before a conventional review cycle notices the pattern. The risk therefore depends not only on whether the system can be wrong, but on how quickly wrongness can compound and how long the institution needs to detect, contain, and revoke authority.

This changes the importance of control latency. Time-to-detect, time-to-interrupt, time-to-revoke, and time-to-recover become governance variables alongside accuracy, explainability, and validation. A control that is theoretically correct but operationally slower than the autonomous system it governs may fail at the moment it matters.

Model Risk Is Necessary But Not Sufficient

Can the model produce unreliable or inappropriate outputs?

This lens examines conceptual soundness, data, validation, performance, limitations, monitoring, and intended use. It remains indispensable when models inform or shape consequential decisions.

A Practical Risk Surface

Illustrative exposure as autonomous authority widens

A Practical Risk Surface — 4 rows by 4 columns.

Low	Low	Low	Low
Moderate	Moderate	High	Moderate
High	Moderate	High	High
High	High	High	High

Conceptual exposure state only; not an empirical rating.

The matrix is intended to reveal dimensions that should influence the delegation boundary, not to score products or vendors.

Source Musewoods analysis, illustrative

Authority Should Be Dynamic

Provisional
A new autonomous actor starts with the smallest authority needed to demonstrate useful behavior safely. The institution observes outcomes, uncertainty, exceptions, and control performance before widening the delegation.
Bounded
The actor receives a defined envelope tied to task, domain, exposure, context, and evidence. The objective is not to eliminate autonomy, but to make the boundary explicit enough that the institution knows what it has delegated.
Expanded
Authority can increase when competence is demonstrated under representative conditions and the institution has evidence that controls, escalation, and recovery work as intended. Expansion should be deliberate and attributable.
Contracted
Authority shrinks when context changes, uncertainty rises, dependencies degrade, outcomes deteriorate, or the institution's risk appetite changes. Dynamic contraction prevents yesterday's evidence from becoming a permanent license.
Revoked
Some events require immediate termination rather than gradual adjustment: compromised credentials, policy breach, unexpected cross-domain behavior, missing evidence, material control failure, or an incident whose blast radius is still unknown. Revocation is therefore part of the runtime architecture, not merely an administrative lifecycle step.

The Control Objective

Separate capability from authority
What a system can technically do should never silently determine what the institution permits it to do. Capability is an engineering property; authority is an institutional decision.
Minimize unbounded blast radius
Delegation should be narrow enough that one failure cannot cascade across unrelated resources, customers, jurisdictions, or business processes. Where reach must be broad, independent constraints become more important.
Make revocation first-class
A kill switch is useful only when the institution knows which identities, permissions, sessions, workflows, and downstream effects have to be contained. Revocation should be designed, tested, and observable.
Verify independently at consequential boundaries
Self-reported confidence is not the same as independent evidence. High-consequence actions may require a separate verifier, policy engine, specialist control, dual authorization, or human authority that does not share the same failure path.
Preserve temporal evidence
The institution needs the policy, context, authority, and evidence that existed when the decision was made, not only their current versions. Otherwise later reconstruction rewrites history.
Let outcomes change future authority
A governed autonomous system should learn institutionally, not only statistically. Outcomes, incidents, reversals, overrides, and near misses should affect risk appetite, policy, competence assessment, and the authority envelope assigned to future actions.