Hazard Analysis for AI and Black Box Systems

Hazard analysis for AI systems works when you stop trying to enumerate the model's failure modes and start bounding what the model is allowed to do. That's the method in one sentence: treat the AI as an untrusted component inside a trusted architecture, then run your hazard analysis on the architecture, where failure modes can actually be listed.

The reason this reframing is necessary sits in the foundations of every functional safety standard in use today. IEC 61508, ISO 13849, and their descendants were written for deterministic systems. A sensor drifts. A relay welds shut. An output stage fails high. These failures are knowable in advance, so the standards built their entire machinery around enumerating them: identify the failure modes, estimate their rates, design against them, verify the design.

A learned model breaks the first step. Its behavior is a function of training data and weights no engineer can inspect meaningfully, and it can produce novel outputs in novel situations. For a generative model, that novelty is the entire point of the component. You cannot enumerate the failure modes of a system whose defining property is behavior you didn't explicitly specify.

Plenty of teams respond to this by either refusing AI components entirely or writing hazard analyses that quietly pretend the model is a normal sensor. The first choice surrenders real capability. The second produces documentation that won't survive contact with an assessor, or with reality.

What does "untrusted component" mean in practice?

It means the safety argument never depends on the model behaving well. The model gets treated the way a security engineer treats input from the open internet: useful, powerful, and assumed to be wrong at the worst possible moment.

Four architectural commitments follow from that assumption:

Bounded authority. The model's outputs are constrained before they reach anything that moves. Speed limits, force limits, workspace limits, command whitelists. The model can request; it cannot command. The boundary enforcement lives in deterministic logic that a hazard analysis can fully characterize.

Defined safe states. For every operating mode, the system has a reachable safe state that does not depend on the model to reach. When the monitors disagree with the model, the system goes there. Defining these states is classical safety engineering, and it stays classical even when the component being supervised is exotic.

Deterministic monitors. Independent, verifiable logic watches the model's outputs against the defined envelope. The monitors are simple enough to certify with existing methods. This is where the enumerable failure modes live, so this is where the standards machinery applies cleanly.

A documented envelope argument. The safety case argues that the envelope holds even when the model misbehaves, because the monitors and bounds were designed assuming it would. The model's cleverness is a performance property. The envelope is the safety property.

The pattern shifts the hazard analysis onto ground where it works. You're no longer asking "what will the model do?", which is unanswerable. You're asking "what can the system do if the model does anything?", which is a bounded, analyzable question.

How do you run the hazard analysis itself?

The sequence follows ISO 12100 methodology, with the AI treated as a hazard source whose outputs are unconstrained until proven otherwise:

1. Identify hazards at the system boundary, not inside the model. The hazard isn't "the model hallucinates." The hazard is "the arm exceeds safe speed in the shared workspace." Model misbehavior is one cause among several.

2. Assume worst-case model output for every hazard. If the model can command motion, analyze the most damaging motion it could request. The analysis holds regardless of how unlikely that request seems, because likelihood estimates for learned behavior don't deserve your trust.

3. Derive safety requirements for the envelope, not the model. Required performance levels land on the monitors, interlocks, and bounds, where ISO 13849 calculations apply normally.

4. Validate the envelope against adversarial cases. Test the monitors with model outputs designed to break them, including malformed, extreme, and rapid-sequence commands.

5. Keep the analysis alive. Models get retrained and fleets get updated. A hazard analysis that reflects last year's weights describes a system you no longer ship. This is where static documents fail structurally: spreadsheets and consultant PDFs cannot accommodate systems that change monthly.

One honest limitation belongs in every version of this method: the envelope has to be defined by people who understand the application domain deeply. Bounding a warehouse robot is tractable. Bounding a system whose safe envelope itself depends on context, like a surgical device, is a harder problem, and pretending otherwise is how bad safety cases get written.

Where does ASAP fit?

ASAP is built for exactly the part of this problem that kills teams in practice: keeping the analysis current and connected. The platform provides an ISO 12100 compliant hazard analysis that updates dynamically with linked mitigations, so when the architecture changes or a new model version widens the output space, the affected hazards, requirements, and validation evidence are visible immediately instead of buried in a stale spreadsheet. Full V-Model lifecycle coverage means the envelope argument stays traceable from the initial hazard identification through continuous monitoring in the field.

There's a second reason this topic belongs to Fennec. ASAP is itself AI-assisted and human-verified at every decision point, which is the same architecture this post recommends: use the model to speed the work, keep deterministic human judgment as the authority. The platform practices the principle it supports.

The methodology carries proof from the largest fleet in the industry. Built over 5 years inside Amazon Robotics, ASAP protects more than 1 million robots operating 24/7, a fleet where autonomy and safety infrastructure had to coexist long before the current generation of AI reached industrial systems. T2 Tool Qualification from TUV Rheinland and HORIBA MIRA confirms independently that the platform reveals defects without introducing errors. The standards for AI in safety functions will keep evolving. Architectures built on bounded authority and living hazard analyses will be ready for whatever they say.

If there's a learned model anywhere near your safety functions and your hazard analysis hasn't caught up, that's the conversation to have now rather than at assessment time. Talk to Fennec about the architecture.