Generative Artificial Intelligence (GenAI) already drafts requirements, code, test cases and safety analyses in many engineering teams. Safety standards assume that the tools behind those work products are deterministic and reviewable. A GenAI tool is neither. At Safetronic 2026, my colleague Vito Magnanimo and I will show how safety design patterns we already trust in the vehicle can make multi-agent GenAI tools safe to use, without asking engineers to re-read everything the tool produces.
October 5, 2026
© Fraunhofer IKS
The default safety measure for AI-generated work products is a person reading them. I think that measure is quietly failing.
It fails because of volume. Agentic tools already produce requirements, code and analyses faster than the teams responsible for them can review, and the gap widens with every gain in throughput. Under that pressure, fluent and well-formatted output gets skimmed. A reviewer who rubber-stamps it adds cost without adding detection, and the safety case ends up crediting a measure that doesn't detect anything. It also fails where it matters most. A fabricated clause number or a requirement ID that doesn't exist reads exactly like a real one, and catching it means checking every reference against its source. For a machine, that's mechanical. For a person at the end of a long review day, it's error-prone.
So it’s about a different target: take the human out of the inner loop. Engineered measures carry the confidence claim, and people are called in for escalations, not for blanket review.
The central decision in the concept may sound excessive at first: we treat the GenAI tool as if it were a safety product.
ISO/IEC TS 22440-1, the upcoming specification on functional safety and AI systems (currently a committee draft), would typically classify a GenAI development tool as AUL-B1 and send it down the tool route of its Annex C. We deliberately go further and engineer the tool to what the draft requires of AI inside a safety-related control system, AUL-A1. That means a full AI safety life cycle, an AI fault analysis, a monitored and redundant architecture, and dedicated testing. The reasoning is simple. Once you stop reviewing every output, and that's the whole point of having the tool, a tool error has the consequence of a product error. The rigour has to match.
The good news is that we didn't have to invent new safety mechanisms. Diverse redundancy with two-out-of-three voting, the monitor/actuator principle behind layered monitoring concepts like E-Gas, plausibility limits and a safe fallback are all patterns assessors already accept. They are combined in a single framework for GenAI tools. Its architecture is shown below.
Figure: A monitored, redundant architecture for GenAI-based tools, aligned with the upcoming ISO/IEC TS 22440-1
Here's how the multi-agent tools we're implementing on this concept work, following the figure from input to output.
Pre-processing checks that a request lies inside the tool's operational design domain and separates instructions from content, so a command smuggled into a retrieved document stays just text. The request then runs through three channels in parallel. Each one is a complete multi-agent implementation of the functionality, with agents that have defined roles and authority, reusable skills, retrieval from a controlled and version-pinned corpus, and calls out to deterministic checkers such as compilers, static analysers and requirement databases. The agent graph is fixed and its loops are bounded, so the orchestration stays ordinary, testable software. What differs between the channels is the model underneath: each comes from a different model family and provider.
A judge from yet another model family compares the three candidates and forms a two-out-of-three decision. Two supervisors sit around this path. The AI monitor checks that the output is consistent with the inputs and the required functionality; it can accept, reject or escalate, but it never repairs. The non-AI monitors check everything that has a determinate answer. Does the cited clause exist in the pinned edition of the standard? Does the requirement ID resolve? Does the code compile? Does each generated test fail on an injected mutant? Only the limiting logic can release an output, and a non-AI veto can't be outvoted. When automatic handling isn't possible, a human gets one clearly flagged item to decide, with the source and the diverging outputs side by side. The human doesn't disappear. The job just changes from reading everything to deciding what the machine can't.
If I had to pick one lesson from this work, it would be this: diversity has to come from different model families, not from different prompts. Two prompts on the same base model are not independent, and a judge from the generator's own family shares its blind spots. Put correlated models into a vote and you get confident agreement on a wrong answer.
One more thing about the figure: it's a superset. Whether a tool derives safety requirements, generates code and tests, or proposes FMEA and FMEDA content, the skeleton stays the same. What changes is the set of monitors. Each tool instantiates the ones its own fault analysis calls for, and every monitor it leaves out needs a recorded rationale. The safe state is identical for all of them: no unverified output reaches a safety work product.
