Artificial intelligence

Machine learning models are exposed to malicious external attacks that drastically falsify results and assessments - to the harm of people in safety-critical situations. Defense strategies already exist. But what happens when quantum computing comes into play? Together with partners, Fraunhofer IKS has taken a close look at the first approaches to averting danger.

Seeing and being seen - this not only counts in the social arena, but also in the efficient and reliable cooperation between humans and robots in industrial production. Fraunhofer IKS will be demonstrating what this is all about in particular at Automatica in Munich at the end of June under the motto: “Simplicity is key”.

Can AI revolutionize production control? This is the question the research project MBO-KISS (Methods for Evaluating and Optimizing AI-generated Control Applications Based on the Physical Simulation of Machines and Their Desired Behavior) aims to address. The project has started at the beginning of the year, with a total duration of three years. The goal is to investigate the possible usage of Large Language Models (LLMs) for generating and securely applying control applications in industrial production.

The previous three parts of our series focused on the technologies “under the hood” of DEEP, the Fraunhofer IKS machine learning toolchain. Now, we take a look at the “big picture” in form of the process steps of the DEEP procedure – how DEEP can be used to get a grip on the problems associated with the use of machine learning (ML) for future flexible quality inspection.

DEEP is a machine learning toolchain by Fraunhofer IKS for the reliable AI-assisted automation of quality inspection systems. For this purpose, DEEP automates various specific Fraunhofer IKS technologies. This part of the series of blog posts focuses on the specific technological contents of modular concept learning.

What are the key ideas shaping the current automation landscape? Read on to learn about three insights that emerged from discussions between participants at Fraunhofer IKS’ Safe Intelligence Day 2025.

As the railway industry advances, integrating Artificial Intelligence (AI) systems has become essential to improve efficiency and enhance the attractiveness of rail transport. However, implementing AI in railway operations comes with several challenges. Ensuring safety is the top priority while simultaneously increasing throughput.

DEEP, the machine learning toolchain of Fraunhofer IKS, enables the automation of quality control in production environments, even if the requirements are complex. The technologies FAST and Robuscope, likewise developed by Fraunhofer IKS, are essential components of this solution.

Quality assurance is one of the, if not the most crucial competitive factors for industrial companies. German companies have long since recognized this and generally do not have to contend with quality issues. But that comes at a cost. Artificial intelligence is now believed to become a game changer in quality assurance. Dr. Niels Syassen, Executive Board member with responsibility for SICK AG’s Technology & Digitalization portfolio, explains how this will work.

Machine learning (ML) is considered a promising technology for the automation of quality control in production environments, even if the requirements are complex. However, ML approaches require sufficiently large datasets for training of the system, which are often unavailable. The ML toolchain DEEP (Date Efficient Evaluation Platform) from Fraunhofer IKS addresses this challenge.

Industrial production is undergoing radical change. With constant technological advancement and societal change, industrial automation must also reinvent itself to meet changing requirements and challenges – with the keyword here being adaptive production.

The performance of machine learning (ML) models is typically assessed using metrics such as accuracy, precision, recall, F1 score, etc. But how trustworthy are these single point estimates really? Such metrics reflect reality only partially and can provide us with a false sense of safety. Especially in safety-critical areas, this can be dangerous. Fraunhofer IKS develops a formal framework for the quantification of uncertainty which aims to make the residual risk “more computable”.

A precisely defined operating environment is not only important for autonomous driving. The Operational Design Domain (ODD) also ensures safety for many other highly automated systems in rail transport, logistics and mobile robotics.

To investigate the future of automation and possible transition paths towards it, Fraunhofer IKS is working on extending automation systems for novel applications. Read on to understand how these prototypes have taken shape and what insights they provide.

The EU has developed a legal framework for artificial intelligence (AI) products. The aim is to protect citizens and companies from potential harm that may be caused by the use of artificial intelligence. What does this mean for companies that develop AI products?

It takes time. And it will take time. But it will come: Fraunhofer IKS is making a decisive research contribution to autonomous driving, explains Dr. Reinhard Stolle, the new head of the Mobility business unit and deputy director of the institute. But there is still a lot to be done.

The new Fraunhofer IKS Safety Companion is an AI agent designed to support human developers with safety-critical tasks. It uses large language models (LLMs) to understand tasks and to identify and implement the appropriate safety concepts. Executive Director Mario Trapp and Florian Geissler, Senior Scientist at Fraunhofer IKS, introduce the Safety Companion in an interview.

As production technologies continue to evolve, it can feel like companies are permanently playing catch-up. Read on to understand the current key changes in industrial automation and why it is worthwhile to embrace them.

Artificial Intelligence (AI) in Industry 4.0 still faces technological obstacles that make the use of AI difficult. Fraunhofer IKS is therefore developing a framework that supports and optimizes the data and AI lifecycle - and thus significantly increases the added value of AI in production.

Reinforcement learning is often the most suitable AI solution for a range of applications in the sectors of autonomous systems, healthcare, and communication, due to the training method. Nevertheless, the nature of these tasks makes data collection potentially resource intensive or in some cases even unachievable. This is where reinforcement learning could benefit from embedding quantum computing methods since hybrid quantum-classical reinforcement learning was empirically shown to need less training steps to reach convergence. Fraunhofer IKS is researching solutions that are also suitable for use in industry.

Deep Learning has increased interest in using artificial intelligence in more and more systems. Reinforcement Learning in Machine Learning has also experienced an upswing. However, despite these impressive recent successes, it still faces the challenge of being accepted and successfully deployed. This is especially true for safety-critical tasks such as robotics, autonomous driving, and industrial control. One approach that can help address this problem is Hierarchical Reinforcement Learning.

Holistic safety for road vehicles is the focus of Safetronic conference (15-16 November 2023 in Leinfelden-Echterdingen). In the video interview, Fraunhofer IKS Institute Director Prof. Dr. Mario Trapp and program committee member Hans-Leo Ross from Vay Technologies discuss safety challenges in the automotive sector.

Reinforcement learning (RL) is one of three main machine learning (ML) paradigms. Unlike the other two, supervised learning and unsupervised learning, in reinforcement learning the learning takes place in direct interaction with the surroundings. A brief introduction follows.

A straightforward online tool makes it possible to test the robustness of an AI model – and does so within a matter of seconds. The tool is called Robuscope, and has been developed by the Fraunhofer IKS.

Modern AI models have demonstrated remarkable capabilities on various computer vision tasks such as image classification. Their decision-making process, though, remains mostly opaque (black box problem). This can be particularly problematic for safety-critical applications like medical imaging or autonomous driving. Fraunhofer IKS explores ways to increase the transparency of this black box.

Achieving safety with AI systems requires a comprehensive and collaborative approach, including technical, ethical, and regulatory considerations. A central aspect is designing AI systems that comply with safety requirements defined in constraints.

Compared with the human brain, artificial neural networks consume much more power — even when completing simple discrimination tasks. Event-based vision offers a promising approach to solving this problem. Yet there are still a number of challenges to overcome.

What are the basic principles of self-adaptive systems? This is the basic prerequisite for bringing the concept of resilience to life. Part 3 of our series.

Using artificial intelligence (AI) in the rail industry could potentially increase both efficiency and quality. At a workshop hosted by Fraunhofer IKS, experts from Deutsche Bahn, Siemens Mobility and the institute itself gathered to discuss the possible uses and challenges associated with AI in rail applications, as well as strategies for ensuring system safety.

If uncertainties in an autonomous system are not dealt with, they impair the system’s functionality and can lead to safety risks. In part 1 of our series on uncertainty, we presented a system for classifying uncertainties. In part 2, we will look at specific sources of uncertainty that have been noted in the relevant literature.

Resilience is the key ingredient for the next generation of cognitive cyber-physical systems. After all, many visions of the future can only become reality if the systems become aware of themselves, their goals and their context in order to be able to adapt to unknown and changing environments. And in doing so always optimize utility while preserving safety. But resilience can only be bought with increased complexity. In this article, part 2 of our series on resilience, we look at the character of complexity to better understand where to start engineering in order to manage this complexity.

Cognitive systems are often affected by the problem of uncertainty, particularly if those systems use machine learning to perceive their environments. A system’s sensors are not the only source of uncertainty, however. The life cycle of any highly complex system involves a large number of decisions that are vulnerable to the effects of uncertain knowledge. Part 1 of our series on uncertainty focuses on the latest developments in this technology.

Hospital staff usually suffer from enormous workloads and staff shortages, especially in intensive care units. This has consequences for patients, particularly when facing the prospect of multi-organ failure. In the future, Artificial Intelligence (AI) aims to enhance the application of a clinical multi-organ support system to further optimize patient treatment.

While many current CDS expert systems (Clinical Decision Support) in laboratories are being criticized due to difficulty in scaling, constant review from the medical perspective and limited data cases, artificial intelligence (AI) has become a highly plausible alternative. »Safe Intelligence for Laboratories«, a webinar hosted by Fraunhofer IKS, focused on the cases and challenges of AI in laboratories from the perspectives of both industry and patients.

How can intelligent systems learn to perform tasks – and do so simply by interacting with their environment? This is the subject of reinforcement learning, a subfield of machine learning. Problems occur primarily when situations arise that the system is not yet familiar with.

Can deep neural networks in image processing reliably recognize new, unknown test patterns? First considerations for different methods — especially for safety-critical applications.

In the near future, medical engineering will be shaped by artificial intelligence and machine learning. However, the complex processes involved in machine learning create a problem for researchers — it is costly and challenging to eliminate errors before autonomous systems are put into use, and to reliably detect errors during operation.

When ensuring the safety of automated driving systems, it is important to take into account technological as well as societal and legal aspects.

As we move toward the “smart city”, sensors are set to also collect data on air quality, traffic, etc., and — in the long term — provide us with cleaner air and help us get where we want to be faster. The Fraunhofer Institute for Cognitive Systems IKS offers the necessary technologies for the safe operation of artificial intelligence (AI) and provides cities and communities with a framework to help them get started.

A sticker on a give way sign. Branches hanging in front of a stop sign. Graffiti on a speed limit sign. These are all completely normal sights on our roads, aren’t they? But things that wouldn’t generally be a problem for humans can really make life difficult for artificial intelligence.

In modern perception applications, such as in medical engineering, models based on artificial intelligence (AI) are increasingly being used due to their strong performance. However, this increasing performance often comes at the cost of the transparency of results. An online tool developed by the Fraunhofer Institute for Cognitive Systems IKS can help here.

Safe intelligence is the central research topic of the Fraunhofer Institute for Cognitive Systems IKS. This topic is very important from a technology perspective as well as with regard to the economy and society. The two Chairs of the Advisory Board for the IKS development project see it the same way — each from their own perspective.

Fraunhofer IKS has established an advisory board to provide expert support to the IKS development project. The advisory board convened for its first regular meeting on April 14. The two Chairs, Thomas Gallner and Prof. Thomas Seidl, talk about their motivation for getting involved. And about the aspects on which they intend to focus.

Artificial intelligence (AI) has to be able to handle uncertainty before we can trust it to deliver in safety-critical use cases, for example, autonomous cars. The Fraunhofer Institute for Cognitive Systems IKS is investigating ways to help AI reason with uncertainty, one being the operational design domain.

Fraunhofer IKS is carrying out research into safe software applications for quantum computers. The work is part of the new Bavarian initiative Munich Quantum Valley.

Science needs to prove that systems based on artificial intelligence (AI) are safe enough. That is what Prof. Simon Burton, who recently became Division Director for Safety at Fraunhofer IKS, is calling for. In this interview he explains the institute’s approaches to ensuring safety.

When it comes to environmental protection and climate action, most people think of renewable energies, electric cars and sustainable consumer goods. Very few would associate it with artificial intelligence (AI), but in fact AI has a crucial part to play in the fight against pollution and climate change.

Deep Neural Networks (DNNs) have proven their potential. This applies above all to their use in the laboratory. Their performance in real-world applications leaves a lot to be desired. One major problem is the estimation of uncertainty. A closer look at methods that meet this challenge.

On the automotive market, traditional manufacturers are in a head-to-head race with new players from the software world. In the end, the winner will be decided not only on the basis of technological innovation, but also by how secure the technologies are.

Machine learning means a disruptive challenge for safety assurance. Without an acceptable safety assurance concept, many great ideas will not find their way to the market. It’s therefore no surprise that AI safety has gained much more attention over the past few months. A brief series of articles will take a closer look at the importance of safety for AI. The first article focuses on the basic understanding of what safety actually means. It highlights why it is so important to understand that safe AI has less to do with AI itself and much more to do with safety engineering.

Machine learning, and especially deep learning, can be used to enable many highly complex applications, such as autonomous driving. However, there are new challenges that need to be overcome to secure such systems. An overview.

Today, autonomous vehicles function reasonably well in test situations since the conditions are severely restricted and thus easy to manage. A key issue however is how to design autonomous vehicles so that they operate dependably even in complex and previously unknown situations. A solution from Fraunhofer IKS is helping to uncover and predict such difficult situations.

In autonomous systems or medical engineering, uncertainty in machine learning (ML) processes needs to be reliably determined. That is why we are comparing four state-of-the-art methods of quantifying the uncertainty of deep neural networks (DNNs) in the context of safety-critical systems — so that the AI can learn when there is scope for doubt.

Complex machine learning (ML) processes present researchers with a problem: Eliminating errors before autonomous systems are put into operation and reliably identifying errors at runtime are laborious and challenging processes. At present, this severely restricts the use of machine learning in safety-critical systems and necessitates new approaches in order to benefit from the advantages offered by ML in these areas in the future.

In this interview apl. Prof. Dr. habil. Mario Trapp, Director of the Fraunhofer Institute for Cognitive Systems IKS, talks about the competencies of the institute that make cognitive systems safe.