From medical diagnosis to quality control, visual information is often used to make important decisions. Automating these decisions can relieve the burden on experts and avoid errors. This requires complex AI systems whose development involves large amounts of data, and which are generally unable to make reliable predictions. FAST attempts to provide an answer to these problems: a framework for developing reliable AI systems with little data.
July 3, 2023
© iStock/Larysa Pashkevich
What do quality control in manufacturing, component inspection during maintenance work, and medical diagnosis based on X-ray images have in common? In all these activities, visual inspection is carried out, and decisions are made based on these inspections. To make decisions, such as whether a product is faulty or not, domain experts inspect images of random samples. With large production volumes, this activity is very repetitive, leading to fatigue and errors. These manual inspections also result in high personnel costs and an inefficient use of expert knowledge.
A study [1] conducted in the USA, for example, showed that around 80% of prostate biopsies are performed on benign tissue, meaning experts spend time inspecting large quantities of benign tissue. Assisting automation solutions, known as Computer-Assisted Diagnosis (CAD) systems, can make better use of experts' time and reduce errors caused by fatigue. The situation is similar in other fields of application for visual inspection.
Because the processing of images and the applications mentioned are highly complex, AI solutions based on Deep Learning (DL) are particularly suitable for such problems. Image processing systems using Artificial Intelligence (AI) are not uncommon. Such solutions also offer a decisive advantage: the problem, such as deciding which produced goods need to be sorted out, does not need to be explicitly described, which is often not even possible. Instead, it is sufficient for the system to learn from example images. Unfortunately, the use of Deep Learning also poses some difficulties. The aforementioned application fields are particularly affected by the following two problems:
1. Very large quantities of annotated sample images are required to successfully train image processing DL solutions. Data sets with several million images are not uncommon, especially for complex objects or scenarios. This leads to high labour and cost expenditure before the system can even be developed and evaluated. In addition, it is often not possible to capture such volumes of data due to frequent changes to the goods produced, small sample sizes or high processing and annotation costs.
2. AI is not reliable enough for many practical applications. This applies especially to systems that make safety-critical decisions, such as quality control for vehicle parts or automated medical diagnosis. Further measures, from conception to runtime monitoring, are needed to validate such solutions.
Both issues are among the greatest challenges in the realisation of AI-based systems for practical applications [2]. To find solutions for the automation of visual inspection tasks, the Fraunhofer Institute for Cognitive Systems IKS has developed the FAST framework. FAST stands for "Feedback-guided Automation of Sub-Tasks". The aim of the framework is to automate decisions, and therefore processes, using an AI component.
FAST is a partial automation solution: It works in parallel with the experts and reduces their workload by automating the decisions for which a reliable assessment is possible. The remaining decisions are processed directly by the experts. FAST essentially consists of two components: an AI component making predictions about the captured images, and a second component quantifying the reliability of these predictions and thus deciding which data can be processed automatically.
The crucial question now is how an AI model can be trained to produce reliable predictions, even with limited training data available. For this purpose, FAST introduces the concept of "sub-problems": A sub-problem is defined by the training data that the system can learn early on and by a significantly smaller number of samples than what the overall problem would require. For example, images with difficult lighting conditions and poor visibility of the core features could be excluded as training data. This would allow a system focused on the core features to be trained. There are multiple ways to identify the data for the sub-problem. Figure 1 visualises the steps for the model development.
Figure 1: FAST system during training
An AI model trained on the data of a sub-problem has a smaller "problem space" and is therefore optimised for this restriction. Because the sub-problem only contains part of the data, fewer initial data is required. In addition, the reduction of possible inputs leads to a model that generates fewer errors and is therefore more reliable.
As the system is used, new data is entered into the system. For this data, the model makes a statement in terms of the problem to be solved, such as which goods need to be sorted out. The reliability of this statement can then be assessed. For example, inputs that fall within the previously defined sub-problem can be processed by the AI model with a high degree of reliability. Reliable statements can be used for automation decisions, while inputs that lead to unreliable statements are passed on to experts. Expert feedback is used to adapt and continuously improve the AI model. Figure 2 shows the process for new inputs.
Figure 2: FAST system during execution
Overall, FAST is developing reliable AI solutions with low data requirements that can be used quickly to relieve experts at an early stage.
[1] Gurcan, Metin N., et al. "Histopathological image analysis: A review." IEEE reviews in biomedical engineering 2 (2009).
[2] "Are Organisations Succeeding at AI and Machine Learning?" Rackspace Technology, 2021, https://www.rackspace.com/site...
This project was funded by the Bavarian State Ministry of Economic Affairs, Regional Development and Energy as part of the project Supporting the Thematic Development of the Institute for Cognitive Systems.
The FAST project is also funded through participation in the Fraunhofer AHEAD programme.
