DEEP Series (Data Efficient Evaluation Platform), Part 1
Reliable AI Enables Automation of Quality Inspection in Industry

In many fields of application, manual activities are automated based on visual information. Examples are quality control of manufactured goods, sorting of parcels in logistics as well as inspection of screw and plug connections by a robot arm. Often these automation systems require complex technological knowledge for development and adaptation and are not suited for modifications during use. Automation systems based on machine learning (ML) promise more flexible solutions even to complex problems.

However, practical problems frequently arise in ML approaches when only small quantities of data are available to train the system. In addition, the system has to go through a protracted iterative development process before it can be used effectively and with sufficient reliability.

Data availability can be a problem

Applications such as visual inspection in most cases require complex ML models. The necessity of large data volumes for the development of such models is a common obstacle to their use. Often it is not possible to collect large quantities of data, e.g. due to a small number of items or complex measurement methods. Even when it is possible to collect sufficient data, the accurate, consistent and exact annotation of the data (= labelling) is time-consuming and costly. Low data quality (e.g. due to a lack of variability) also means that even large data volumes do not ensure that a reliable ML system will result.

Labelling remains a challenge also during the deployment of the system: if the environment (e.g. the light conditions) or the imaged objects change, the ML model usually needs to be reworked. In order to include these changes in the ML model, it is necessary to collect additional data which represents the changed conditions. A good strategy for the selection of such data points can significantly reduce the expenditure (e.g. costs of annotation) for the adaptation.

Ramp-up phase and reliability

After the collection of data, it is often necessary to adapt learned models manually or consider special cases separately. Generally, several iterations of training and practical evaluation are required to provide a functioning automation solution. Automation tasks, in particular in the context of inspection and quality assurance, have to meet strict requirements regarding correctness and reliability. Reliability means avoiding errors, quantifying uncertainties and detecting previously unknown data (= out-of-distribution data). Complex deep learning (DL) models in particular are often unable to correctly represent uncertainties. Such uncertainties can be necessary, for example, to highlight or identify unknown situations and objects, respectively, or identify when a decision should not be automated based on the available data.

How has industry solved these problems until now?

The solutions for automated visual inspection frequently implemented in industrial environments until now involve restricting the automation system or the field of application. This means that the complexity of the problem is reduced in order to apply less complex solutions (such as conventional computer-vision methods). As a consequence, the automation problems need to be significantly less complex or the inspection environment is kept particularly constant. Examples for this are consistent light conditions, sensors, orientation, low variance of the objects, etc. Including new objects for inspection in such systems is almost impossible or takes a lot of effort. More flexible, reusable automation systems significantly better address the challenges faced by ML solutions as described above.

• Anomaly detection
  Anomaly detection involves the identification of data points which deviate from the “good data” among the training data and are consequently considered anomalies. In the case of visual inspection, this could be defects or incorrect data. Since defects are mostly rare, data acquisition with a focus on “good data” is easier. The system then learns to detect all deviations from good images. The quality of anomaly detection depends on the number of collected “good images” and the consistency of the environment. Since only a deviation of the objects to be inspected is learned but not a change in the inspection environment, even small changes, e.g. in lighting or the background, can result in incorrect assessments. Even if the training data provides good coverage of all occurring variations, there can still be marginal cases which will be difficult to detect.
• Out-of-distribution (OOD) detection
  Method for detection of unknown data (out-of-distribution, OOD). In OOD detection, input data which does not fall into the domain of the training data (e.g. different environmental parameters or unknown objects) is detected and tagged. The quality of OOD detection depends on the available data. The OOD system is taught with all available data. Full coverage of the data often is not possible, however, in particular in case of environments with few restrictions. This results in errors for underrepresented classes.
• Uncertainty quantification (UQ)
  Methods for quantification of the uncertainty of predictions (uncertainty quantification) decide during runtime whether a statement can be considered reliable. Generally, a fixed limit value for the “reliability”, which is difficult to specify for small evaluation data volumes, must be defined in advance.
• Active learning (AL)
  Active learning (AL) is an iterative process for data-efficient development of an ML model. In every iteration, new data from a collection of data which has not been annotated yet is selected for annotation by an expert. The selection of data follows a procedure to assess which data points would provide the most significant performance increase for the ML model. After annotation, the ML model is updated with the new data and the next iteration is launched. These iterations are performed until the ML model achieves a sufficient performance level. For AL, the initial data pool generally requires a large data volume. Active learning is primarily used for datasets with little annotation. The method is frequently hampered by the so-called “cold-start problem”, however. As the model of the current iteration is used as a method for data selection, the training is dependent on the performance of the model although it is still in the initial phase of its development. As a consequence, the progress of training starts out slow.

DEEP, the solution offered by Fraunhofer IKS

To address these problems, the Fraunhofer Institute for Cognitive Systems IKS has developed the machine learning toolchain DEEP (Data Efficient Evaluation Platform). DEEP is a software tool for reliable AI-assisted automation of quality inspection systems. DEEP enables evaluating the potential of data for image recognition using AI at the click of a button and finding out which ML methods are particularly suited for the specific data of an application. For this purpose, DEEP automates various specific technologies of Fraunhofer IKS, which will be described in detail in the following parts of this series of blog posts.

The process steps of DEEP are simple. Once the image data has been uploaded to the DEEP tool, this data is rated and, in a first step, divided into training data, test data and evaluation data. Based on the data, the tool tests various ML models using different training parameters. This process identifies those models which best match the specific requirement and the availability of data. The performance of the various models is analysed with regard to their accuracy and robustness. Targeted recommendations which models are best suited for the specific data are made based on these analyses. In order to further improve the quality of the model, analyses are performed using the Fraunhofer tools FAST, Robuscope and Concept Learning.

A prominent feature of DEEP is its ability to handle data efficiently. The tool does not require large datasets to deliver meaningful results. Small data volumes already enable DEEP to perform a well-founded feasibility analysis. This is a significant advantage for companies which have limited data resources at their disposal or are in the early stages of data collection.

One of the fields of application in which DEEP offers significant advantages is quality assurance: the software tool optimises production processes by means of accurate and reliable detection of deviations or faulty parts. The same is true for automation: the scope of manual, repetitive tasks to be performed by skilled personnel is reduced and automated image recognition systems increase efficiency. DEEP therefore maximises the potential of the existing data. The features of DEEP include

• fast and uncomplicated evaluation of the data for suitability of a specific project,
• determination of the performance of image evaluation models,
• interpretation of the results for accuracy and robustness,
• a general recommendation for further development of the AI model, and
• safe and private processing of the analysed data.

In the following parts of this series of blog posts, we will present the parts of the DEEP toolchain and address the specific issues. This includes deeper insight into the Fraunhofer IKS technologies FAST for reliable AI-assisted partial automation with restriction of the models to parts of the data space (subdomain models), Robuscope for specific suggestions for improvement of ML models, and Concept Learning with reusable models for frequently occurring concepts.

This work was funded by the Bavarian Ministry for Economic Affairs, Regional Development and Energy as part of a project to support the thematic development of the Institute for Cognitive Systems.