Human-Robot Interaction
“So that robots know what to do”

Leaving cooperative, heterogeneous robots alone in unstructured environments isn’t such a good idea. After all, only with an AI-powered management solution will they be able to solve complex tasks without causing any damage. The Fraunhofer IKS is working on developing such a system.

mask I Stock 1171902434 B4 LLS klein

Mobile robots, such as driverless transport systems in the world of logistics, work very efficiently in completing their orders by the dozen – if not the hundred – in enclosed spaces, moving their cargo from A to B at high speed. They complete their tasks horizontally or vertically, whether by lifting sequences or self-propulsion. To prevent collisions, in most cases, a central control system is used that has an overview of all the orders, the network of paths and the positions of the individual robots.

Should problems arise despite this level of control, the robots also have sensors that allow them to detect and avoid obstacles. Most of the environment is “structured,” though: This means that although individual pallets may temporarily block the pathways in a warehouse, the environment – the layout of the hall, including the columns, the hall floor and the shelves – is fixed. Most automated driverless transport systems use similar systems, simplifying the interactions and their interchangeability. System changes and/or the transfer of loads between different mobile robot systems take place at transfer points, whether manual or automated.

And what happens when human employees are brought into the mix?

If the enclosed area is also open to employees, the transport system must be able to meet the corresponding safety requirements; at the expense of the efficiency. In the event of an encounter, the driverless transport systems must reduce their speed, come to a halt, and, if possible, choose an alternative route. Let’s expand this scenario a little further and consider a case of mixed operations in the hall, in which mobile robots, people, conventional forklifts and tugger trains all need to work side by side. And if we also include the current trend of the automated logistics not only taking place in the hall but outside it, the level of complexity increases even further. In such a scenario, a central control system, if not the entire transport system, can rapidly reach its limits – and may even break down.

The researchers at the Fraunhofer Institute for Cognitive Systems IKS have set themselves the task of looking at exactly these scenarios, and significantly improving the processing of orders with the use of heterogeneous mobile robots on the basis of their own management system. With this goal in mind, the researchers aren’t just taking a closer look at the logistics sector, but also at the more chaotic operations on a construction site or the service robotics in the healthcare sector; in hospitals, for example. In all scenarios, specific examples can be found in which problems will arise in the future through the use of heterogeneous robotics, which can only be solved with an AI-supported management system.

What is heterogeneous robotics?

“In a heterogeneous system, robots work together which differ, for example, in terms of their choice of sensors, their design, their capabilities, or even their manufacturer,” explains René Beck, PhD student and architect of the management solution at Fraunhofer IKS. According to Beck, this not only creates the possibility for the straightforward integration of new or different models into the operations, but also enables a far wider range of tasks, or brings with it a different set of problems to be managed, which a homogeneous multi-robot system can’t solve on its own. From the perspective of the management module, however, this naturally leads to a significant increase in complexity, and specifically the question of which tasks can be solved by which robots, and with what levels of quality and safety.

Another challenge: The management solution is to be designed to cope with highly dynamic environments, such as a construction site. These are hectic places of mixed activity: The environment is in constant flux and also changes completely when moving to a new building site. To cope with such environments, the robots must have a high degree of autonomy that enables them to respond quickly to environmental influences. At the same time, an increased collaboration with the environmental awareness and motion planning helps to avoid critical situations at an early stage. Accordingly, safety factors play an increased role in order to be able to guarantee that both the people and materials are protected.

The use of distributed sensor information is a key component of the solution. One example of this is the dynamic creation of a map from the sensor data of different robots at the same time, known as Collaborative Simultaneous Localization And Mapping (CSLAM). Where the management system comes into play in this case might be switching between robot A to robot B because robot A is unable to cope with the current terrain, for example.

The research approach for the AI-based management system is to be prototyped using the ROS2 middleware and a simulation of the system. This should simplify the handling of the robot networks and guarantee the scalability. ROS2 and the Webots simulation environment offer sufficient possibilities for the evaluation of the approach.

“Safety considerations are necessary”

Three questions for René Beck, PhD student and architect of the management solution


Michael Stiller:

Control systems for driverless transport systems have been around for decades, and now even for vehicles that not only follow a predefined path but can also detect obstacles and avoid them. What do you think these control systems lack, or where do they reach their limits?


René Beck::

The systems usually reach their limits when they leave their familiar, regular and planned operational framework. This happens, for example, in unstructured outdoor environments where it isn’t possible to define how the robot is to perform its task in advance


Michael Stiller:

What is the current status of your work and by when do you expect to have the results of a proof of concept?


René Beck:

We are currently pursuing the goal of creating an easy-to-use simulation environment for multi-robot scenarios. Here, the focus is on providing hardware-related functions across different robots, such as CSLAM for complex applications. This significantly simplifies the development of application scenarios that are based on this. An initial version of the task management should be ready by the end of the year.


Michael Stiller:

Imagine that you have successfully evaluated your management system. What would be the next steps for successfully driving the dissemination of your approach forward?


René Beck:

To elevate our system to a product-proximate status, safety considerations are especially necessary for specific applications. To do this requires more research, however, especially in relation to the interactions between mobile robots and humans.

This project was funded by the Bavarian State Ministry of Economic Affairs, Regional Development and Energy as part of the project thematic support for the Development of the Institute for Cognitive Systems.

Read next

Industrial Automation
Laying the Foundation for a New Manufacturing Paradigm

Alex da silva photo
Alexandre Sawczuk da Silva
Safety engineering / Fraunhofer IKS
Safety engineering