Human-Robot Interaction
“So that robots know what to do”

Mobile robots, such as driverless transport systems in the world of logistics, work very efficiently in completing their orders by the dozen – if not the hundred – in enclosed spaces, moving their cargo from A to B at high speed. They complete their tasks horizontally or vertically, whether by lifting sequences or self-propulsion. To prevent collisions, in most cases, a central control system is used that has an overview of all the orders, the network of paths and the positions of the individual robots.

Should problems arise despite this level of control, the robots also have sensors that allow them to detect and avoid obstacles. Most of the environment is “structured,” though: This means that although individual pallets may temporarily block the pathways in a warehouse, the environment – the layout of the hall, including the columns, the hall floor and the shelves – is fixed. Most automated driverless transport systems use similar systems, simplifying the interactions and their interchangeability. System changes and/or the transfer of loads between different mobile robot systems take place at transfer points, whether manual or automated.

And what happens when human employees are brought into the mix?

If the enclosed area is also open to employees, the transport system must be able to meet the corresponding safety requirements; at the expense of the efficiency. In the event of an encounter, the driverless transport systems must reduce their speed, come to a halt, and, if possible, choose an alternative route. Let’s expand this scenario a little further and consider a case of mixed operations in the hall, in which mobile robots, people, conventional forklifts and tugger trains all need to work side by side. And if we also include the current trend of the automated logistics not only taking place in the hall but outside it, the level of complexity increases even further. In such a scenario, a central control system, if not the entire transport system, can rapidly reach its limits – and may even break down.

The researchers at the Fraunhofer Institute for Cognitive Systems IKS have set themselves the task of looking at exactly these scenarios, and significantly improving the processing of orders with the use of heterogeneous mobile robots on the basis of their own management system. With this goal in mind, the researchers aren’t just taking a closer look at the logistics sector, but also at the more chaotic operations on a construction site or the service robotics in the healthcare sector; in hospitals, for example. In all scenarios, specific examples can be found in which problems will arise in the future through the use of heterogeneous robotics, which can only be solved with an AI-supported management system.

What is heterogeneous robotics?

“In a heterogeneous system, robots work together which differ, for example, in terms of their choice of sensors, their design, their capabilities, or even their manufacturer,” explains René Beck, PhD student and architect of the management solution at Fraunhofer IKS. According to Beck, this not only creates the possibility for the straightforward integration of new or different models into the operations, but also enables a far wider range of tasks, or brings with it a different set of problems to be managed, which a homogeneous multi-robot system can’t solve on its own. From the perspective of the management module, however, this naturally leads to a significant increase in complexity, and specifically the question of which tasks can be solved by which robots, and with what levels of quality and safety.

Another challenge: The management solution is to be designed to cope with highly dynamic environments, such as a construction site. These are hectic places of mixed activity: The environment is in constant flux and also changes completely when moving to a new building site. To cope with such environments, the robots must have a high degree of autonomy that enables them to respond quickly to environmental influences. At the same time, an increased collaboration with the environmental awareness and motion planning helps to avoid critical situations at an early stage. Accordingly, safety factors play an increased role in order to be able to guarantee that both the people and materials are protected.

The use of distributed sensor information is a key component of the solution. One example of this is the dynamic creation of a map from the sensor data of different robots at the same time, known as Collaborative Simultaneous Localization And Mapping (CSLAM). Where the management system comes into play in this case might be switching between robot A to robot B because robot A is unable to cope with the current terrain, for example.

The research approach for the AI-based management system is to be prototyped using the ROS2 middleware and a simulation of the system. This should simplify the handling of the robot networks and guarantee the scalability. ROS2 and the Webots simulation environment offer sufficient possibilities for the evaluation of the approach.

“Safety considerations are necessary”

Three questions for René Beck, PhD student and architect of the management solution

This project was funded by the Bavarian State Ministry of Economic Affairs, Regional Development and Energy as part of the project thematic support for the Development of the Institute for Cognitive Systems.