Multi-Agent Reinforcement Learning
When Intelligent Systems Must Cooperate Autonomously

Unmanned aerial vehicles (UAVs) coordinating the establishment of a communications network in an area devastated by an earthquake, autonomous ground vehicles navigating in formation through logistically demanding terrain, maritime systems working together to detect and counter underwater threats. In all of these aforementioned scenarios, autonomous systems should in the future be able to coordinate themselves in order to reliably achieve mission success.

A promising approach is Multi-Agent Reinforcement Learning (MARL), which can enable the self-management of autonomous teams.

The range of possible tasks for such teams is enormous:

• Swarm coordination following a disaster: Drones coordinate themselves for search and rescue missions in order to cover the largest possible areas.
• Logistics missions in contested environments: Autonomous unmanned ground vehicles independently coordinate the supply of bases located in contested and difficult-to-reach areas.
• AI agents planning and executing military missions: Various AI agents covering specialized domains such as available geospatial intelligence, assessment of friendly forces, or evaluation of the current situation combine their partial observations to contribute to the best possible planning and execution of military missions.

Reinforcement Learning as a game changer

In many real-world problems, recognizing that a task has been completed is straightforward, but explicitly defining how to achieve it is difficult. In such cases, it can be a good idea to let a machine learning agent interact with its environment and learn through “trial and error”. Reinforcement Learning (RL) is a machine learning paradigm that enables us to do exactly that. In its standard single agent form, an agent is placed in an environment (often a simulated environment at first) where it is equipped to perform certain actions leading to changes in its state (e.g. a robot moves forward in a maze, thereby changing its coordinates and receiving new sensor readings) and receives a reward (e.g. whether it came closer to the goal position). As the agent continues to interact with its environment and observes the resulting rewards, the agent gradually improves its action policy (its strategy for choosing an optimal action depending on the environmental state). This paradigm has been highly successful and is widely adopted in controlled settings, such as manufacturing and robotics.

MARL extends single-agent RL to scenarios where multiple agents share the same environment. These agents are tasked not only with maximizing their individual rewards, but also with optimizing a shared or collective objective. Depending on the relationship between agents’ objectives, different types of MARL can be distinguished: cooperative (all agents share the same goal), competitive (agents have opposing goals) and mixed (agents within the same team share a common goal, but their team’s goal conflicts with that of another team).

Outlook: The Future Belongs to Self-Organized Autonomous Teams

Autonomous systems such as drone swarms or AI-assisted C2 (command and control) systems will become indispensable across many areas of internal and external security in the years ahead. However, the goal of reliably and transparently operating intelligent systems can only be achieved if steady progress continues to be made in the following areas:

• Advanced safety assurance concepts capable of keeping pace with the growing complexity of individual systems and systems of systems (SoS).
• Safe and secure Human-AI teaming, in which humans retain control over critical decisions wherever possible (human in the loop), without sacrificing the gains in speed and efficiency achieved by autonomous teams.
• Resilient and robust systems that are best protected against attacks such as adversarial attacks as well as adverse environmental conditions.

Across all of these research areas, the Fraunhofer IKS is working together with its partners to safely bring intelligent systems into operational use in the domains of internal and external security.