Benchmarking Robustness and Generalization in Multi-Agent Systems

A Case Study on Neural MMO

Journal article (2023)

Authors

Yangkun Chen Parametrix.ai, Tsinghua University
Chenghui Yu Parametrix.ai, Tsinghua University
Hengman Zhu Parametrix.ai
Shuai Liu Bilibili
Yibing Zhang Chengdu Goldwin Electronics Technology
Joseph Suarez Massachusetts Institute of Technology
Liang Zhao International Digital Economy Academy
J. He Interactive Intelligence - EEMCS
Jiaxin Chen Parametrix.ai
More Authors... External organisation
Research Group
Interactive Intelligence (EEMCS) (TU Delft)
Copyright: © 2023 Yangkun Chen, Chenghui Yu, Hengman Zhu, Shuai Liu, Yibing Zhang, Joseph Suarez, Liang Zhao, J. He, Jiaxin Chen, More Authors
More Info
expand_more

Published Date

2023

Language

English

Reuse Rights

Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.

Faculty

Electrical Engineering, Mathematics and Computer Science

Department

Intelligent Systems

Research Group

Interactive Intelligence

Abstract

We present the results of the second Neural MMO challenge, hosted at IJCAI 2022, which received 1600+ submissions. This competition targets robustness and generalization in multi-agent systems: participants train teams of agents to complete a multi-task objective against opponents not seen during training. We summarize the competition design and results and suggest that, considering our work as a case study, competitions are an effective approach to solving hard problems and establishing a solid benchmark for algorithms. We will open-source our benchmark including the environment wrapper, baselines, a visualization tool, and selected policies for further research.

Files

3545946.3598978.pdf
(pdf | 3.42 Mb)
- Embargo expired in 27-11-2023
Unknown license