Designing an adaptable benchmark and competition simulation for integrated planning and execution

Effectively using a planning system as the executive of an agent acting in real time poses a variety of challenges in integrating planning and execution. Many integrated systems have been developed with a focus on particular challenges, and it has been typically difficult to test, benchmark, and compare these systems. To do so requires a benchmark that has transparent and well-defined rules, and can be adapted to exhibit the problem characteristics of interest. In this paper, we propose a new benchmark simulation for integrated planning and execution, designed to be accessible and adaptive. We describe the simple core scenario of the simulation and how it can be configured to present more challenging scenarios. We describe our plans for the development of the simulation as a competition, benchmarking, and teaching tool, and encourage the community to contribute to its design.