Reinforcement learning task planner for construction task, assisted by LLM

Construction sites are non-deterministic environments where traditional tasks planning techniques for multi-robot systems do not work well for long term actions. Constant changes in the environment force a continuous update and evaluation of the state of the system. The paper proposes a reinforcement learning agent assisted by multi-modal foundation model agents to target tasks planning in construction sites. A natural language user request involving tools, consumables, locations and actions is used to command a robot system in the environment. The foundation model agents assist in the identification of relevant information in the user request, the selection of actions, and the object identification. The goal of the system is to execute the desired task at the proper location with the necessary tools and consumables. The proposed system is able to perform in environments with different number of locations and under user requests containing different number of tools, consumables and actions.