MICA: a multimodal intelligent cognitive assessment framework integrating generative AI and social robot for early cognitive intervention

The rapidly growing older adult population underscores the urgent need for innovative solutions to detect, classify, and monitor early cognitive decline. Traditional cognitive screening methods, such as paper-and-pencil tests like the Montreal Cognitive Assessment (MoCA) and Mini-Mental State Examination (MMSE), suffer from notable limitations, including low patient engagement, limited motivational appeal, inadequate sensitivity to subtle cognitive decline, and susceptibility to practice effects with repeated administrations. Furthermore, such tests require substantial clinician time, as they must be administered by trained healthcare practitioners, increasing healthcare costs. Given the current shortage of effective interventions and reliable screening methods for early-stage cognitive decline, socially assistive robots offer a promising dual function: They can deliver personalized and engaging cognitive stimulation and social support while monitoring cognitive health. This paper proposes a new Multimodal Intelligent Cognitive Assessment (MICA) framework integrated into the Pepper social robot and enhanced by generative AI technologies. MICA consists of three core components:(1) a conversational and cognitive exercise interface powered by generative AI, enabling natural, engaging interactions tailored to various cognitive domains; (2) multimodal perception capabilities, incorporating emotion recognition using DeepFace, robust speech recognition, and real-time personalized adaptation based on emotional and cognitive feedback, and (3) an advanced performance logging system designed to systematically record patient accuracy, response time, and emotional states. Initial evaluations with Pepper demonstrated real-time emotion detection and adaptive exercises, which illustrate the potential for high levels of engagement and early intervention. Although current evaluations were conducted by the authors, more comprehensive user studies are planned to validate the effectiveness of MICA within the populations of interest.