The ethics of psychological manipulation in adversarial conversational AI: confronting the recognition-behaviour gap

Conversational AI systems, powered by advanced Large Language Models, have rapidly developed human-like persuasion capabilities that raise concerns about psychological manipulation. This provocation examines the ethical problems that arise when these systems exploit cognitive biases and social compliance mechanisms during interactions with users. Building on established theoretical work and recent empirical research, we identify a particularly concerning pattern: the recognition-behaviour gap, where users consciously identify manipulative strategies yet fail to protect themselves accordingly. Current ethical frameworks fall short in addressing these sophisticated risks in conversational contexts. We propose a targeted ethical framework for AI governance centred on four key dimensions: preserving user autonomy, enhancing transparency, systematically monitoring for vulnerabilities, and implementing contextual safeguards. This paper confronts these ethical challenges directly and calls for practical protective measures to safeguard user autonomy as conversational AI becomes increasingly prevalent in everyday life.