All eyes on the ranker: Participatory auditing to surface blind spots in ranked search results

Search engines that present users with a ranked list of search results are a fundamental technology for providing public access to information. Evaluations of such systems are typically conducted by domain experts and focus on model-centric metrics, relevance judgments, or output-based analyses, rather than on how accountability, harm, or trust are experienced by users. This paper argues that participatory auditing is essential for revealing users’ causal and contextual understandings of how ranked search results produce impacts, particularly as ranking models appear increasingly convincing and sophisticated in their semantic interpretation of user queries. We report on three participatory auditing workshops (n=21) in which participants engaged with a custom search interface across four tasks, comparing a lexical ranker (BM25) and a neural semantic reranker (MonoT5), exploring varying levels of transparency and user controls, and examining an intentionally adversarially manipulated ranking. Reflexive activities prompted participants to articulate causal narratives linking to search system properties, rather than merely evaluating result quality. They discussed what they considered to be desirable properties of search results, such as relevance, fairness, and transparency, and how these shape users' understandings of search rankings. Synthesising the findings, we contribute a taxonomy of user-perceived impacts of ranked search results, spanning epistemic, representational, infrastructural, and downstream social impacts. At the same time, interactions with the more convincing neural ranking model revealed potential limits to participatory auditing itself: perceived system competence and accumulated trust reduced critical scrutiny during the workshop, allowing manipulations to go undetected. Participants repeatedly expressed a desire to gain visibility into the full search pipeline, comprising data corpus, ranking model and logic, as well as recourse mechanisms, in order to meaningfully audit the system. Together, these findings show how participatory auditing can surface user perceived impacts and accountability gaps that remain unseen when relying on conventional audits, while also revealing where participatory auditing may encounter challenges and limitations, thus requiring refinement.