The "analysis of competing hypotheses" in intelligence analysis

Mandeep K. Dhami, Ian K. Belton, David R. Mandel

Research output: Contribution to journalArticle

Abstract

The intelligence community uses "structured analytic techniques" to help analysts think critically and avoid cognitive bias. However, little evidence exists of how techniques are applied and whether they are effective. We examined the use of the analysis of competing hypotheses (ACH)—a technique designed to reduce "confirmation bias". Fifty intelligence analysts were randomly assigned to use ACH or not when completing a hypothesis testing task that had probabilistic ground truth. Data on analysts' judgement processes and conclusions were collected using written protocols that were then coded for statistical analyses. We found that ACH-trained analysts did not follow all of the steps of ACH. There was mixed evidence for ACH's ability to reduce confirmation bias, and we observed that ACH may increase judgement inconsistency and error. It may be prudent for the intelligence community to consider the conditions under which ACH would prove useful and to explore alternatives.

LanguageEnglish
JournalApplied Cognitive Psychology
Early online date21 Mar 2019
DOIs
Publication statusE-pub ahead of print - 21 Mar 2019

Fingerprint

Intelligence
Aptitude

Keywords

  • confirmation bias
  • hypothesis testing
  • intelligence analysis
  • judgement and decision making

Cite this

@article{cb63b265742a431c8f5d425ec717f9c1,
title = "The {"}analysis of competing hypotheses{"} in intelligence analysis",
abstract = "The intelligence community uses {"}structured analytic techniques{"} to help analysts think critically and avoid cognitive bias. However, little evidence exists of how techniques are applied and whether they are effective. We examined the use of the analysis of competing hypotheses (ACH)—a technique designed to reduce {"}confirmation bias{"}. Fifty intelligence analysts were randomly assigned to use ACH or not when completing a hypothesis testing task that had probabilistic ground truth. Data on analysts' judgement processes and conclusions were collected using written protocols that were then coded for statistical analyses. We found that ACH-trained analysts did not follow all of the steps of ACH. There was mixed evidence for ACH's ability to reduce confirmation bias, and we observed that ACH may increase judgement inconsistency and error. It may be prudent for the intelligence community to consider the conditions under which ACH would prove useful and to explore alternatives.",
keywords = "confirmation bias, hypothesis testing, intelligence analysis, judgement and decision making",
author = "Dhami, {Mandeep K.} and Belton, {Ian K.} and Mandel, {David R.}",
year = "2019",
month = "3",
day = "21",
doi = "10.1002/acp.3550",
language = "English",
journal = "Applied Cognitive Psychology",
issn = "0888-4080",

}

The "analysis of competing hypotheses" in intelligence analysis. / Dhami, Mandeep K.; Belton, Ian K.; Mandel, David R.

In: Applied Cognitive Psychology, 21.03.2019.

Research output: Contribution to journalArticle

TY - JOUR

T1 - The "analysis of competing hypotheses" in intelligence analysis

AU - Dhami, Mandeep K.

AU - Belton, Ian K.

AU - Mandel, David R.

PY - 2019/3/21

Y1 - 2019/3/21

N2 - The intelligence community uses "structured analytic techniques" to help analysts think critically and avoid cognitive bias. However, little evidence exists of how techniques are applied and whether they are effective. We examined the use of the analysis of competing hypotheses (ACH)—a technique designed to reduce "confirmation bias". Fifty intelligence analysts were randomly assigned to use ACH or not when completing a hypothesis testing task that had probabilistic ground truth. Data on analysts' judgement processes and conclusions were collected using written protocols that were then coded for statistical analyses. We found that ACH-trained analysts did not follow all of the steps of ACH. There was mixed evidence for ACH's ability to reduce confirmation bias, and we observed that ACH may increase judgement inconsistency and error. It may be prudent for the intelligence community to consider the conditions under which ACH would prove useful and to explore alternatives.

AB - The intelligence community uses "structured analytic techniques" to help analysts think critically and avoid cognitive bias. However, little evidence exists of how techniques are applied and whether they are effective. We examined the use of the analysis of competing hypotheses (ACH)—a technique designed to reduce "confirmation bias". Fifty intelligence analysts were randomly assigned to use ACH or not when completing a hypothesis testing task that had probabilistic ground truth. Data on analysts' judgement processes and conclusions were collected using written protocols that were then coded for statistical analyses. We found that ACH-trained analysts did not follow all of the steps of ACH. There was mixed evidence for ACH's ability to reduce confirmation bias, and we observed that ACH may increase judgement inconsistency and error. It may be prudent for the intelligence community to consider the conditions under which ACH would prove useful and to explore alternatives.

KW - confirmation bias

KW - hypothesis testing

KW - intelligence analysis

KW - judgement and decision making

UR - http://www.scopus.com/inward/record.url?scp=85066618078&partnerID=8YFLogxK

U2 - 10.1002/acp.3550

DO - 10.1002/acp.3550

M3 - Article

JO - Applied Cognitive Psychology

T2 - Applied Cognitive Psychology

JF - Applied Cognitive Psychology

SN - 0888-4080

ER -