Recent reviews concerned with the Bilingual Advantage (BA) in Executive Functioning report evidence suggesting that research in this area often suffers from methodical flaws and biases favouring results that are both significant and in agreement with previous findings (Paap, Johnson & Sawi, 2015; de Bruin, Treccani & Della Sella, 2015). The present metaanalysis included 102 experiments investigating the BA: 62 reported significant results supporting it, 6 reported significant results showing a Bilingual Disadvantage and 34 showed no significant difference between language groups or mixed results. Thus, 39.22% of published articles are not in support of the BA. The 62 supporting articles were submitted to a p-curve analysis to assess the potential effect of p-hacking (Simonsohn, Nelson, & Simmons, 2014a, 2014b). Results show that the majority of studies find highly significant results (65% p ≤ 0.025), suggesting that evidential value is present, but an increase in number of publications from p = 0.03 to p = 0.05 suggests that p-hacking may have occurred to some extent. Comparison to the underpowered p-curve suggests that the submitted studies suffered from low statistical power, with an average estimated power of 48% (90%-CI: 32% - 63%). Underpowered studies tend to overestimate their effect size (Ioannidis, 2008) and considering the small average effect size (d=0.3) reported in a previous meta-analysis (de Bruin, Treccani & Della Sella, 2015) the magnitude of the BA has to be questioned. This is in agreement with previously reported methodological issues (Paap et al. 2015), but the p-curve suggests that p-hacking, while present to a small extent, is outweighed by highly significant results. The effect of publication bias would consequently be very limited. The same applies to a confirmation bias with 39.22% of published research challenging the BA to some degree. A more serious issue is the combination of low statistical power with small effect sizes, which will need to be investigated further.
|Publication status||Published - 1 Jul 2016|
|Event||UK Cognitive Linguistics Conference - |
Duration: 19 Jul 2016 → 22 Jul 2016
|Conference||UK Cognitive Linguistics Conference|
|Period||19/07/16 → 22/07/16|