Are methodological quality and completeness of reporting associated with citation-based measures of publication impact? A secondary analysis of a systematic review of dementia biomarker studies

Shona Mackinnon, Bogna A Drozdowska, Michael Hamilton, Anna H Noel-Storr, Rupert McShane, Terry Quinn

Research output: Contribution to journalArticle

2 Citations (Scopus)

Abstract

OBJECTIVE: To determine whether methodological and reporting quality are associated with surrogate measures of publication impact in the field of dementia biomarker studies.

METHODS: We assessed dementia biomarker studies included in a previous systematic review in terms of methodological and reporting quality using the Quality Assessment of Diagnostic Accuracy Studies (QUADAS) and Standards for Reporting of Diagnostic Accuracy (STARD), respectively. We extracted additional study and journal-related data from each publication to account for factors shown to be associated with impact in previous research. We explored associations between potential determinants and measures of publication impact in univariable and stepwise multivariable linear regression analyses.

OUTCOME MEASURES: We aimed to collect data on four measures of publication impact: two traditional measures-average number of citations per year and 5-year impact factor of the publishing journal and two alternative measures-the Altmetric Attention Score and counts of electronic downloads.

RESULTS: The systematic review included 142 studies. Due to limited data, Altmetric Attention Scores and electronic downloads were excluded from the analysis, leaving traditional metrics as the only analysed outcome measures. We found no relationship between QUADAS and traditional metrics. Citation rates were independently associated with 5-year journal impact factor (β=0.42; p<0.001), journal subject area (β=0.39; p<0.001), number of years since publication (β=-0.29; p<0.001) and STARD (β=0.13; p<0.05). Independent determinants of 5-year journal impact factor were citation rates (β=0.45; p<0.001), statement on conflict of interest (β=0.22; p<0.01) and baseline sample size (β=0.15; p<0.05).

CONCLUSIONS: Citation rates and 5-year journal impact factor appear to measure different dimensions of impact. Citation rates were weakly associated with completeness of reporting, while neither traditional metric was related to methodological rigour. Our results suggest that high publication usage and journal outlet is not a guarantee of quality and readers should critically appraise all papers regardless of presumed impact.

LanguageEnglish
Article numbere020331
Pages1-7
Number of pages7
JournalBMJ Open
Volume8
Issue number3
DOIs
Publication statusPublished - 22 Mar 2018

Fingerprint

secondary analysis
Journal Impact Factor
dementia
Dementia
Publications
Biomarkers
diagnostic
Conflict of Interest
Sample Size
electronics
Linear Models
determinants
Regression Analysis
Outcome Assessment (Health Care)
conflict of interest
guarantee
Research
regression

Keywords

  • dementia biomarker studies
  • citation-based measures
  • publication impact

Cite this

Mackinnon, Shona ; Drozdowska, Bogna A ; Hamilton, Michael ; Noel-Storr, Anna H ; McShane, Rupert ; Quinn, Terry. / Are methodological quality and completeness of reporting associated with citation-based measures of publication impact? A secondary analysis of a systematic review of dementia biomarker studies. In: BMJ Open. 2018 ; Vol. 8, No. 3. pp. 1-7.
@article{e6e110ca43214be89dc5c5cae85c9583,
title = "Are methodological quality and completeness of reporting associated with citation-based measures of publication impact? A secondary analysis of a systematic review of dementia biomarker studies",
abstract = "OBJECTIVE: To determine whether methodological and reporting quality are associated with surrogate measures of publication impact in the field of dementia biomarker studies.METHODS: We assessed dementia biomarker studies included in a previous systematic review in terms of methodological and reporting quality using the Quality Assessment of Diagnostic Accuracy Studies (QUADAS) and Standards for Reporting of Diagnostic Accuracy (STARD), respectively. We extracted additional study and journal-related data from each publication to account for factors shown to be associated with impact in previous research. We explored associations between potential determinants and measures of publication impact in univariable and stepwise multivariable linear regression analyses.OUTCOME MEASURES: We aimed to collect data on four measures of publication impact: two traditional measures-average number of citations per year and 5-year impact factor of the publishing journal and two alternative measures-the Altmetric Attention Score and counts of electronic downloads.RESULTS: The systematic review included 142 studies. Due to limited data, Altmetric Attention Scores and electronic downloads were excluded from the analysis, leaving traditional metrics as the only analysed outcome measures. We found no relationship between QUADAS and traditional metrics. Citation rates were independently associated with 5-year journal impact factor (β=0.42; p<0.001), journal subject area (β=0.39; p<0.001), number of years since publication (β=-0.29; p<0.001) and STARD (β=0.13; p<0.05). Independent determinants of 5-year journal impact factor were citation rates (β=0.45; p<0.001), statement on conflict of interest (β=0.22; p<0.01) and baseline sample size (β=0.15; p<0.05).CONCLUSIONS: Citation rates and 5-year journal impact factor appear to measure different dimensions of impact. Citation rates were weakly associated with completeness of reporting, while neither traditional metric was related to methodological rigour. Our results suggest that high publication usage and journal outlet is not a guarantee of quality and readers should critically appraise all papers regardless of presumed impact.",
keywords = "dementia biomarker studies, citation-based measures, publication impact",
author = "Shona Mackinnon and Drozdowska, {Bogna A} and Michael Hamilton and Noel-Storr, {Anna H} and Rupert McShane and Terry Quinn",
note = "{\circledC} Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2018. All rights reserved. No commercial use is permitted unless otherwise expressly granted.",
year = "2018",
month = "3",
day = "22",
doi = "10.1136/bmjopen-2017-020331",
language = "English",
volume = "8",
pages = "1--7",
journal = "BMJ Open",
issn = "2044-6055",
number = "3",

}

Are methodological quality and completeness of reporting associated with citation-based measures of publication impact? A secondary analysis of a systematic review of dementia biomarker studies. / Mackinnon, Shona; Drozdowska, Bogna A; Hamilton, Michael; Noel-Storr, Anna H; McShane, Rupert; Quinn, Terry.

In: BMJ Open, Vol. 8, No. 3, e020331, 22.03.2018, p. 1-7.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Are methodological quality and completeness of reporting associated with citation-based measures of publication impact? A secondary analysis of a systematic review of dementia biomarker studies

AU - Mackinnon, Shona

AU - Drozdowska, Bogna A

AU - Hamilton, Michael

AU - Noel-Storr, Anna H

AU - McShane, Rupert

AU - Quinn, Terry

N1 - © Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2018. All rights reserved. No commercial use is permitted unless otherwise expressly granted.

PY - 2018/3/22

Y1 - 2018/3/22

N2 - OBJECTIVE: To determine whether methodological and reporting quality are associated with surrogate measures of publication impact in the field of dementia biomarker studies.METHODS: We assessed dementia biomarker studies included in a previous systematic review in terms of methodological and reporting quality using the Quality Assessment of Diagnostic Accuracy Studies (QUADAS) and Standards for Reporting of Diagnostic Accuracy (STARD), respectively. We extracted additional study and journal-related data from each publication to account for factors shown to be associated with impact in previous research. We explored associations between potential determinants and measures of publication impact in univariable and stepwise multivariable linear regression analyses.OUTCOME MEASURES: We aimed to collect data on four measures of publication impact: two traditional measures-average number of citations per year and 5-year impact factor of the publishing journal and two alternative measures-the Altmetric Attention Score and counts of electronic downloads.RESULTS: The systematic review included 142 studies. Due to limited data, Altmetric Attention Scores and electronic downloads were excluded from the analysis, leaving traditional metrics as the only analysed outcome measures. We found no relationship between QUADAS and traditional metrics. Citation rates were independently associated with 5-year journal impact factor (β=0.42; p<0.001), journal subject area (β=0.39; p<0.001), number of years since publication (β=-0.29; p<0.001) and STARD (β=0.13; p<0.05). Independent determinants of 5-year journal impact factor were citation rates (β=0.45; p<0.001), statement on conflict of interest (β=0.22; p<0.01) and baseline sample size (β=0.15; p<0.05).CONCLUSIONS: Citation rates and 5-year journal impact factor appear to measure different dimensions of impact. Citation rates were weakly associated with completeness of reporting, while neither traditional metric was related to methodological rigour. Our results suggest that high publication usage and journal outlet is not a guarantee of quality and readers should critically appraise all papers regardless of presumed impact.

AB - OBJECTIVE: To determine whether methodological and reporting quality are associated with surrogate measures of publication impact in the field of dementia biomarker studies.METHODS: We assessed dementia biomarker studies included in a previous systematic review in terms of methodological and reporting quality using the Quality Assessment of Diagnostic Accuracy Studies (QUADAS) and Standards for Reporting of Diagnostic Accuracy (STARD), respectively. We extracted additional study and journal-related data from each publication to account for factors shown to be associated with impact in previous research. We explored associations between potential determinants and measures of publication impact in univariable and stepwise multivariable linear regression analyses.OUTCOME MEASURES: We aimed to collect data on four measures of publication impact: two traditional measures-average number of citations per year and 5-year impact factor of the publishing journal and two alternative measures-the Altmetric Attention Score and counts of electronic downloads.RESULTS: The systematic review included 142 studies. Due to limited data, Altmetric Attention Scores and electronic downloads were excluded from the analysis, leaving traditional metrics as the only analysed outcome measures. We found no relationship between QUADAS and traditional metrics. Citation rates were independently associated with 5-year journal impact factor (β=0.42; p<0.001), journal subject area (β=0.39; p<0.001), number of years since publication (β=-0.29; p<0.001) and STARD (β=0.13; p<0.05). Independent determinants of 5-year journal impact factor were citation rates (β=0.45; p<0.001), statement on conflict of interest (β=0.22; p<0.01) and baseline sample size (β=0.15; p<0.05).CONCLUSIONS: Citation rates and 5-year journal impact factor appear to measure different dimensions of impact. Citation rates were weakly associated with completeness of reporting, while neither traditional metric was related to methodological rigour. Our results suggest that high publication usage and journal outlet is not a guarantee of quality and readers should critically appraise all papers regardless of presumed impact.

KW - dementia biomarker studies

KW - citation-based measures

KW - publication impact

U2 - 10.1136/bmjopen-2017-020331

DO - 10.1136/bmjopen-2017-020331

M3 - Article

VL - 8

SP - 1

EP - 7

JO - BMJ Open

T2 - BMJ Open

JF - BMJ Open

SN - 2044-6055

IS - 3

M1 - e020331

ER -