Creating and Applying an Evaluation Framework for the National Decision Support Programme in Scotland

Kathrin Cresswell, Margaret Callaghan, Hajar Mozaffar, Aziz Sheikh, George Crooks (Editor), Digital Health & Care Institute (DHI)

Research output: Book/ReportCommissioned report

Abstract

Context: The Scottish Government recognises the importance of decision support to improve knowledge management in health and care settings as a strategic priority. To this end, they funded the 2015 National Decision Support Roadmap. This laid out a plan for procuring and building a Decision Support Platform delivering a range of small-scale demonstrators (including several mobile platforms for specific user groups e.g. polypharmacy and diabetes), and building clinician and policy engagement for further funding.

Aims: We were commissioned to undertake a formative evaluation of the National Decision Support Programme to help facilitate the effective roll-out of systems included in the Roadmap more widely.

Methods: We collected qualitative data through a series of in-depth interviews and observations of workshops demonstrating technological systems. Participants included policy makers and clinical leads involved in the National Decision Support Programme. As the Programme was in the early stages of strategy development and system implementation at the time of data collection, we focused on exploring expectations and drivers of Cambio (a pilot platform) being tested in primary care. This system delivers an open standards based algorithm editor and engine which is linked with bespoke decision support applications delivered as web and mobile products and integrated into primary care electronic health record systems. The web and mobile solutions linked to the Cambio algorithms platform were developed by Scottish partners (Tactuum and University of West of Scotland).

Employing a flexible methodological approach tailored to changing circumstances and need offered important opportunities for realising true impact through ongoing formative feedback to policymakers and active engagement of key clinical stakeholders. Our work was informed by sociotechnical principles and a health information infrastructure perspective. Qualitative data were coded with the help of NVivo software and analysed through a combination of inductive and deductive approaches.

Findings: We collected data through 30 interviews and eight non-participant ethnographic observations of early stakeholder engagement workshops. We developed and applied a theoretically-informed evaluation framework, which we refined throughout our analysis.

Overall, we observed a strong sense of support from all stakeholders for Cambio as an exemplar of an open standards based, customisable decision support platform, and proposals to roll this model out across NHS Scotland. Strategic drivers included facilitating integration of care, preventative care, patient self-management, shared decision-making, patient engagement, and the availability of information. However, in order to achieve desired benefits, participants highlighted the need for strong national leadership, system usability (which was perceived to be negatively affected by alert fatigue and integration with existing systems), and ongoing monitoring of potential unintended consequences emerging from implementations (e.g. clinical workloads).

Conclusions and implications: In order to address potential tensions between national leadership and local usability as well as unintended consequences, there is a need to have overall national ownership to support the implementation of the Roadmap, whilst the implementation of individual applications needs to be devolved. This could be achieved through allowing a degree of local customisation of systems and tailoring of alerts, ongoing system development with continuing stakeholder engagement including “hands-on” experience for clinicians, a limited number of pilots that are carefully evaluated to mitigate emerging risks early, and development of a nuanced benefits realisation framework that combines smaller and locally relevant measures determined by implementing sites with national progress measures.
LanguageEnglish
Place of PublicationGlasgow
Number of pages38
DOIs
Publication statusPublished - 15 May 2019

Fingerprint

Scotland
Primary Health Care
Health
Interviews
Knowledge Management
Education
Patient Participation
Polypharmacy
Preventive Medicine
Ownership
Electronic Health Records
Self Care
Workload
Administrative Personnel
Fatigue
Decision Making
Software
Knowledge management
Medical problems
Delivery of Health Care

Keywords

  • digital health
  • decision support systems
  • health care
  • information systems
  • health analytics
  • medical record systems
  • NHS Scotland
  • DSS
  • electronic health records
  • EHR

Cite this

Cresswell, K., Callaghan, M., Mozaffar, H., Sheikh, A., & Digital Health & Care Institute (DHI) (2019). Creating and Applying an Evaluation Framework for the National Decision Support Programme in Scotland. Glasgow. https://doi.org/10.17868/67863
Cresswell, Kathrin ; Callaghan, Margaret ; Mozaffar, Hajar ; Sheikh, Aziz ; Digital Health & Care Institute (DHI) . / Creating and Applying an Evaluation Framework for the National Decision Support Programme in Scotland. Glasgow, 2019. 38 p.
@book{e5494ab578464b5f9453c1cdc4c203fc,
title = "Creating and Applying an Evaluation Framework for the National Decision Support Programme in Scotland",
abstract = "Context: The Scottish Government recognises the importance of decision support to improve knowledge management in health and care settings as a strategic priority. To this end, they funded the 2015 National Decision Support Roadmap. This laid out a plan for procuring and building a Decision Support Platform delivering a range of small-scale demonstrators (including several mobile platforms for specific user groups e.g. polypharmacy and diabetes), and building clinician and policy engagement for further funding. Aims: We were commissioned to undertake a formative evaluation of the National Decision Support Programme to help facilitate the effective roll-out of systems included in the Roadmap more widely. Methods: We collected qualitative data through a series of in-depth interviews and observations of workshops demonstrating technological systems. Participants included policy makers and clinical leads involved in the National Decision Support Programme. As the Programme was in the early stages of strategy development and system implementation at the time of data collection, we focused on exploring expectations and drivers of Cambio (a pilot platform) being tested in primary care. This system delivers an open standards based algorithm editor and engine which is linked with bespoke decision support applications delivered as web and mobile products and integrated into primary care electronic health record systems. The web and mobile solutions linked to the Cambio algorithms platform were developed by Scottish partners (Tactuum and University of West of Scotland).Employing a flexible methodological approach tailored to changing circumstances and need offered important opportunities for realising true impact through ongoing formative feedback to policymakers and active engagement of key clinical stakeholders. Our work was informed by sociotechnical principles and a health information infrastructure perspective. Qualitative data were coded with the help of NVivo software and analysed through a combination of inductive and deductive approaches. Findings: We collected data through 30 interviews and eight non-participant ethnographic observations of early stakeholder engagement workshops. We developed and applied a theoretically-informed evaluation framework, which we refined throughout our analysis. Overall, we observed a strong sense of support from all stakeholders for Cambio as an exemplar of an open standards based, customisable decision support platform, and proposals to roll this model out across NHS Scotland. Strategic drivers included facilitating integration of care, preventative care, patient self-management, shared decision-making, patient engagement, and the availability of information. However, in order to achieve desired benefits, participants highlighted the need for strong national leadership, system usability (which was perceived to be negatively affected by alert fatigue and integration with existing systems), and ongoing monitoring of potential unintended consequences emerging from implementations (e.g. clinical workloads). Conclusions and implications: In order to address potential tensions between national leadership and local usability as well as unintended consequences, there is a need to have overall national ownership to support the implementation of the Roadmap, whilst the implementation of individual applications needs to be devolved. This could be achieved through allowing a degree of local customisation of systems and tailoring of alerts, ongoing system development with continuing stakeholder engagement including “hands-on” experience for clinicians, a limited number of pilots that are carefully evaluated to mitigate emerging risks early, and development of a nuanced benefits realisation framework that combines smaller and locally relevant measures determined by implementing sites with national progress measures.",
keywords = "digital health, decision support systems, health care, information systems, health analytics, medical record systems, NHS Scotland, DSS, electronic health records, EHR",
author = "Kathrin Cresswell and Margaret Callaghan and Hajar Mozaffar and Aziz Sheikh and {Digital Health & Care Institute (DHI)}",
editor = "George Crooks",
year = "2019",
month = "5",
day = "15",
doi = "10.17868/67863",
language = "English",

}

Cresswell, K, Callaghan, M, Mozaffar, H, Sheikh, A & Digital Health & Care Institute (DHI) 2019, Creating and Applying an Evaluation Framework for the National Decision Support Programme in Scotland. Glasgow. https://doi.org/10.17868/67863

Creating and Applying an Evaluation Framework for the National Decision Support Programme in Scotland. / Cresswell, Kathrin; Callaghan, Margaret; Mozaffar, Hajar; Sheikh, Aziz; Digital Health & Care Institute (DHI) .

Glasgow, 2019. 38 p.

Research output: Book/ReportCommissioned report

TY - BOOK

T1 - Creating and Applying an Evaluation Framework for the National Decision Support Programme in Scotland

AU - Cresswell, Kathrin

AU - Callaghan, Margaret

AU - Mozaffar, Hajar

AU - Sheikh, Aziz

AU - Digital Health & Care Institute (DHI)

A2 - Crooks, George

PY - 2019/5/15

Y1 - 2019/5/15

N2 - Context: The Scottish Government recognises the importance of decision support to improve knowledge management in health and care settings as a strategic priority. To this end, they funded the 2015 National Decision Support Roadmap. This laid out a plan for procuring and building a Decision Support Platform delivering a range of small-scale demonstrators (including several mobile platforms for specific user groups e.g. polypharmacy and diabetes), and building clinician and policy engagement for further funding. Aims: We were commissioned to undertake a formative evaluation of the National Decision Support Programme to help facilitate the effective roll-out of systems included in the Roadmap more widely. Methods: We collected qualitative data through a series of in-depth interviews and observations of workshops demonstrating technological systems. Participants included policy makers and clinical leads involved in the National Decision Support Programme. As the Programme was in the early stages of strategy development and system implementation at the time of data collection, we focused on exploring expectations and drivers of Cambio (a pilot platform) being tested in primary care. This system delivers an open standards based algorithm editor and engine which is linked with bespoke decision support applications delivered as web and mobile products and integrated into primary care electronic health record systems. The web and mobile solutions linked to the Cambio algorithms platform were developed by Scottish partners (Tactuum and University of West of Scotland).Employing a flexible methodological approach tailored to changing circumstances and need offered important opportunities for realising true impact through ongoing formative feedback to policymakers and active engagement of key clinical stakeholders. Our work was informed by sociotechnical principles and a health information infrastructure perspective. Qualitative data were coded with the help of NVivo software and analysed through a combination of inductive and deductive approaches. Findings: We collected data through 30 interviews and eight non-participant ethnographic observations of early stakeholder engagement workshops. We developed and applied a theoretically-informed evaluation framework, which we refined throughout our analysis. Overall, we observed a strong sense of support from all stakeholders for Cambio as an exemplar of an open standards based, customisable decision support platform, and proposals to roll this model out across NHS Scotland. Strategic drivers included facilitating integration of care, preventative care, patient self-management, shared decision-making, patient engagement, and the availability of information. However, in order to achieve desired benefits, participants highlighted the need for strong national leadership, system usability (which was perceived to be negatively affected by alert fatigue and integration with existing systems), and ongoing monitoring of potential unintended consequences emerging from implementations (e.g. clinical workloads). Conclusions and implications: In order to address potential tensions between national leadership and local usability as well as unintended consequences, there is a need to have overall national ownership to support the implementation of the Roadmap, whilst the implementation of individual applications needs to be devolved. This could be achieved through allowing a degree of local customisation of systems and tailoring of alerts, ongoing system development with continuing stakeholder engagement including “hands-on” experience for clinicians, a limited number of pilots that are carefully evaluated to mitigate emerging risks early, and development of a nuanced benefits realisation framework that combines smaller and locally relevant measures determined by implementing sites with national progress measures.

AB - Context: The Scottish Government recognises the importance of decision support to improve knowledge management in health and care settings as a strategic priority. To this end, they funded the 2015 National Decision Support Roadmap. This laid out a plan for procuring and building a Decision Support Platform delivering a range of small-scale demonstrators (including several mobile platforms for specific user groups e.g. polypharmacy and diabetes), and building clinician and policy engagement for further funding. Aims: We were commissioned to undertake a formative evaluation of the National Decision Support Programme to help facilitate the effective roll-out of systems included in the Roadmap more widely. Methods: We collected qualitative data through a series of in-depth interviews and observations of workshops demonstrating technological systems. Participants included policy makers and clinical leads involved in the National Decision Support Programme. As the Programme was in the early stages of strategy development and system implementation at the time of data collection, we focused on exploring expectations and drivers of Cambio (a pilot platform) being tested in primary care. This system delivers an open standards based algorithm editor and engine which is linked with bespoke decision support applications delivered as web and mobile products and integrated into primary care electronic health record systems. The web and mobile solutions linked to the Cambio algorithms platform were developed by Scottish partners (Tactuum and University of West of Scotland).Employing a flexible methodological approach tailored to changing circumstances and need offered important opportunities for realising true impact through ongoing formative feedback to policymakers and active engagement of key clinical stakeholders. Our work was informed by sociotechnical principles and a health information infrastructure perspective. Qualitative data were coded with the help of NVivo software and analysed through a combination of inductive and deductive approaches. Findings: We collected data through 30 interviews and eight non-participant ethnographic observations of early stakeholder engagement workshops. We developed and applied a theoretically-informed evaluation framework, which we refined throughout our analysis. Overall, we observed a strong sense of support from all stakeholders for Cambio as an exemplar of an open standards based, customisable decision support platform, and proposals to roll this model out across NHS Scotland. Strategic drivers included facilitating integration of care, preventative care, patient self-management, shared decision-making, patient engagement, and the availability of information. However, in order to achieve desired benefits, participants highlighted the need for strong national leadership, system usability (which was perceived to be negatively affected by alert fatigue and integration with existing systems), and ongoing monitoring of potential unintended consequences emerging from implementations (e.g. clinical workloads). Conclusions and implications: In order to address potential tensions between national leadership and local usability as well as unintended consequences, there is a need to have overall national ownership to support the implementation of the Roadmap, whilst the implementation of individual applications needs to be devolved. This could be achieved through allowing a degree of local customisation of systems and tailoring of alerts, ongoing system development with continuing stakeholder engagement including “hands-on” experience for clinicians, a limited number of pilots that are carefully evaluated to mitigate emerging risks early, and development of a nuanced benefits realisation framework that combines smaller and locally relevant measures determined by implementing sites with national progress measures.

KW - digital health

KW - decision support systems

KW - health care

KW - information systems

KW - health analytics

KW - medical record systems

KW - NHS Scotland

KW - DSS

KW - electronic health records

KW - EHR

U2 - 10.17868/67863

DO - 10.17868/67863

M3 - Commissioned report

BT - Creating and Applying an Evaluation Framework for the National Decision Support Programme in Scotland

CY - Glasgow

ER -

Cresswell K, Callaghan M, Mozaffar H, Sheikh A, Digital Health & Care Institute (DHI) . Creating and Applying an Evaluation Framework for the National Decision Support Programme in Scotland. Glasgow, 2019. 38 p. https://doi.org/10.17868/67863