What Do We Know About What We Know? A Review of Behaviour-Based Energy Efficiency Data Collection Methodology

Behaviour-based energy interventions (i.e., programmes targeting savings through consumer energy use) are based on the idea that people can be encouraged to use less energy if the underlying determinants of behaviour change in some way. Research on such programmes suggests potential savings, but results vary and much is still unknown about the specific variables that impact programme effectiveness. This is due in part to the way these programmes are typically evaluated.
Most energy efficiency evaluations use changes in the amount of energy consumed (measured in kWh) as the dependent variable for determining effectiveness. Although this is an ideal measure of
whether energy efficiency interventions work, additional information could add significantly to our understanding about how and for whom they work. Recent efforts have been made to include such information and more and more studies are now collecting self-reported data from participants in order to better understand how and for whom behaviour-based energy interventions work.
However, widespread agreement on what data to collect and how to collect it is still lacking. Such standardisation is common in related fields such as education and psychology, but has yet to take hold in energy programme evaluation. The development of consistent, validated measures would improve our overall ability to account for variation in treatment effects and improve programme functioning and delivery.
As such, the current report presents a methodological review of behaviour-based energy intervention studies in the customer feedback and residential building retrofit areas, which were conducted over the past 10 years to determine what data has been collected and how it has been collected. This review will form the basis of further work undertaken by the study authors and for Subtask 9 of the IEA DSM Task 24 extension (Phase II – see www.ieadsm.org/task/task-24-phase-2/).
The work presented here suggests that future research should evaluate programmes using standardised measures across a range of key variables. The use of standard measures would enable cross-comparisons to be made across different studies, and the incorporation of questions about context, behaviours, attitudes, knowledge, and user experience would provide researchers with insights into a richer understanding of how and for whom different behaviour-based interventions work best. Ultimately, this should result in more streamlined and effective programmes that are targeted appropriately for different audiences.
In addition, studies would do well to make better use of mixed methods for data collection. Only 26 of the 85 studies reviewed here used interviews to collect data, with 9 studies running focus groups. This type of data collection allows for triangulation, which can be helpful when trying to get deeper insights into the holistic impacts of behaviour-based energy interventions.
Finally, we recommend that study authors provide better transparency in the methods they use. With only 4 of the 85 studies publishing their actual evaluation instrument, it is not possible for researchers to refer to and build upon instruments that have already been developed. Creating and sharing validated data collection instruments would facilitate a consistency of measurement that could be implemented across the countless additional studies expected to be conducted in the coming years. Such consistency can improve and aggregate our overall knowledge across
studies.