“Implementation Matters” – But Where’s the Evidence?
Annie O’Brien
The University of Manchester
Link to the Article: A systematic review informing recommendations for assessing implementation variability in universal, school-based social and emotional learning interventions
It is widely accepted that “implementation matters” in school-based intervention research (e.g., Durlak & DuPre, 2008; Humphrey et al., 2018; Low et al., 2014; Panayiotou et al., 2020) meaning how a programme is delivered is as important as the programme content itself. But what if the empirical research underpinning this foundational belief isn’t as strong as we assumed?
I recently had the honour of presenting our systematic review on this topic at the EUSPR Conference 2024, where I received the award for best oral presentation. This post will delve into our surprising findings about the limited and often low-quality evidence supporting the implementation-outcome relationship in the field of social and emotional learning, highlighting key challenges in current research. I also introduce our new Implementation Quality Appraisal Checklist (IQAC), to support researchers in conducting rigorous research of implementation in school-based intervention research. I’m excited to share an overview of our findings, published in Social and Emotional Learning: Research, Practice and Policy, to empower prevention and implementation scientists.
The Implementation Gap
Our systematic review uncovered a surprising truth: the widely accepted claim that “implementation matters” in the field of social and emotional learning (SEL) is actually supported by few, and often low-quality, empirical studies. We found that less than 10% of primary school-based SEL intervention studies actually examined the relationship between how SEL interventions were delivered and the outcomes students achieved. Furthermore, nearly half of the included studies were classified as low quality (45%), raising concerns about the strength of current evidence upon which the claim that “implementation matters” is built.
Quantity and Quality of Delivery: What Really Matters
While dosage and fidelity were the most frequently examined dimensions of implementation, our review found that they weren’t always the most predictive of student outcomes. Instead, teacher quality of delivery and student responsiveness were more important in medium-to-high quality studies. This suggests a need to shift focus beyond simply if a programme was delivered and how much, to also considering how well it was delivered and how students engaged.
Key Challenges in Implementation Research
Our review highlighted several challenges hindering progress in understanding the implementation-outcome relationship in SEL interventions:
- Inconsistent Reporting: There is considerable variation in how intervention implementation is monitored and/or reported. Many studies only provide descriptive data to confirm whether or not the programme lessons were actually delivered (i.e., dosage) and/or delivered as intended by the developer (i.e., fidelity). The purpose of this descriptive reporting is primarily to validate trial outcomes, but often there is no further formal investigation of whether and/or how variation in delivery is related to desired student outcomes.
- Disproportionate Attention: There is an overemphasis on fidelity and dosage, while other crucial dimensions, such as quality of delivery, participant responsiveness, and student reach, receive comparatively less attention. This is despite theoretical models emphasising the importance of these lesser studied dimensions (e.g., Berkel et al., 2011; Lendrum et al., 2016) and emerging evidence of their importance (e.g., Dowling & Barry, 2020; Humphrey et al., 2018).
- Varying Methodological Quality: Most notably, there is an abundance of different methodological approaches used to measure and assess the relationship between implementation variability and outcomes, which vary significantly in quality and rigor. Many measures of implementation processes are bespoke single use tools that do not undergo the same psychometric testing and reporting of student-level outcome measures in intervention evaluation research. This heterogeneity makes it difficult to interpret and compare findings across studies (and indeed introduced many challenges to conducting this review!).
A Blueprint for Future Research: The IQAC
To address these critical knowledge gaps and advance implementation science, we developed and published a free, open-access Implementation Quality Appraisal Checklist (IQAC). This checklist is designed to guide researchers in planning, assessing, and reporting high-quality evaluations of implementation variability in school-based intervention research. The IQAC provides a methodological framework for conducting rigorous quantitative assessments of the implementation-outcomes relationship from design through to analysis and reporting.
Call to Action for Prevention Scientists
This blog post aims to empower prevention and implementation scientists to:
- Ask critical questions when reading and reviewing research on the implementation-outcomes relationship.
- Consider a holistic approach to measuring and assessing implementation variability. This review found that is not only quantity but quality of delivery that matters; it is therefore important to look beyond just dosage and fidelity to include measures of quality of delivery and student responsiveness.
- Conduct more, and rigorous, research on the implementation-outcome relationships, using tools like the IQAC to elevate one’s work.
As SEL continues to expand globally, ensuring the quality of implementation evidence is vital. Building on the findings from this review, the next stage of my PhD research is to examine the impact of an SEL intervention (Passport: Skills for Life) on student outcomes (O’Brien et al., 2023) using complier average causal effect (CACE) which models compliance within a causal framework. We also adopt a multiverse approach to examine the relationship between each implementation dimension and student outcomes, thus exploring the influence of arbitrary decisions making (or researcher degrees of freedom) on results. This study is preregistered (https://osf.io/z62wa).
We hope this blog not only enhances accessibility of our review findings but also equips the field with tools and insights to improve how we understand and support SEL implementation. Only then can the claim that “implementation matters” truly be backed by strong, actionable evidence.
Conducting this review was challenging but rewarding. If the topic and the findings are of interest to you, please do reach out – it is always a pleasure to chat all things implementation with fellow enthusiasts!
Preparing for the next EUSPR Conference (September 23-26, 2025 in Berlin)
Are you planning to participate in the next EUSPR Conference? If so, do you intend to present further details about your work?
Yes! I have submitted the abstract of this systematic review as well as the multiverse paper mentioned in the blog.
Is there anything specific you’re looking forward to at this year’s conference?
Meeting more researchers and the ECR activities – these were my favourite part of last year’s conference.
About the author
Annie is in the third year of her PhD at The University of Manchester’s Institute of Education. Her PhD research focuses on methods used to examine, analyse and report implementation of school-based social-emotional learning interventions. Her PhD studentship is connected to a large-scale randomised control trial of the social-emotional learning programme ‘Passport: Skills for Life’, a trial taking place in 60 primary schools across Greater Manchester. More information on the trial is available here: https://www.passport2success.co.uk. Annie also works as a research assistant with Centre for Health Economics examining the cost-effectiveness of school-based interventions to the NHS.
You can contact her at annie.obrien[at]manchester.ac.uk for any enquiries
References
Berkel, C., Mauricio, A. M., Schoenfelder, E., & Sandler, I. N. (2011). Putting the pieces together: An integrated model of program implementation. Prevention Science, 12, 23-33. https://doi.org/https://doi.org/10.1007/s11121-010-0186-1
Dowling, K., & Barry, M. M. (2020). The effects of implementation quality of a school-based social and emotional well-being program on students’ outcomes. European journal of investigation in health, psychology and education, 10(2), 595-614. https://doi.org/https://doi.org/10.3390/ejihpe10020044
Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American journal of community psychology, 41, 327-350. https://doi.org/https://doi.org/10.1007/s10464-008-9165-0
Humphrey, N., Barlow, A., & Lendrum, A. (2018). Quality matters: Implementation moderates student outcomes in the PATHS curriculum. Prevention Science, 19(2), 197-208. https://doi.org/doi.org/10.1007/s11121-017-0802-4
Lendrum, A., Humphrey, N., & Greenberg, M. (2016). Implementing for success in school-based mental health promotion: The role of quality in resolving the tension between fidelity and adaptation. In Mental Health and Wellbeing through Schools (pp. 53-63). Routledge.
Low, S., Van Ryzin, M. J., Brown, E. C., Smith, B. H., & Haggerty, K. P. (2014). Engagement matters: Lessons from assessing classroom implementation of steps to respect: A bullying prevention program over a one-year period. Prevention Science, 15, 165-176.
O’Brien, A., Hamilton, S., Humphrey, N., Qualter, P., Boehnke, J. R., Santos, J., Demkowicz, O., Panayiotou, M., Thompson, A., Lau, J., Burke, L., & Lu, Y. (2023). Examining the impact of a universal social and emotional learning intervention (Passport) on internalising symptoms and other outcomes among children, compared to the usual school curriculum: study protocol for a school-based cluster randomised trial. Trials, 24(1), 703. https://doi.org/10.1186/s13063-023-07688-0
Panayiotou, M., Humphrey, N., & Hennessey, A. (2020). Implementation matters: Using complier average causal effect estimation to determine the impact of the Promoting Alternative Thinking Strategies (PATHS) curriculum on children’s quality of life [doi:10.1037/edu0000360]. Journal of Educational Psychology, 236-253. https://doi.org/10.1037/edu0000360
OpenEdition suggests that you cite this post as follows:
EUSPR Early Careers Forum (June 16, 2025). “Implementation Matters” – But Where’s the Evidence? Preventing disease and ill health. Retrieved May 19, 2026 from https://doi.org/10.58079/144ti


