Cased-based modelling and scenario simulation for ex-post evaluation

Schimpf, Corey; Barbrook-Johnson, Pete; Castellani, Brian

Abstract

Despite 20 years of increasing acceptance, implementing complexity-appropriate methods for ex-post evaluation remains a challenge: instead of focusing on complex interventions, methods need to help evaluators better explore how policies (no matter how simple) take place in real-world, open, dynamic systems where many intertwined factors about the cases being targeted affect outcomes in numerous ways. To assist in this advance, we developed case-based scenario simulation, a new visually intuitive evaluation tool grounded in a data-driven, case-based, computational modelling approach, which evaluators can use to explore counterfactuals, status-quo trends, and what-if scenarios for some potential set of real or imagined interventions. To demonstrate the value and versatility of case-based scenario simulation we explore four published evaluations that differ in design (cross sectional, longitudinal, and experimental) and purpose (learning or accountability), and present a prospective view of how case-based scenario simulation could support and enhance evaluators’ efforts in these complex contexts.

Citation

Schimpf, C., Barbrook-Johnson, P., & Castellani, B. (2021). Cased-based modelling and scenario simulation for ex-post evaluation. Evaluation, 27(1), 116-137. https://doi.org/10.1177/1356389020978490

• Journal Article Type: Article
• Online Publication Date: Jan 14, 2021
• Publication Date: 2021-01
• Deposit Date: Mar 7, 2021
• Journal: Evaluation
• Print ISSN: 1356-3890
• Electronic ISSN: 1461-7153
• Publisher: SAGE Publications
• Peer Reviewed: Peer Reviewed
• Volume: 27
• Issue: 1
• Pages: 116-137
• DOI: https://doi.org/10.1177/1356389020978490