A diagnostic approach to improve the design and evaluation of decision support interventions

Abstract

A range of decision analysis methods are proposed to aid managers in solving complex problems. Due to a lack of systematic evaluation, it is difficult to say whether and why a certain intervention works in a given context. Claimed benefits often cannot be substantiated, hindering the uptake of decision analysis and impeding learning for more targeted design. Building on theories of persuasion, Rouwette et al (2003, 2009, 2011) developed and tested a framework to evaluate the performance of facilitated group model building (and by extension problem structuring). Although conditions for persuasion were given, the variable combinations could not predict the observed changes. I adapted the framework aiming to improve operationalization for problem structuring and MCDA interventions. I added variables to capture behavioral and social aspects known to affect group collaboration and decision outcomes. When testing the resulting pre-/post assessment in a problem structuring intervention with 13 Dutch sewer managers, a surprise happened. Despite their prior agreement, it was met with resistance and social dynamics unfolded that hindered its completion. In hindsight, the pre-assessment revealed issues that if considered for designing rather than evaluating the intervention might have avoided failure. I will present the approach and the insights gained as well as how these might be adapted for better diagnosis of the situation to inform intervention design and evaluation.