Lessons from Conducting a Distributed Quasi-Experiment

Budgen, David; Kitchenham, Barbara; Charters, Stuart; Gibbs, Shirley; Pohthong, Amnart; Keung, Jacky; Brereton, Pearl

Abstract

Context: Due to the lack of suitably skilled participants, software engineering experiments often lack the statistical power needed to detect the levels of effect that may be encountered. Aim: To investigate whether this can be remedied by running an experiment across multiple sites, organised as a single study rather than as a set of replications. Method: We performed a `trial' of the idea using a topic (structured abstracts) that some of us had studied previously and which required no participant training. We used five sites, each with 16 participants. Results: We were able to demonstrate the benefits of increased statistical power (and of structured abstracts). We report on our experiences with designing and conducting the study and identify some key lessons about how future studies of this form might be organised. Conclusions: The distributed model offers a flexible, robust form that is capable of delivering better statistical power than would be achieved by running a set of parallel replicated studies.

Citation

Budgen, D., Kitchenham, B., Charters, S., Gibbs, S., Pohthong, A., Keung, J., & Brereton, P. (2013, October). Lessons from Conducting a Distributed Quasi-Experiment. Presented at 2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), Baltimore, USA

• Presentation Conference Type: Conference Paper (published)
• Conference Name: 2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)
• Start Date: Oct 10, 2013
• End Date: Oct 11, 2013
• Publication Date: 2013
• Deposit Date: Jun 3, 2013
• Pages: 143-152
• DOI: https://doi.org/10.1109/ESEM.2013.12
• Keywords: abstracts; software engineering; educational institutions; materials; software; electronic mail; data collection
• Public URL: https://durham-repository.worktribe.com/output/1155675