An evaluation of quality checklist proposals – A participant-observer case study

Kitchenham, Barbara A.; Brereton, O. Pearl; Budgen, David; Li, Zhi

Abstract

Background: A recent set of guidelines for software engineering systematic literature reviews (SLRs) includes a list of quality criteria obtained from the literature. The guidelines suggest that the list can be used to construct a tailored set of questions to evaluate the quality of primary studies. Aim: This paper aims to evaluate whether the list of quality criteria help researchers construct tailored quality checklists. Method: We undertook a participant-observer case study to investigate the list of quality criteria. The “case” in this study was the planning stage of a systematic literature review on unit testing. Results: The checklists in our SLR guidelines do not provide sufficient help with the construction of a quality checklist for a specific SLR either for novices or for experienced researchers. However, the checklists are reasonably complete and lead to the use of a common terminology for quality questions selected for a specific systematic literature review. Conclusions: The guidelines document should be amended to include a much shorter generic checklist. Researchers might find it useful to adopt a team-based process for quality checklist construction and provide suggestions for answering quality checklist questions.

Citation

Kitchenham, B. A., Brereton, O. P., Budgen, D., & Li, Z. (2009, April). An evaluation of quality checklist proposals – A participant-observer case study. Presented at 13th International Conference on Evaluation and Assessment in Software Engineering, EASE 2009, Durham University

• Presentation Conference Type: Conference Paper (published)
• Conference Name: 13th International Conference on Evaluation and Assessment in Software Engineering, EASE 2009
• Start Date: Apr 20, 2009
• End Date: Apr 21, 2009
• Publication Date: 2009-04
• Deposit Date: Feb 23, 2025
• Peer Reviewed: Peer Reviewed
• DOI: https://doi.org/10.14236/ewic/ease2009.7
• Public URL: https://durham-repository.worktribe.com/output/3500785