* ICSE 2018 *
Sun 27 May - Sun 3 June 2018 Gothenburg, Sweden
Fri 1 Jun 2018 14:00 - 14:20 at J2 room - Surveys and Reviews Chair(s): Anne Koziolek

BACKGROUND: The evidence-based software engineering approach advocates the use of evidence from empirical studies to support the decisions on the adoption of software technologies by practitioners in the software industry. To this end, many guidelines have been proposed to contribute to the execution and repeatability of literature reviews, and to the confidence of their results, especially regarding systematic literature reviews (SLR). AIMS: To investigate similarities and differences, and to characterize the challenges and pitfalls of the planning and generated results of SLR research protocols dealing with the same research question and performed by similar teams of novice researchers in the context of the software engineering field. METHOD: We qualitatively compared (using Jaccard and Kappa coefficients) and evaluated (using DARE) same goal SLR research protocols and outcomes undertaken by similar research teams. RESULTS: Seven similar SLR protocols regarding quality attributes for use cases executed in 2010 and 2012 enabled us to observe unexpected differences in their planning and execution. Even when the participants reached some agreement in the planning, the outcomes were different. The research protocols and reports allowed us to observe six challenges contributing to the divergences in the results: researchers’ inexperience in the topic, researchers’ inexperience in the method, lack of clearness and completeness of the papers, lack of a common terminology regarding the problem domain, lack of research verification procedures, and lack of commitment to the SLR. CONCLUSIONS: According to our findings, it is not possible to rely on results of SLRs performed by novices. Also, similarities at a starting or intermediate step during different SLR executions may not directly translate to the next steps, since non-explicit information might entail differences in the outcomes, hampering the repeatability and confidence of the SLR process and results. Although we do have expectations that the presence and follow-up of a senior researcher can contribute to increasing SLRs’ repeatability, this conclusion can only be drawn upon the existence of additional studies on this topic. Yet, systematic planning, transparency of decisions and verification procedures are key factors to guarantee the reliability of SLRs.

Fri 1 Jun

icse-2018-Technical-Papers
14:00 - 15:30: Technical Papers - Surveys and Reviews at J2 room
Chair(s): Anne KoziolekKarlsruhe Institute of Technology
icse-2018-Journal-first-papers152785440000014:00 - 14:20
Talk
Link to publication DOI Pre-print
icse-2018-Technical-Papers152785560000014:20 - 14:40
Research paper
Rolando Reyes, Oscar DiesteUniversidad Politécnica de Madrid, Efraín R. Fonseca C., Natalia JuristoFacultad de Informática - UPM
DOI Pre-print Media Attached File Attached
icse-2018-Technical-Papers152785680000014:40 - 15:00
Talk
Xin Huang, He ZhangNanjing University, Xin Zhou, Yang SongUniversity of North Carolina Wilmington, Muhammad Ali Babar
icse-2018-Journal-first-papers152785800000015:00 - 15:20
Talk
Luca Gazzola Università degli Studi di Milano-Bicocca, Daniela MicucciUniversity of Milano-Bicocca, Italy, Leonardo MarianiUniversity of Milano Bicocca
Link to publication Pre-print
icse-2018-Technical-Papers152785920000015:20 - 15:30
Talk