* ICSE 2018 *
Sun 27 May - Sun 3 June 2018 Gothenburg, Sweden
Fri 1 Jun 2018 14:20 - 14:40 at J2 room - Surveys and Reviews Chair(s): Anne Koziolek

Background: Statistical concepts and techniques are often applied incorrectly, even in mature disciplines such as medicine or psychology. Surprisingly, there are very few works that study statistical problems in software engineering (SE).

Aim: Assess the existence of statistical errors in SE experiments.

Method: Compile the most common statistical errors in experimental disciplines. Survey experiments published in ICSE to assess whether errors occur in high quality SE publications.

Results: The same errors as identified in others disciplines were found in ICSE experiments, where 30% of the reviewed papers included several error types such as: a) missing statistical hypotheses, b) missing sample size calculation, c) failure to assess statistical test assumptions, and d) uncorrected multiple testing. This rather large error rate is greater for research papers where experiments are confined to the validation section. The origin of the errors can be traced back to: a) researchers not having sufficient statistical training, and, b) a profusion of exploratory research.

Conclusions: This paper provides preliminary evidence that SE research suffers from the same statistical problems as other experimental disciplines. However, the SE community appears to be unaware of any shortcomings in its experiments, whereas other disciplines work hard to avoid these threats. Further research is necessary to find the underlying causes and set up corrective measures, but there are some potentially effective actions and are a priori easy to implement: a) improve the statistical training of SE researchers, and b) enforce quality assessment and reporting guidelines in SE publications.

Fri 1 Jun

Displayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change

14:00 - 15:30
Surveys and ReviewsTechnical Papers / Journal first papers at J2 room
Chair(s): Anne Koziolek Karlsruhe Institute of Technology
Challenges and pitfalls on surveying evidence in the software engineering technical literature:an exploratory study with novices
Journal first papers
Talita Vieira Ribeiro Federal University of Rio de Janeiro, Jobson Massollar , Guilherme Horta Travassos
Link to publication DOI Pre-print
Research paper
Statistical Errors in Software Engineering Experiments: A Preliminary Literature Review
Technical Papers
Rolando Reyes , Oscar Dieste Universidad Politécnica de Madrid, Efraín R. Fonseca C. , Natalia Juristo Facultad de Informática - UPM
DOI Pre-print Media Attached File Attached
Synthesizing Qualitative Research in Software Engineering: A Critical Review
Technical Papers
Xin Huang , He Zhang Nanjing University, Xin Zhou , Yang Song University of North Carolina Wilmington, Muhammad Ali Babar
Automatic Software Repair: A Survey
Journal first papers
Luca Gazzola Università degli Studi di Milano-Bicocca, Daniela Micucci University of Milano-Bicocca, Italy, Leonardo Mariani University of Milano Bicocca
Link to publication Pre-print
Q&A in groups
Technical Papers