* ICSE 2018 *
Sun 27 May - Sun 3 June 2018 Gothenburg, Sweden
Fri 1 Jun 2018 14:20 - 14:40 at J2 room - Surveys and Reviews Chair(s): Anne Koziolek

Background: Statistical concepts and techniques are often applied incorrectly, even in mature disciplines such as medicine or psychology. Surprisingly, there are very few works that study statistical problems in software engineering (SE).

Aim: Assess the existence of statistical errors in SE experiments.

Method: Compile the most common statistical errors in experimental disciplines. Survey experiments published in ICSE to assess whether errors occur in high quality SE publications.

Results: The same errors as identified in others disciplines were found in ICSE experiments, where 30% of the reviewed papers included several error types such as: a) missing statistical hypotheses, b) missing sample size calculation, c) failure to assess statistical test assumptions, and d) uncorrected multiple testing. This rather large error rate is greater for research papers where experiments are confined to the validation section. The origin of the errors can be traced back to: a) researchers not having sufficient statistical training, and, b) a profusion of exploratory research.

Conclusions: This paper provides preliminary evidence that SE research suffers from the same statistical problems as other experimental disciplines. However, the SE community appears to be unaware of any shortcomings in its experiments, whereas other disciplines work hard to avoid these threats. Further research is necessary to find the underlying causes and set up corrective measures, but there are some potentially effective actions and are a priori easy to implement: a) improve the statistical training of SE researchers, and b) enforce quality assessment and reporting guidelines in SE publications.

Fri 1 Jun

14:00 - 15:30: Technical Papers - Surveys and Reviews at J2 room
Chair(s): Anne KoziolekKarlsruhe Institute of Technology
icse-2018-Journal-first-papers152785440000014:00 - 14:20
Link to publication DOI Pre-print
icse-2018-Technical-Papers152785560000014:20 - 14:40
Research paper
Rolando Reyes, Oscar DiesteUniversidad Politécnica de Madrid, Efraín R. Fonseca C., Natalia JuristoFacultad de Informática - UPM
DOI Pre-print Media Attached File Attached
icse-2018-Technical-Papers152785680000014:40 - 15:00
Xin Huang, He ZhangNanjing University, Xin Zhou, Yang SongUniversity of North Carolina Wilmington, Muhammad Ali Babar
icse-2018-Journal-first-papers152785800000015:00 - 15:20
Luca Gazzola Università degli Studi di Milano-Bicocca, Daniela MicucciUniversity of Milano-Bicocca, Italy, Leonardo MarianiUniversity of Milano Bicocca
Link to publication Pre-print
icse-2018-Technical-Papers152785920000015:20 - 15:30