Statistical Errors in Software Engineering Experiments: A Preliminary Literature Review
Background: Statistical concepts and techniques are often applied incorrectly, even in mature disciplines such as medicine or psychology. Surprisingly, there are very few works that study statistical problems in software engineering (SE).
Aim: Assess the existence of statistical errors in SE experiments.
Method: Compile the most common statistical errors in experimental disciplines. Survey experiments published in ICSE to assess whether errors occur in high quality SE publications.
Results: The same errors as identified in others disciplines were found in ICSE experiments, where 30% of the reviewed papers included several error types such as: a) missing statistical hypotheses, b) missing sample size calculation, c) failure to assess statistical test assumptions, and d) uncorrected multiple testing. This rather large error rate is greater for research papers where experiments are confined to the validation section. The origin of the errors can be traced back to: a) researchers not having sufficient statistical training, and, b) a profusion of exploratory research.
Conclusions: This paper provides preliminary evidence that SE research suffers from the same statistical problems as other experimental disciplines. However, the SE community appears to be unaware of any shortcomings in its experiments, whereas other disciplines work hard to avoid these threats. Further research is necessary to find the underlying causes and set up corrective measures, but there are some potentially effective actions and are a priori easy to implement: a) improve the statistical training of SE researchers, and b) enforce quality assessment and reporting guidelines in SE publications.
Conference DayFri 1 JunDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
14:00 - 15:30
|Challenges and pitfalls on surveying evidence in the software engineering technical literature:an exploratory study with novices|
Journal first papers
Talita Vieira RibeiroFederal University of Rio de Janeiro, Jobson Massollar, Guilherme Horta TravassosLink to publication DOI Pre-print
|Statistical Errors in Software Engineering Experiments: A Preliminary Literature Review|
Rolando Reyes, Oscar DiesteUniversidad Politécnica de Madrid, Efraín R. Fonseca C., Natalia JuristoFacultad de Informática - UPMDOI Pre-print Media Attached File Attached
|Synthesizing Qualitative Research in Software Engineering: A Critical Review|
|Automatic Software Repair: A Survey|
Journal first papers
Luca Gazzola Università degli Studi di Milano-Bicocca, Daniela MicucciUniversity of Milano-Bicocca, Italy, Leonardo MarianiUniversity of Milano BicoccaLink to publication Pre-print
|Q&A in groups|