* ICSE 2018 *
Sun 27 May - Sun 3 June 2018 Gothenburg, Sweden
Tue 29 May 2018 14:00 - 15:30 at R18 - Research Methods

Experimentation is a key issue in science and engineering. But it is one of software engineering’s stumbling blocks. Quite a lot of experiments are run nowadays, but it is a risky business. Software engineering has some special features, leading to some experimentation issues being conceived of differently than in other disciplines. The aim of this technical briefing is to help participants to avoid common pitfalls when analyzing the results of software engineering experiments. The technical briefing is not intended as a data analysis course, because there is already plenty of literature on this subject. It reviews several key issues that we have identified in published software engineering experiments, and addresses them based on the knowledge acquired after 19 years running experiments. The technical briefing starts by recalling what a SE experiment is, and its distinctive features: control and causality. Then, five topics related to data analysis will be explored. The technical briefing will end with participants’ questions.

The five topics covered are:

One-tailed vs. two-tailed tests. SE experimenters very often opt for a one-tailed hypothesis, but this can be a shortcoming in many experiments.
Choosing the right data analysis technique. The selected data analysis technique should match the experimental design. However, the choice is not straightforward, and several issues have to be taken into consideration.
Analyzing tricky designs. We will discuss two designs which are commonly not properly analyzed: blocked and crossover designs. -Parametric vs. non-parametric tests. We will discuss the different options that can be used when data do not meet the parametric tests assumptions.
The 3 musketeers: statistical significance, effect size and power. We will discuss the meaning and implications of the three parameters, how they relate to each other, and how they should be used to properly interpret the results of an experiment.
Slides (TB ICSE18 Vegas.pdf)1.91MiB

Tue 29 May

Displayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change

14:00 - 17:30
Research MethodsTB - Technical Briefings at R18
Analyzing Software Engineering Experiments: Everything You Always Wanted to Know but Were Afraid to Ask
TB - Technical Briefings
Sira Vegas Universidad Politecnica de Madrid
File Attached
How to Analyze Git Repositories with Command Line Tools: We're not in Kansas anymore
TB - Technical Briefings
Diomidis Spinellis Athens University of Economics and Business, Georgios Gousios TU Delft
DOI Pre-print