Do Automated Program Repair Techniques Repair Hard and Important Bugs?
Existing evaluations of automated repair techniques focus on the fraction of the defects for which the technique can produce a patch, the time needed to produce patches, and how well patches generalize to the intended specification. However, these evaluations have not focused on the applicability of repair techniques and the characteristics of the defects that these techniques can repair. Questions such as “Can automated repair techniques repair defects that are hard for developers to repair?” and “Are automated repair techniques less likely to repair defects that involve loops?” have not, as of yet, been answered. To address such questions, we annotate two large benchmarks totaling 409 C and Java defects in real-world software, ranging from 22K to 2.8M lines of code, with measures of the defect’s importance, the developer-written patch’s complexity, and the quality of the test suite. We then analyze relationships between these measures and the ability to produce patches for the defects of seven automated repair techniques – AE, GenProg, Kali, Nopol, Prophet, SPR, and TrpAutoRepair. We find that automated repair techniques are less likely to produce patches for defects that required developers to write a lot of code or edit many files, or that have many tests relevant to the defect. Java techniques are more likely to produce patches for high-priority defects. Neither the time it took developers to fix a defect nor the test suite’s coverage correlate with the automated repair techniques’ ability to produce patches. Finally, automated repair techniques are less capable of fixing defects that require developers to add loops and new function calls, or to change method signatures. These findings identify strengths and shortcomings of the state-of-the-art of automated program repair along new dimensions. The presented methodology can drive research toward improving the applicability of automated repair techniques to hard and important bugs.
Wed 30 MayDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
11:00 - 12:30
|Context-Aware Patch Generation for Better Automated Program Repair|
Ming Wen The Hong Kong University of Science and Technology, Junjie Chen Peking University, Rongxin Wu Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Dan Hao Peking University, Shing-Chi Cheung Department of Computer Science and Engineering, The Hong Kong University of Science and TechnologyPre-print
|Towards Practical Program Repair with On-Demand Candidate Generation|
Jinru Hua , Mengshi Zhang University of Texas at Austin, USA, Kaiyuan Wang , Sarfraz Khurshid University of Texas at AustinFile Attached
|A Correlation Study between Automated Program Repair and Test-Suite Metrics|
Journal first papers
Jooyong Yi Innopolis University, Shin Hwei Tan , Sergey Mechtaev National University of Singapore, Marcel Böhme Monash University, Abhik Roychoudhury National University of SingaporePre-print
|Do Automated Program Repair Techniques Repair Hard and Important Bugs?|
Journal first papers
Manish Motwani University of Massachusetts, Amherst, Sandhya Sankaranarayanan , René Just University of Massachusetts, USA, Yuriy Brun University of Massachusetts, AmherstLink to publication DOI Pre-print
|Q&A in groups|