Do Automated Program Repair Techniques Repair Hard and Important Bugs?
Existing evaluations of automated repair techniques focus on the fraction of the defects for which the technique can produce a patch, the time needed to produce patches, and how well patches generalize to the intended specification. However, these evaluations have not focused on the applicability of repair techniques and the characteristics of the defects that these techniques can repair. Questions such as “Can automated repair techniques repair defects that are hard for developers to repair?” and “Are automated repair techniques less likely to repair defects that involve loops?” have not, as of yet, been answered. To address such questions, we annotate two large benchmarks totaling 409 C and Java defects in real-world software, ranging from 22K to 2.8M lines of code, with measures of the defect’s importance, the developer-written patch’s complexity, and the quality of the test suite. We then analyze relationships between these measures and the ability to produce patches for the defects of seven automated repair techniques – AE, GenProg, Kali, Nopol, Prophet, SPR, and TrpAutoRepair. We find that automated repair techniques are less likely to produce patches for defects that required developers to write a lot of code or edit many files, or that have many tests relevant to the defect. Java techniques are more likely to produce patches for high-priority defects. Neither the time it took developers to fix a defect nor the test suite’s coverage correlate with the automated repair techniques’ ability to produce patches. Finally, automated repair techniques are less capable of fixing defects that require developers to add loops and new function calls, or to change method signatures. These findings identify strengths and shortcomings of the state-of-the-art of automated program repair along new dimensions. The presented methodology can drive research toward improving the applicability of automated repair techniques to hard and important bugs.
Wed 30 May
|11:00 - 11:20|
Ming WenThe Hong Kong University of Science and Technology, Junjie ChenPeking University, Rongxin WuDepartment of Computer Science and Engineering, The Hong Kong University of Science and Technology, Dan HaoPeking University, Shing-Chi CheungDepartment of Computer Science and Engineering, The Hong Kong University of Science and TechnologyPre-print
|11:20 - 11:40|
Jinru Hua, Mengshi ZhangUniversity of Texas at Austin, USA, Kaiyuan Wang, Sarfraz KhurshidUniversity of Texas at AustinFile Attached
|11:40 - 12:00|
Jooyong YiInnopolis University, Shin Hwei Tan, Sergey MechtaevNational University of Singapore, Marcel BöhmeMonash University, Abhik RoychoudhuryNational University of SingaporePre-print
|12:00 - 12:20|
Manish MotwaniUniversity of Massachusetts, Amherst, Sandhya Sankaranarayanan, René JustUniversity of Massachusetts, USA, Yuriy BrunUniversity of Massachusetts, AmherstLink to publication DOI Pre-print
|12:20 - 12:30|