* ICSE 2018 *
Sun 27 May - Sun 3 June 2018 Gothenburg, Sweden
Thu 31 May 2018 15:00 - 15:20 at H1 room - Human and Social Aspects of Computing II Chair(s): Margaret-Anne Storey

A GUI skeleton is the starting point for implementing a UI design image. To obtain a GUI skeleton from a UI design image, developers have to visually understand UI elements and their spatial layout in the image, and then translate this understanding into proper GUI components and their compositions. Automating this visual understanding and translation would be beneficial for bootstraping mobile GUI implementation, but it is a challenging task due to the diversity of UI designs and the complexity of GUI skeletons to generate. Existing tools are rigid as they depend on heuristically-designed visual understanding and GUI generation rules. In this paper, we present a neural machine translator that combines recent advances in computer vision and machine translation for translating a UI design image into a GUI skeleton. Our translator learns to extract visual features in UI images, encode these features’ spatial layouts, and generate GUI skeletons in a unified neural network framework, without requiring manual rule development. For training our translator, we develop an automated GUI exploration method to automatically collect large-scale UI data from real-world applications. We carry out extensive experiments to evaluate the accuracy, generality and usefulness of our approach.

Thu 31 May

14:00 - 15:30: Technical Papers - Human and Social Aspects of Computing II at H1 room
Chair(s): Margaret-Anne StoreyUniversity of Victoria
icse-2018-Technical-Papers152776800000014:00 - 14:20
Hung Phan, Hoan NguyenIowa State University, Ngoc Tran, Linh Truong, Anh Nguyen, Tien NguyenThe University of Texas - Dallas
icse-2018-Technical-Papers152776920000014:20 - 14:40
Andrew HeadUC Berkeley, Caitlin SadowskiGoogle, Emerson Murphy-HillNorth Carolina State University, Andrea Knight
icse-2018-Technical-Papers152777040000014:40 - 15:00
Brian HempelUniversity of Chicago, Justin Lubin, Grace Lu, Ravi ChughUniversity of Chicago
Pre-print Media Attached
icse-2018-Technical-Papers152777160000015:00 - 15:20
Chunyang ChenNanyang Technological University, Singapore, Ting SuNanyang Technological University, Singapore, Guozhu MengNanyang Technological University, Singapore, Zhenchang XingAustralia National University, Yang LiuNanyang Technological University, Singapore
icse-2018-Technical-Papers152777280000015:20 - 15:30