Sugilite

31 downloads

Uploaded by

Anonymous

Version

1.0 (build 1)  1 version(s)

Sugilite versions

1.0 (build 1)

31 downloads

Close

Size

2.5 MiB

Publish Date

2023-06-12

Category

Tools

Package

edu.cmu.hcii.sugilite

Minimum OS

Android 6.0

SHA1

bea0a79374959d4958e54a971f3f420c2b6848ec

Description

SUGILITE is a new programming-by-demonstration (PBD) system that enables users to create automation on smartphones. SUGILITE uses Android’s accessibility API to support automating arbitrary tasks in any Android app (or even across multiple apps). When the user gives verbal commands that SUGILITE does not know how to execute, the user can demonstrate by directly manipulating the regular apps’ user interface. By leveraging the verbal instructions, the demonstrated procedures, and the apps’ UI hierarchy structures, SUGILITE can automatically generalize the script from the recorded actions, so SUGILITE learns how to perform tasks with different variations and parameters from a single demonstration. Extensive error handling and context checking support forking the script when new situations are encountered, and provide robustness if the apps change their user interface. Our lab study suggests that users with little or no programming knowledge can successfully automate smartphone tasks using SUGILITE.