Document Layout and Formatting Helper for Blind Authors

This project, funded through NSF CAREER award, aims to facilitate independence for blind authors in producing documents that meet the presentation standards expected by sighted readers. This includes the development of an impact-weighted taxonomy of common document presentation errors of blind authors, an exploration of blind persons' mental models and strategies for learning and coping, and how these models and strategies contribute to the success of independent document formatting and layout activities. These will inform the design of an integrated solution to document preparation for blind authors in the form of a Microsoft Word compatible formatting and layout checker, as well as a fix to the compatibility problems between Word's grammar and spell checkers and commonly used screen readers. This tool will be complemented by an accessible content-sharing environment for blind authors with an interface for sighted help from Amazon Mechanical Turk.

Morales, Adams, Kurniawan.

LASSIE (Live-in ASSistance for Independence and Eldercare) 

This LASSIE robot is designed to be a monitoring assistive living robot, which is low cost and can be steered over the Internet by family member to remotely monitor and help assess the well-being of an older relative living alone. A commercially available iRobot Create platform was used as a starting point for the Assistive Living Robot, however the system could potentially be used on another robotic base. Our system (i) takes advantage of commercially available systems to reduce development cost and effort, (ii) acts as a video and audio communication tool between older persons and their family members or caregivers and (iii) can analyze the video feed to detect heart rate and breathing rate. The proposed system could be integrated with in-home monitoring sensors (Smart Grid, Professor Patrick Mantey), which could be the trigger for alerting family members that an out of ordinary event is occurring, therefore, our robot could be woken up and sent to be used as over the Internet watchful eye.


Blind Photography

Photography is a visual way to capture a moment in time. Photos can be used for artistic expression, and to remember significant events. Because photo taking, organizing and sharing traditionally requires visual information, those with no or limited sight often have problems with these activities. Previous work has made photo capturing without sight easier, however, there is little work that makes photo browsing and sharing blind-accessible. My dissertation research aims at facilitating independence for blind persons to take, organize and share photos through user-centered development of a smartphone application that can be used without sight. The work starts with an investigation of current practices of blind persons in these activities, continued with a review of existing applications, and finally the design and long-term evaluation of the application.

The overarching needs that this smartphone application aims to meet are the following:

Photo Taking: Aiming, focusing, positioning, and framing; Easy way to get sighted help; Accessible device; Improving photo quality.
Photo Organizing and Editing: Identifying what’s in the picture; Labeling the pictures; Manipulating pictures
Photo Sharing: Easy way to get sighted help; Accessible photo sharing method.


Adams, Kurniawan. In collaboration with Harada, Sato, Asakawa, Takagi, IBM Research - Tokyo.


Speech Therapy Game for Children with Cleft Palate

We are developing a video game to aid young children undergoing speech therapy after cleft palate surgery. Children undergoing speech therapy must first unlearn their compensations and ommissions that they use to get around certain syllables they cannot do. Unlearning involves significant amounts of practice at home, and parents have extreme difficulty in motivating children to practice. Working directly with children who are undergoing speech therapy, we are designing a simple game using a novel speech recognition engine to help motivate them to perform their therapy exercises. We aim to accelerate the rate of recovery and give therapists more tools to use with children. We also intend to forward research in educational games.

Rubin, Kurniawan. In collaboration with UC Davis Medical Center.

Digital Birth


Labor and childbirth is a multidimensional experience for which it is difficult to prepare (without having done it before). It has been shown that emotional, informational, and physical support is key in shortening labor, increasing maternal happiness, and decreasing the need for unnecessary interventions during labor. This mobile (iPhone) video game training tool aims to train birth partners about the stages of labor and ways to support a first-time mom.

Holloway, Kurniawan. In collaboration with Shaw-Battista, School of Nursing, UCSF and Moodie, Anthropology Dept, UCSC. Research funded by CITRIS.