Research

Tabletop Interactive Play System (TIPS) observes parent–child interaction through a camera and microphone, then uses a projector to display relevant ASL signs retrieved from multiple sign language libraries. This line of work investigates how AI can offer context-responsive ASL recommendations and balance usefulness with autonomy in parent-child ASL interactions (ASSETS’ 23, IDC’ 25).

RhymASL introduces a playful storytelling interactive tool that highlights phonological features of ASL signs. The system generates interactive rhyming ASL stories that families can watch and shape together, encouraging ASL interactions and building phonological awareness ASSETS’ 25.

Cat Moment is a socially-aware AR interface that can orchestrate positive dynamics in support of inclusive conversation in small group learning.

We researched the interaction between pedestrians and an interactive display on the street. Multiple machine learning models including graph neural networks are employed on labeled data to decide. whether the person will engage, and predict how long it takes before the person starts engagement (ICMI’ 2024).

We designed the state/action abstraction, sub-policy design, and multi-objective reward modeling to support reinforcement learning on a photo-taking robot from explicit human feedback and refine its movement, timing, and interaction strategy.