Yufei Ye

Research Abstract:

My research aims to computer vision system that understands everyday human interactions with rich spatial information, in particular hand-object interactions (HOI). Such systems can benefit VR/AR to perceive reality and to modify its virtual twin, and robotics to learn manipulation by watching humans. Previous methods are limited to constrained lab environments or pre-selected objects with known 3D shapes. My works explore learning general interaction priors from large-scale data that can generalize to novel everyday scenes for both perception and prediction. My research consists of two parts. The first part focuses on HOI prediction -- predicting plausible human grasps for any objects. We found that image synthesis serves as a shortcut for 3D prediction for better generalization.  The second part focuses on reconstructing interactions in 3D space for generic objects by leveraging data-driven prior. We build system where interaction prior is critical in the presence of mutual occlusion for reconstruction from both single-view and video.

Bio:

Yufei Ye is a PhD student in Robotics Institute at Carnegie Mellon University, advised by Prof. Shubham Tulsiani and Prof. Abhinav Gupta. Prior to this, she obtained her B.E. in computer science from Tsinghua University, working with Prof. Shi-Min Hu. She has participated in Meta-CMU AI Mentor program and has interned at NVIDIA, MetaAI, and Yitu Technology. She has won NVIDIA Graduate Fellowship (2022-23). Her research interests lie at the intersection of 3D computer vision, robotics and machine learning. Her long-term goal is to enable artificial intelligent agents to better understand the interaction of the world — how to perceive the interaction, how to actively interact with the world.