Publications
カンファレンス (国際) Shape-N-Motion: Fine-Grained Hand Object Manipulation Recognition with Ultrasonic and IMU
Kaito Fujishige (Institute of Science Tokyo), Kota Tsubouchi, Yuuki Nishiyama (The University of Tokyo), Masamichi Shimosaka (Institute of Science Tokyo)
The 24th International Conference on Pervasive Computing and Communications (PerCom 2026)
2026.3.17
Fine-grained hand--object manipulations (e.g., tapping vs.\ scrolling on the same smartphone) are informative for lifelogging and remote health monitoring, but are often confused by wearable IMU (Inertial Measurement Unit) and audible audio because wrist trajectories are similar and many actions are nearly silent. We propose \emph{Shape-N-Motion}, a wrist-worn multimodal system that fuses fingertip-oriented ultrasonic echoes and a wrist IMU for direct time-series classification without hand-shape reconstruction. We build a 45-class benchmark (N=6) refined from existed protocol to explicitly reflect contact states, state changes, and tool use, and evaluate cross-participant generalization with LOPO (Leave-One-Participant-Out). \emph{Shape-N-Motion} achieves 0.467 accuracy / 0.417 Macro-F1, outperforming the strongest IMU+audio baseline by +0.081 accuracy and +0.079 Macro-F1.