Publications
論文誌 (国際) Leveraging Screen-Integrated Speakers for Hand-Pose Recognition in Mobile Interfaces
Kaori Ikematsu*, Kunihiro Kato* (Tokyo University of Technology), *co-1st authors
International Journal of Human–Computer Interaction (IJHCI)
2025.12.23
Active acoustic sensing with a smartphone's built-in speaker -microphone pair enables around-device interaction, but recognizing hand poses above the display remains difficult with conventional top-speaker and bottom-microphone placement. We leverage screen-integrated speakers (Acoustic Surface) that vibrate the display, extending sensing coverage above the screen and integrating near-surface hand-pose recognition with touch. On commodity smartphones that integrate this technology, we compare a screen-bottom configuration with a typical top-bottom layout. Across datasets of single-hand and double-hand poses, our models recognize 12 single-hand classes at 94.5%, 10 double-hand classes at 90.6%, and 20 combined classes at 90.4%, surpassing the conventional layout. We also conducted a formative within-set elicitation study where participants assessed poses and proposed pose -command mappings. We synthesize results and feedback into four design guidelines for near-surface hand-pose input. The findings expand input above the display without external hardware and offer guidance for around-device interaction on commodity devices.
Paper :
Leveraging Screen-Integrated Speakers for Hand-Pose Recognition in Mobile Interfaces
（外部サイト）