Sign Language Recognition with Multimodal Sensors and Deep Learning Methods
Problem
Sign language recognition with a single camera suffers from occlusion and limited accuracy, hindering communication for hearing-impaired people.
Why it matters
Communication barriers exclude hearing-impaired people from everyday interaction.
What the AI did
A CNN-BiLSTM model fused skeleton keypoints, hand joint angles, and finger curvature from a monocular camera and bending-sensor gloves to recognize signs.
Results
Qualitative claim of higher recognition accuracy via multimodal fusion; no numbers stated in source
Impact
Multimodal fusion qualitatively improved recognition accuracy over camera-only approaches, per abstract.
Record
| Venue | Electronics |
|---|---|
| Published | 2023 |
| Work type | article |
| Authors | Chenghong Lu, Misaki Kozakai, Lei Jing |
| Citations | 34 (OpenAlex, at import time) |
| References | 30 |
| Model or method | CNN-BiLSTM |
| Organization | not stated in source |
| Primary source | https://doi.org/10.3390/electronics12234827 |
| Open access | gold |
| Keywords | sign language recognition, multimodal sensors, data gloves, CNN-BiLSTM, computer vision |
Importance rating
incremental / niche
Improves sign language recognition for hearing-impaired communication, a real accessibility need, but demonstrated only on a small lab dataset of 32 Japanese signs from seven people.