Sign language, though powerful enough to express any idea or thought, only bridges the communication gap between deaf people. Talking to someone with healthy hearing still usually requires a deaf person to write or type to be understood.
Now technology is helping to introduce an automatic translator that can read sign language, allowing deaf people to speak in their natural way while allowing everyone else to understand what they’re saying. Researchers at Institute of Computing Technology at the Chinese Academy of Sciences have teamed up with Microsoft Research Asia to harness the power of the company’s Kinect 3D camera to look for signature movements that code for words and letters. The Kinect, particularly the latest generation model, has an uncanny ability to detect slight variations in motion and color changes, allowing software to track minute movements necessary to express oneself with sign language. The new system reads sign and immediately prints out the translated text on the screen, but it can also do the opposite, generating an avatar that signs from the text typed to it. This way both parties to the conversation use the interface easiest for them, and we’re hopeful that voice input and output will soon be introduced to the application for seamless interaction between the hard of hearing and those with healthy ears.
From Microsoft:
In this project—which is being shown during the DemoFest portion of Faculty Summit 2013, which brings more than 400 academic researchers to Microsoft headquarters to share insight into impactful research—the hand tracking leads to a process of 3-D motion-trajectory alignment and matching for individual words in sign language. The words are generated via hand tracking by theKinect for Windows software and then normalized, and matching scores are computed to identify the most relevant candidates when a signed word is analyzed.
The algorithm for this 3-D trajectory matching, in turn, has enabled the construction of a system for sign-language recognition and translation, consisting of two modes. The first, Translation Mode, translates sign language into text or speech. The technology currently supports American sign language but has potential for all varieties of sign language.
More from Inside Microsoft Research blog: Digital Assistance for Sign-Language Users…
Paper: Sign Language Recognition and Translation with Kinect
