> While this is clearly a massive achievement, there is still a huge amount of work ahead. It currently takes five people to establish the recognition patterns for just one word.
I don't get it. They're saying they need five (only five!) seperate people to train the thing, and then it works pretty well for everyone? That seems to be working pretty darn well to me, I don't see the problem or 'huge amount of work'.
I agree, this is an awesome project. I'm particularly curious about how CSL speakers have to constrain nuance in their signing for the translator to remain accurate.
I know nothing about CSL, but ASL phonology is rather complicated. There is a lot of assimilation, for example, and the meaning of signs can change based on the spacial orientation of consecutive signs.
My old linguistics professor told me about a joke someone made at an ASL conference. He was signing with a colleague, and talking about his progress improving his ASL. She then begins making the sign for "improve":
But instead of making contact with her upper arm, she shifts her right arm downwards and repeats the starting position of the sign. This is the ASL equivalent of a "...NOT" joke, as in, "your signing has improved... NOT!"
The degree of space between starting and ending positions in the "improve" sign can articulate different degrees of improvement.
While I suspect this kind of nuance isn't yet translatable by this project's sign-recognition software, I would really love to see what kind of progress they're making in this area.
Having worked for leading Video Relay Service for the deaf, I can say without a doubt that, unless this is seriously flawless, it won't catch on very well with the deaf community. Also, it's been anticipated for a long time. I don't see it disrupting VRS services for at least a decade. Furthermore, in the video, everyone signing is doing so at a snail's pace.
That being said, the FCC still pays $6 a minute for VRS services.
Yes, the pace is extremely unnatural, akin to enunciating every syllable in spoken language according to “standard” pronunciation. However, this is not yet a consumer product: it’s foundational research toward better systems in the future. Eventually it could be as good as existing speech-to-text solutions—which is to say not very good, but tolerable.
There are three primary problems with what I see in this video. I base these arguments on the 3 years of American Sign Language I studied in University, and the thousands of dollars I spent drinking and talking with Deaf people and other ASL speakers. I'm by no means perfectly fluent, but I get by. Beer helps.
First issue: The signing they are doing in the video is very simple and limited. Not just in the number of signs, but in the complexity of what they are doing. Their face remains perfectly neutral. They move slowly and carefully. They are signing individual signs one at a time. That's not how a signed language works!
Signing without facial expression is like speaking in complete monotone, making no eye contact with the person you're speaking to. You can probably get your point across, but you're not having a real conversation, and you're missing something important. The lack of flow and motion to the signing is also awkward as hell. In short, the people in those videos are not actually fluent in sign, they're trained actors who have memorized certain motions.
Second issue: There are concepts in sign language that have no equivalent in English, or many spoken languages.
One example- in ASL you have variables/registers, in the programming sense. You can sign "John", then point to your left. From now on in this conversation, if you point left, you mean John. If while you speak, you want to describe John moving to your right, you can do that (explained next). Now pointing right means "John".
How might you 'move' John to your right? You could have a single finger pointed up, like John was a finger puppet, and walk him to the right. If he got into a car, you could have that single finger get into another specific hand form that generally means 'vehicle'. Then you could draw a tree with your free hand, and drive the car into the tree. You just told the story about John driving into a tree. There were no specific nouns or verbs used. This is how a lot of conversations work in ASL.
The point I'm trying to illustrate above is that description and conversations happen very differently in signed languages than they do in English. It isn't that the words come in a different order, it's that the idea you are trying to communicate is explained in a completely different way. You're comparing apples and coconuts.
Third issue: every city has its own dialect. Signed languages aren't written down, there isn't much global media of people signing, and the result is that dialects change constantly. Drive to any other major city, entire words have changed. Talk to a family across the street, they might have different signs for some words. Sure, the core language is the same, but you've got a lot of different nouns and verbs.
Going even deeper, some people here have already mentioned the difference between ASL and Signed English. Some people sign English words one at a time in the order of English grammar. Some people go into the 'pure' ASL realm, as I described above, and use very complex concepts not used in English. It's not a choice, they'll have simply learned ASL that way. Most people are somewhere in between the two extremes, and it's a continuous domain, not discrete, with a huge variety.
I won't discourage research into this, and I think what they're doing is awesome- I considered doing a master's thesis on this exact idea. I just want to put a nice big disclaimer here that this isn't useful yet, and there's a long way to go.
Edit: Wall of text much mabbo? Woah man, take a breather.
Well I seem to remember when they were giving computers the ability to interpret speech they were pretty limited in what they could do. Even training your own machine with available software at the time required you to go through sequences.
So I see this implementation as the same, its part of the long road. First, start with the basics. Prove the machine can recognize what you intend. Then step it up. From there you can broaden it other dialects and what not.
You made the comment that using this system would be like speaking in a monotone with a blank face and no eye contact. I don't know sign very well, but I was wondering whether a better analogy would like having a conversation over IM? That medium lacks the same non-verbal cues you mentioned.
This would be a good application for google glass (would only work one way). Other than that, I see no benefit over a simple pencil and paper for face to face communication, or a simple text chat for anything other than face to face. Although I will say that I have had text chats with deaf people before, and the text was definitely based off sign language (at times it was difficult to understand because the grammar is so different).
Written language and reading is often not as fluent for deaf people as hearing: written English is based (loosely) on how the words sound after all, and if you've never heard them, you lose a lot of the useful tools to map between text and meaning. It's ironic that many people's first steps in sign language are to learn to sign the alphabet - as a tool it's almost useless for communicating with a deaf person.
Really? You see no benefit to video for communication instead of just typing in a chat window? What about facial expression and body language and every other reason people use video over audio regularly?
One benefit I see over a simple pencil and paper is the ability to translate between languages. I believe they showed an example of that in the video in which Chinese Sign Language was translated to English presumably on the fly.
I cannot speak for other sign languages, but ASL uses a few different forms of word order -- Object, Subject, Verb is used a good bit. There are often rhetorical questions thrown in that are immediately answered by the signer. Facial expression is also very important to convey meaning.
A few examples:
The signs for 'wish' and 'hungry' look the same. The difference is the facial expression.
If you wanted to sign, "All we want to do is eat your brains", the order would most likely be: 'We want do-what? Your brains eat.'
There is another form of signing called 'Signed English.' It borrows most of ASL's signs but puts them in standard english order. It has been a bit controversial in the past, though.
You also run into odd colloquialisms or regional signs. I'm not quite sure what to call them. A quick example is the word 'grass.' I learned the sign for grass but, for some reason, that sign means 'truck' where I live currently.
Anyway, I'm rambling. If you have any specific questions I can try to answer them.
Another good regional example is cities. Illinois signers have a specific sign for Chicago, for instance, whereas outside the state it's just finger spelled.
Sign is incredibly conceptual and has a large spatial component. This is definitely a cool step though.
It seems odd at first, but 'ASL Songs' allow deaf and hard of hearing individuals to better experience that bit of culture (music.) Some folks are simply hard of hearing and may have trouble understanding what they're hearing.
ASL can be very expressive and can be an art form itself. There's a neat thing in the deaf culture called 'ABC Stories.'
Basically ASL (and other forms of sign language) use constructs and tense that don't necessarily translate well into specific words without lots of experience.
If text was such preferred and dominant ways of communication for deaf people then sign language would not have been invented in the first place. The fact that it is used more commonly to communicate with people with hearing disabilities tells me that this is interesting.
Besides this is a research project. Just goes off to show the potential a full body recognition platform such as Kinect can have.
I remember doing Sign Language recognition through Kinect in my Master's thesis. I did it for a vocabulary of 140 words of Indian Sign Language with a success rate of 70% using the Machine Learning approach. Here is the video of the prototype that I built:
www.youtube.com/watch?v=2oqD-_UCHxQ
Good on them! I remember in my Masters program contemplating gesture recognition with sign language. Ultimately I went with proving people are who they are based on their body language; but this was number 2 on the list.
Does anyone know what techniques they are using to accomplish this? I did the changes in Active Appearance Models, converted them into 'sound files', then ran them through Microsoft's HTK Markov Model system to get my stuff.
> While this is clearly a massive achievement, there is still a huge amount of work ahead. It currently takes five people to establish the recognition patterns for just one word.
I don't get it. They're saying they need five (only five!) seperate people to train the thing, and then it works pretty well for everyone? That seems to be working pretty darn well to me, I don't see the problem or 'huge amount of work'.
Probably something lost in the journalism.