Vocal Joystick controls PCs for those with hand injuries

An artist used a software program called Vocal Joystick to create this drawing of Mount Fuji. The software gives people with reduced or nonexistent use of their hands the ability to control a computer cursor by mouthing vowel sounds.

(Credit: Daniel Terdiman/CNET News)

SEATTLE--For many Iraq war veterans who have returned home with debilitating injuries that, for example, make it impossible to use their hands, doing anything on a computer is a hopeless task.

But a research project under way in the University of Washington's electrical engineering, linguistics and computer science departments could be the latest tool at such veterans' disposal, as well as for anyone who lacks the full use of their hands.

The project, known as the Vocal Joystick, is designed to allow someone to control a computer cursor using nothing more than their voice.

University of Washington graduate student Jon Malkin, who spoke at the Gnomedex conference here Saturday, described it is an extension of speech recognition technology.

It works by having a user train the Vocal Joystick software with his or her voice.

"We can do a lot with that," Malkin said. "Speech is a very complex signal."

Vocal Joystick is being developed by students and faculty in the electrical engineering, linguistics and computer science departments at the University of Washington. Users can move a cursor around by mouthing a series of vowel sounds that correspond to different directional movements.

(Credit: Daniel Terdiman/CNET News)

Indeed, he explained that the software allows a user "four degrees of freedom," meaning that the cursor can be moved up, down, left and right, all with different vowel sounds.

For example, to move left, the user would sound out the "ee" tone in "feet." To go right, it would be the "aw" sound in "law." Hitting the equivalent of the mouse button would be the "u" sound in "but."

It might be tempting to think that only the most basic tasks are possible with this kind of software, but Malkin said that one artist who has used the Vocal Joystick had employed it to paint a picture of Mount Fuji that he then showed the audience.

The picture was very good, something that many people with full use of their hands would never have been able to create.

He said the artist had created the picture in just three hours.

One simple manifestation of the project is this game in which a person controls a fish that must swim around to eat other fish.

(Credit: Daniel Terdiman/CNET News)

Malkin also demonstrated the software in real-time, showing how it is used in conjunction with a simple game where a player controls a fish swimming around trying to catch other fish.

He proceeded to sound out vowel after vowel, and sure enough, on-screen, his fish moved around dexterously, chomping up snack after snack. The Gnomedex crowd went wild.

[eric rice]

I was surprised to learn during this session that MS Paint and Dragon NaturallySpeaking can be used to create 'verbal drawings' as well (although, a bit more primitive than Vocal Joystick's example).

[inachu]

Works with World of Warcraft?

[shoodoo(4est)]

This is amazing.....i wanna play CS 1.6 with it