Georgia Tech Gesture Toolkit: Supporting Experiments in Gesture Recognition (Westeyn, Brashear, Atrash, Starner)

Summary:

This paper describes a toolkit, GT^2k, to aid in gesture recognition via HMM, which makes use of another existing toolkit, HTK, that supports speech recognition. GT^2k allows for training models with both real-time and off-line recognition. They discuss four sample applications for which their toolkit has been applied. A gesture panel in automobiles recognizes simple gestures with camera data and gets 99.2% accuracy. A security system is built to have recognition of patterned blinking, to go with face recognition, so that the system can't be fooled with a still photograph of an authorized person, and it gets an 89.6% accuracy. TeleSign is a system that does mobile sign language recognition, but real-time recognition was not yet implemented, but they got 90% accuracy by combining vision and accelerometer data. A workshop activity recognition system is built to recognize actions made while constructing an object in a workshop, such as hammering or sawing, based on accelerometer data, and they found 93% accuracy, though again not in real-time.

Discussion:

The blinking idea is interesting, but I have a mental image of someone with holes cut in a photograph, maybe with fake eyelids attached that blink when a string is pulled. Plus, from an interface standpoint, it seems like most people who have already been trained by today's hand-operated devices would be more comfortable using their hands to input a code. Maybe an input pad hanging on a cord that the user can pull close and hide from people standing behind them would be a simpler solution.

It does seem like a valuable idea to have a toolkit that can be used in multiple gesture recognition applications, and anything that could improve accuracy could benefit a lot of people.

A Similarity Measure for Motion Stream Segmentation and Recognition (Li & Prabhakaran)

This paper deals with recognizing motion streams, as generated by human body motions like sign language, by using a SVD-based similarity measure. They represent position with a matrix: columns represent positions of different joints, and rows represent different instants in time. Motions are considered to be similar if the matrices that represent them are similar in that they have the same number of attributes (columns), but they may have a different number of rows, since a fast gesture can have the same meaning as a slow one. They discuss how SVD shows the geometric structure of a matrix and can be used to compare two matrices. They performed a study with a CyberGlove, covering 18 motions (ASL for Goodbye, Idiom, 35, etc.), with 24 motion streams generated, with 5 to 10 motions in a stream. This data allowed consideration of segmentation issues. They also got motion capture data from 62 isolated motions from dance movements (each repeated 5 times). They find near 100% accuracy for isolated motions and around 94% accuracy in motion streams with their kWAS algorithm (looking at the first k eigenvectors with k=6). It is much more accurate and faster than EROS and slightly better than MAS, which are the other algorithms they compare theirs to.


Discussion:

I appreciate that they used words that would be used in regular ASL rather than just letters, since having a broad selection of recognized words will probably be more useful than finger spelling for native speakers of ASL. It's also to see an unusual application like dance, even if it does include only isolated motions -- I'm curious as to how complex these repeated short motions were, and if it was enough to count as a basic step that the dance as a whole might be made up of, and so someone might do repeatedly as practice to master that basic step, or if it is just a brief segment of the dance that is not common to be practiced by itself. It might also be interesting to see how well a practice step's motion data corresponds to the data for that same step integrated into a more complicated set of dance motions, and if there is a greater difference between these cases for expert versus novice dancers.

Hand Tension as a Gesture Segmentation Cue (Harling & Edwards)

Summary:

This paper focuses on "the segmentation problem" -- discriminating between two or more fluidly connected gestures. The authors emphasize that their approach is recognition-led: rather than looking at what gestures would be useful for a particular interface and creating a recognizer for just those, they are making recognizers that could possibly be incorporated into various interfaces. They divide gestures into postures and gestures (static & dynamic), and each of these groups is divided by whether or not hand motion and orientation are considered (giving categories like Static Posture Static Location: SPSL -- similarly, DPSL, SPDL, DPDL, in order of complexity). Segmenting gestures from a less complex class is easier than from a more complex class. They suggest that fingertip acceleration maxing away from the body may indicate an intention to produce another gesture. They also suggest considering the minima on the hand tension graph or other changes in the graph's shape. They give an equation to model finger tension based on finger-joint angles. Tension is considered as a sum of the tension in each finger.

They tested the hand tension model with two sets of data using a Mattel Power Glove, which scores finger bentness from 1-4 on 4 fingers. They tried two BSL sentence fragments: "MY NAME" and "MY NAME ME". The graphs indicate tension maximizes where the intentional postures occur and minima occurs between them. They admit that more data is a necessary next step before firm conclusions can be made.

Discussion:

This paper seems to focus on the equivalent of recognizing sketched geometric primitives so as to eventually be able to recognize complex shapes made up of them, which makes it an excellent followup to recent class discussion. Supposing that they didn't choose an overly easy pair of sample sentence fragments to segment, their approach seems pretty promising, and even if most gestures don't turn out to divide well based on tension, it seems likely that it will be worth including in some way for the cases where it is useful. I wonder if it would be worth having some sensor of tension in the glove, supposing the glove fits well, maybe using a elastic string along the inside/outside of the finger that stretches and puts pressure on a sensor or falls slack and doesn't, and if this would compare favorably to the kind of angle-based tension that they are talking about or if it is just redundant and unnecessary.