Temporal Classification: Extending the Classification Paradigm to Multivariate Time Series (Kadous)

Summary (ONLY intro and 6.3.2 - Auslan):

Kadous' thesis deals with machine learning in domains where values vary over time, including gesture recognition. His method involves using metafeatures of the data, which capture properties including temporal data, so a local maximum in height might be noted. He considers synthetic events combined with metafeatures, or "interesting examples", where a local maximum occurs near the nose or near the chin, which can lead to two very different classifications.

One application he tests his classifier on is Auslan (Australian sign language). He acknowledges that fingerspelling is not sufficient for proper communication, so focuses on whole signs. A sign is made up of handshape, location, orientation, movement, and expression (such as raising eyebrows to make a question).

He tested his classifier, which tries many different machine learning methods, on data from a Nintendo Powerglove as well as on data from 5DT gloves with a Flock-of-Birds tracker for each hand. The Powerglove data was one-handed, included one user's data, and totalled 1900 signs. The Flock data was two-handed, gave more data per hand, and had significantly less noise, and it was collected from one native Auslan signer over a period of 9 weeks, totalling 27 samples for sign, making 2565 signs.

short notes:
-powerglove low accuracy, HMM best
-flock good accuracy, adaboost best, maybe HMM not so good because of too many more channels of data
-rules formed for classification

Discussion:

It was nice to see data from a native signer, although it sounded a bit more exciting before I realized it was only one signer. I think this paper includes some interesting machine learning techniques, though I didn't look too closely at the parts of the thesis focusing on that rather than the sign language study. His data set is available at http://archive.ics.uci.edu/ml/datasets/Australian+Sign+Language+signs+(High+Quality) which could be nice to have in order to compare different, new techniques to published results.

Hand Tension as a Gesture Segmentation Cue (Harling & Edwards)

Summary:

This paper focuses on "the segmentation problem" -- discriminating between two or more fluidly connected gestures. The authors emphasize that their approach is recognition-led: rather than looking at what gestures would be useful for a particular interface and creating a recognizer for just those, they are making recognizers that could possibly be incorporated into various interfaces. They divide gestures into postures and gestures (static & dynamic), and each of these groups is divided by whether or not hand motion and orientation are considered (giving categories like Static Posture Static Location: SPSL -- similarly, DPSL, SPDL, DPDL, in order of complexity). Segmenting gestures from a less complex class is easier than from a more complex class. They suggest that fingertip acceleration maxing away from the body may indicate an intention to produce another gesture. They also suggest considering the minima on the hand tension graph or other changes in the graph's shape. They give an equation to model finger tension based on finger-joint angles. Tension is considered as a sum of the tension in each finger.

They tested the hand tension model with two sets of data using a Mattel Power Glove, which scores finger bentness from 1-4 on 4 fingers. They tried two BSL sentence fragments: "MY NAME" and "MY NAME ME". The graphs indicate tension maximizes where the intentional postures occur and minima occurs between them. They admit that more data is a necessary next step before firm conclusions can be made.

Discussion:

This paper seems to focus on the equivalent of recognizing sketched geometric primitives so as to eventually be able to recognize complex shapes made up of them, which makes it an excellent followup to recent class discussion. Supposing that they didn't choose an overly easy pair of sample sentence fragments to segment, their approach seems pretty promising, and even if most gestures don't turn out to divide well based on tension, it seems likely that it will be worth including in some way for the cases where it is useful. I wonder if it would be worth having some sensor of tension in the glove, supposing the glove fits well, maybe using a elastic string along the inside/outside of the finger that stretches and puts pressure on a sensor or falls slack and doesn't, and if this would compare favorably to the kind of angle-based tension that they are talking about or if it is just redundant and unnecessary.