Facial expression recognition software

Danny Yavuzkurt (ayavuzk@fas.harvard.edu)
Wed, 07 Apr 1999 17:41:27 -0400

Greetings, all. I'm a longtime lurker, and just wanted to kick off my active participation on this list with something a little more interesting than 'hi, here I am, I want to talk about extropy'. Here's something potentially very interesting to those of you into AI, which also has tendrils into those of you who worry about government or corporate misuse of intelligent human-monitoring software...

And, of course.. it has a lot to do with better future computer interfaces, operating systems, games...

All in all, one of the most interesting tech articles I've seen in a long time..


Computer Program Recognizes Facial Expressions (04/05/99, 2:36 p.m. ET)
By R. Colin Johnson, EE Times

The Salk Institute for Biological Studies has mimicked the neural networks of the visual cortex to create a facial-expression recognition system that is said to be faster and more accurate than a human expert. The computer program compares video images with stored templates of prototypical expressions, such as "sad" or "angry."

The system's creators said it is as accurate as a trained psychologist and can capture and identify subliminal expressions that pass over a face momentarily, before a "posed" expression is consciously assumed. That makes the technology potentially applicable as a lie detector.

"Computer interfaces could be greatly improved by recognizing a user's expressions," said Salk Institute researcher Terrance Sejnowski, who ran the system through its paces in a study with Paul Ekman, professor of psychology at the University of California, in San Francisco, and Joseph C. Hager at the Network Information Research, in Salt Lake City. "But today, computers have a difficult time [with that], something people do without even thinking."

When trained psychologists are hired to slow down a videotape of a face-to-face meeting, they often find micro-expressions that appear almost instantaneously at the mention of new information. When someone is lying, micro-expressions quickly give way to posed ones. Where it takes as long as an hour for a trained expert to review and score a one-minute tape for micro-expressions, the Salk team's software needs only about five minutes to get what the developers said are the same results.

The 60 prototypical expressions used as program templates were developed by Ekman over the past five years. His analysis breaks facial expressions down into component movements of at least 60 of the most important facial muscles. For example, the sixth template, called T6, encodes crinkling of the eyes, and an upturning of the lips is coded by the 12th template. A smile is then a T6 combined with a T12.

Many muscles, according to the researchers, can give away a person's true feelings without them knowing it, because people are not well versed in the subtleties of facial expressions. Sadness, for instance, is characteristic of the central frontalis muscle: It raises the inner corners of the brows, producing wrinkles in the middle of the forehead. Even in a deceptive attempt to pose an expression, such muscles are difficult to keep contracted for extended periods unless the sadness is genuine.

Ekman reported he once detected the "despair" micro-expression on a suicidal patient who was trying to cover up the fact to get a release from the hospital. With an automated tool, psychologists could have their computers review videotaped sessions for micro-expressions, pinpointing segments the therapist should personally review.

Sejnowski warned investigation has just begun to scratch the surface of facial-expression recognition. Hard engineering work lies ahead, for instance, to compensate for variable lighting and head positions. Improvements are also needed in identifying coordinated combinations of muscle movements.

"...all life is but a set of pictures in the brain, among which there is no difference betwixt those born of real things and those born of inward dreamings, and no cause to value the one above the other..."

-Howard Phillips Lovecraft, "The Silver Key"