Re: Language in the meat machine

hal@finney.org
Fri, 16 Jul 1999 08:58:23 -0700

Jeff Davis, <jdavis@socketscience.com>, writes: 

>                           NEW WORD ORDER 
>
>               THE ATTACK OF THE INCREDIBLE GRADING
>                                  MACHINE
>
>                            BY CLIVE THOMPSON 
>
> At or near:

>
> http://www.linguafranca.com/

Actually at http://www.linguafranca.com/9907/nwo.html

This is a technique for grading student essay papers which is supposedly as accurate as professors' grades:

: But the technique Landauer came up with is an altogether different
: beast. His software completely ignores style, grammar, and syntax. In
: fact, it relies on none of the familiar rules of language at all. It
: concerns itself solely with the following question: Does the student's
: essay use words appropriate to the subject matter? To answer this
: question, Landauer's software performs a series of operations. First,
: it assembles a customized database of texts on the assigned topic. It
: measures the spatial relationships among all the words in these texts,
: noting where each word appears and which other words it is near. The
: software then performs a similar analysis on the student essay. Finally,
: it compares the student essay with the texts in its database. The theory
: behind the method is this: For any given essay, good content is a function
: of using certain words in the vicinity of certain other words, and that
: accomplishment can be expressed numerically.

:

: It sounds too complicated (or too simple) to be any good, but Landauer
: and his team claim their program can crank out grades as accurately as
: any professor.

This has an obvious flaw. The professor is checking for style, syntax, and logic. If students know that this program is going to check their papers, all they need to do is to write something that produces the right words in roughly the right positions, without regard to sentence structure.

There have been "travesty generators" around for years which will take a text and chop it up into words or phrases and output them at random. The resulting mishmash almost sorta makes sense when you look at it a piece at a time, but it is actually complete gibberish. The text editor Emacs has had for 20 years a function called dissociated-press which does this. My guess is that loading a couple of standard references no the topic into your buffer and running dissociated-press will produce something that passes these guys' test with flying colors, but which would get a big fat F in an instant if a human looked at it.

Hal