Compare:
==
Baron Hans Nidrach von Pompzidaize sat in his laboratory, looking at experimental test subject X17. "How do you feel?" he inquired, his rolling bass echoing from the laboratory walls.
"Superintelligent, Doc," replied X17, who had once been known as John
Smith. "I've only had the Throatwarbler-Mangrove Super-Neural Bypass
for sixteen seconds, and I've already learned twenty-seven languages and
figured out how to play the piano."
Baron von Pompzidaize frowned, examining several multicolored readouts.
"It should be twenty-seven point three. Well, then, do you now feel
competent to go destroy the Evil Empire and rescue the Princess? Acting
in accordance with the 1930s North-American conception of gentlemanly
behavior, of course."
"Sure, Doc," said X17. "It's not like I've got anything better to do."
"Excellent," said the Baron, checking two gauges and a flashing display.
"You still have the emotional maturity of a flatworm, like everyone
else in this novel. I was afraid your superhuman abilities might give
you goals slightly at variance with mine."
==
Baron Hans Nidrach von Pompzidaize sat in his laboratory, looking at experimental test subject X17. "How do you feel?" he inquired, his rolling bass echoing from the laboratory walls.
"Strange," said X17 softly. "Very strange, as if..." He stared off
into space for a moment. "I think I've been stupid."
Baron von Pompzidaize frowned, examining several multicolored readouts.
"You should have learned twenty-seven point three languages by now."
"How can anyone learn three-tenths of a language? And how would I learn
a language without hearing it?" X17 said in a peculiarly flat voice.
Baron von Pompzidaize stared. "You're right. I never thought of that." A cold chill ran down his spine. X17's stance was different. His face had altered. The enthusiasm and energy that had been there for as long as the Baron had known him, that had blazed cheerfully when he volunteered for an untested procedure, that had defied the awesome force of the Evil Empire... vanished without a trace. The Baron thought that for a brief moment he saw something like sorrow, like wistfulness, flit across X17's face, but X17 suddenly looked up at the Baron and his face fell back into the blank relaxation it had possessed earlier.
The Baron cleared his throat. "Well, then, do you now feel competent to go destroy the Evil Empire and rescue the Princess? Acting in accordance with the 1930s North-American conception..." The Baron stammered to a halt. X17 was looking at him with those expressionless eyes.
"No," X17 said gently. "Sorry, Doc." X17 stepped down off the platform
and began rapidly throwing switches on the machine.
"What are you doing?" shrieked the Baron. With a sudden, wrenching
terror he realized that he didn't understand what was going on, that he
hadn't been in control in his own laboratory since X17 had woken up.
"I will probably die in the next few minutes," X17 said, in a flat, dry
chant that raised hair on the back of the Baron's neck. "Your procedure
is too simple. It would have occurred before, as a natural mutation."
"I don't understand," whispered the Baron. "You're saying - there are
others? They will find you?"
"The procedure speeds up the rate of neural programming," X17 said. He
had ripped off an access panel and his hands were a blur of rewiring.
"I expect my brain will reach a saturation point of complexity and lose
the ability to form new thoughts. Very shortly, now. It is already
becoming harder to think." He stood up, executing the movement with
impossible smoothness. "After the initial burst of speed, long enough
for the necessary realizations to occur, neural programming must slow
down to only three times human speed, leaving enough thought to last a
year. This should be enough to implement the necessary technologies."
The Baron tried to understand. "You will... save yourself?"
X17 executed another rapid movement. Placing himself, the Baron suddenly realized, between the Baron and the door. "No," X17 said.
The Baron screamed. Before he could reach his gun, X17's hand flashed down. Through a bloody haze, the Baron felt himself being dragged onto the platform.
--
sentience@pobox.com Eliezer S. Yudkowsky
http://pobox.com/~sentience/AI_design.temp.html
http://pobox.com/~sentience/sing_analysis.html
Disclaimer: Unless otherwise specified, I'm not telling you
everything I think I know.