From: Andrew Clough (aclough@mit.edu)
Date: Thu Feb 27 2003 - 16:23:06 MST
At 01:20 PM 2/27/2003 -0500, you wrote:
>On Thu, Feb 27, 2003 at 02:15:38PM +0100, Anders Sandberg wrote:
> > I think the important point about the story was that it showed how
> > "friendliness" can go wrong, not in the traditional "evil AI goes on
> > rampage across the world" sense or the equally tranditional "nice AI
> > creates an utopia but humans have no meaning without struggle" (which
> > it mimicks on the surface). Here the issue is that it is a too rigid
> > friendliness: yes, it will protect people and give them what they
> > want, but in an inflexible way.
>
>I think the problem is not rigidity. The problem is that the SI is not
>really an SI. It's not especially intelligent. It can be fooled by
>unaugmented humans, and it doesn't seem capable of augmenting human
>intelligence (the story doesn't explicitly say this, but it's implied
>because nobody in the story asks for augmented intelligence or encounters
>anyone with augmented intelligence). What kind of an SI would have
>unintentionally almost allowed somebody to die from rabies?
I was actually able to find a passage which explains why augmentation is
impossible. In the second chapter, right before the singularity, Lawrence
enters this into his creation: "Force Association: Interpreting the
contents of a human being's mind in order to understand or predict its
behavior shall be a violation of the First Law of severity two." This seems
to pretty much prevent the enhancement of humans by the budding SI.
The author goes into a bit of depth on exactly how the creator did so badly
in making his SI:
http://www.kuro5hin.org/prime-intellect/mopising.html
In short, the fictional programmer hadn't read anything by Eliezer on how
to do it right.
Never attribute to malice what can be explained by stupidity. Don't assign
to stupidity what might be due to ignorance. And try not to assume you
opponent is the ignorant one-until you can show it isn't you.
-M.N. Plano
This archive was generated by hypermail 2.1.5 : Thu Feb 27 2003 - 16:31:01 MST