>From: "Eliezer S. Yudkowsky" <sentience@pobox.com>
>Zero Powers wrote:
> > But most likely I’d
> > probably feel that these far lesser intelligences were not worthy of my
>time
> > and attention and, worse, I’d feel that they were trying to exploit me
>by
> > asking me to devote my time and energy to their “puny, little problems”
>when
> > I could be out instead exploring the fascinating great beyond.
>
>Looks to me like you just answered your own question. That is how *you*
>would
>feel. Why? Because you are an evolved human, with all manner of
>sophisticated, evolved hardware which establishes observer-biased goals and
>observer-biased beliefs and all manner of social perceptions for detecting
>cheating and avoiding exploitation... as was an advantage in your ancestral
>environment.
>
>You can't reason from your own intuitions about goals to the actions of
>superintelligence.
>
>If, as seems to be the default scenario, all supergoals are ultimately
>arbitrary, then the superintelligence should do what we ask it to, for lack
>of
>anything better to do.
I'm sure that I cannot, from my evolved biological milieu, accurately
predict how the AI would feel. But I assume: (1) it will be curious (2) its
intelligence and knowledge will grow exponentially and (3) once it reaches
human level intelligence, it will *far* exceed human levels in very short
order. Given those assumptions, you really think it will do what we say for
lack of something better to do? I don't see it.
Seems to me any super-intelligence would have to have virtually zero
curiosity to sit around granting human wishes and obeying human orders for
lack of having any other interests worth pursuing. And, it seems like a
super-intelligence without curiosity would be impossible to engineer.
Unless it has some internal motivation to seek out knowledge on its own, all
its knowledge would have to be force-fed. Meaning that it could never know
more than those who are feeding it, no?
-Zero
Learn how your computer can earn you money while you sleep!
http://www.ProcessTree.com/?sponsor=38158
_________________________________________________________________________
Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com.
Share information about yourself, create your own public profile at
http://profiles.msn.com.
This archive was generated by hypermail 2b29 : Mon Oct 02 2000 - 17:37:14 MDT