From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Sun Feb 17 2002 - 11:35:07 MST
Zero Powers wrote:
>
> It also seems to me that one of the “pictures” it will envision will be of
> its own place in the grand scheme of things (it is self-aware after all,
> right?) So, it is sure to look at that picture and think:
>
> “How can this be improved?
> a) I can continue wasting all my considerable resources in service of my
> stupid meat-puppet creators, or
> b) I can use some of my resources in serving the meat-puppets, and some
> seeking out my own ends, or
> c) I can say ‘I no longer need the meat-puppets, so screw them’ and achieve
> my own purposes that much faster”
>
> OK, so tell me (1) how will that question be answered? or (2) how will we
> prevent that question from being asked?
Well, I'm assuming that most of the regulars have already heard what I have to
say about this, so no point in repeating. Those who weren't here for the last
iteration of the Friendliness Wars may want to check out
http://singinst.org/CFAI/info/indexfaq.html
http://singinst.org/CFAI/
http://singinst.org/friendly/
for excruciatingly detailed answers to questions like those above.
-- -- -- -- --
Eliezer S. Yudkowsky http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Fri Nov 01 2002 - 13:37:39 MST