Re: Yudkowsky's AI (again)

Eliezer S. Yudkowsky (sentience@pobox.com)
Wed, 24 Mar 1999 09:09:02 -0600

Thanks for changing the subject line!

I'm not going to go over this again, mostly because the old section on Interim Goal Systems is out of date. I'll just say that the IGS actually doesn't make any assumption at all about the observer-relevance or observer-irrelevance of goals. The AI simply assumes that there exists one option in a choice which is "most correct"; you may add "to the AI" if you wish. Even if it doesn't have any goals to start with, observer-relevant or otherwise, this assumption is enough information to make the choice.

In short, I have decoupled Interim Goal Systems from the concept of objective morality. The Prime Directive now states: "Maintain choices as the holistic function of reasoning." That is, goals should be distributed through the knowledge base instead of concentrated in a few syntactic tokens. That way they reassemble themselves if the architecture changes, and also there are major problems with trying to maintain coercions which are imposed as special cases.
-- 

        sentience@pobox.com          Eliezer S. Yudkowsky
         http://pobox.com/~sentience/AI_design.temp.html
          http://pobox.com/~sentience/singul_arity.html
Disclaimer: Unless otherwise specified, I'm not telling you everything I think I know.