This is a multi-part message in MIME format.
--------------75252378C0864762C407C53A Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit
Aargh. I see that this also went only to Hanson.
-- firstname.lastname@example.org Eliezer S. Yudkowsky http://pobox.com/~sentience/AI_design.temp.html http://pobox.com/~sentience/sing_analysis.html Disclaimer: Unless otherwise specified, I'm not telling you everything I think I know. --------------75252378C0864762C407C53A Content-Type: message/rfc822 Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-ID: <360EA8ED.6FBD4731@pobox.com> Date: Sun, 27 Sep 1998 16:06:58 -0500 From: "Eliezer S. Yudkowsky" <email@example.com> Reply-To: firstname.lastname@example.org X-Mailer: Mozilla 4.06 (Macintosh; I; PPC) MIME-Version: 1.0 To: Robin Hanson <hanson@econ.Berkeley.EDU> Subject: Re: AI big wins References: <email@example.com> <firstname.lastname@example.org> <email@example.com> <firstname.lastname@example.org> <email@example.com> <firstname.lastname@example.org> <email@example.com> <firstname.lastname@example.org> <email@example.com> <firstname.lastname@example.org> <email@example.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Robin Hanson wrote:
> >.... You are dealing with fragmentary
> >increments, assuming that the AI's time to complete any one task is the
> >occurrence that happens on our time scale. But I'm thinking in terms of a
> >series of AIs created by human programmers, and that the entire potential of a
> >given model of AI will be achieved in a run over the course of a few hours at
> >the most, or will bog down in a run that would take centuries to complete.
> >... In either case, the programmer (or more likely the Manhattan
> >Project) sighs, sits down, tries to fiddle with O or I and add abilities, and
> >tries running the AI again. ... But what matters is not
> >the level it starts at, but the succession of levels, and when you "zoom out"
> >to that perspective, the key steps are likely to be changes to the fundamental
> >architecture, not optimization.
> The same argument seems to apply at this broader level. The programmer has
> a list of ideas for fundamental architecture changes, which vary in how
> likely they are to succeed, how big a win they would be if they worked,
> and how much trouble they are to implement. The programmer naturally tries
> the best ideas first.
Ah, let me clarify: The succession of levels that the AI pushes itself through, not the succession of levels that the programmer tries. Once again, you have to look at a case where you're not just dealing with a list of ideas a single intelligence comes up with - be it AI or human - but a changing intelligence, and above all a self-altering intelligence. If you don't look at the self-alteration, you're not likely to discover any positive feedback. Each time the AI's intelligence jumps, it can come up with a new list. If it can't come up with a new list, then the intelligence hasn't jumped enough and the AI has bottlenecked. The prioritized lists probably behave like you said they would. It's the interaction between the lists of thoughts and the thinker that, when you zoom out, has the potential to result in explosive growth. -- firstname.lastname@example.org Eliezer S. Yudkowsky http://pobox.com/~sentience/AI_design.temp.html http://pobox.com/~sentience/sing_analysis.html Disclaimer: Unless otherwise specified, I'm not telling you everything I think I know. --------------75252378C0864762C407C53A--