Re: AI big wins

Robin Hanson (
Fri, 25 Sep 1998 15:34:22 -0700

In response to my question:
>> >> Mere mention of the work "feedback" is not sufficient ...
>> >I didn't just "mention" it; I talked about the behavior of the sum of the
>> >series of I'1 = C(O, P, I), I'2 = C(P, O, I + I'1), ...
>> Why is a strongly convex C more realistic?

Eliezer simply reposts the text of his earlier post:
>If optimization yields a small jump, then the next increment of
>optimization is likely to be zero, ... If optimization yields a big jump,
>... the AI is likely to redesign itself in a fairly major way ... at least
>1.0 to 1.1. ... either these large steps keep repeating to superintelligence,
>or at some point the AI can't redesign or optimize itself. ... if you're
>talking about the slow reworking of code, line by line, you're really talking
>about a large jump in slow motion because the AI is slow ... it's all
>part of the same "increment", the same I' or O'. If the partial reworking
>adds even more intelligence, then the equation runs even faster.

Let's say that at each moment the AI has a list of optimization ideas to try, and some indication of which ideas are more promising. Let's assume per your claim that each optimization will either fail completely, or result in a 10% improvement. There are then two tasks: evaluate and implement each idea.

As the AI progresses there are contrary forces working for and against accelerated growth. As the AI gets more optimized, it is able to implement any one optimization idea in a short time. It may also be able to evaluate each idea in a shorter time. But working against this is the fact that the AI will wisely first work on the easiest most likely to succeed ideas. So as time goes on the AI has to evaluate more and more ideas before it comes to one that works, and it takes more and more work to implement each idea.

So why should AI abilities get better faster than the problems get harder? This is not our experience in other analogous areas of learning, such as improving computer hardware.

Robin Hanson RWJF Health Policy Scholar, Sch. of Public Health 510-643-1884 140 Warren Hall, UC Berkeley, CA 94720-7360 FAX: 510-643-8614