In response to my question:
>> >> Mere mention of the work "feedback" is not sufficient ...
>> >I didn't just "mention" it; I talked about the behavior of the sum of the
>> >series of I'1 = C(O, P, I), I'2 = C(P, O, I + I'1), ...
>> Why is a strongly convex C more realistic?
Eliezer simply reposts the text of his earlier post:
>If optimization yields a small jump, then the next increment of
>optimization is likely to be zero, ... If optimization yields a big jump,
>... the AI is likely to redesign itself in a fairly major way ... at least
>1.0 to 1.1. ... either these large steps keep repeating to superintelligence,
>or at some point the AI can't redesign or optimize itself. ... if you're
>talking about the slow reworking of code, line by line, you're really talking
>about a large jump in slow motion because the AI is slow ... it's all
>part of the same "increment", the same I' or O'. If the partial reworking
>adds even more intelligence, then the equation runs even faster.
Let's say that at each moment the AI has a list of optimization ideas to try,
and some indication of which ideas are more promising. Let's assume per your
claim that each optimization will either fail completely, or result in a 10%
improvement. There are then two tasks: evaluate and implement each idea.
As the AI progresses there are contrary forces working for and against
accelerated growth. As the AI gets more optimized, it is able to implement
any one optimization idea in a short time. It may also be able to evaluate
each idea in a shorter time. But working against this is the fact that the
AI will wisely first work on the easiest most likely to succeed ideas. So
as time goes on the AI has to evaluate more and more ideas before it comes
to one that works, and it takes more and more work to implement each idea.
Robin Hanson
hanson@econ.berkeley.edu http://hanson.berkeley.edu/
RWJF Health Policy Scholar, Sch. of Public Health 510-643-1884
140 Warren Hall, UC Berkeley, CA 94720-7360 FAX: 510-643-8614