By the way, Eliezer, I have a question/quibble/objection to your doubled doubling description of the singularity: it ignores hardware development cycles. Certainly a fully functional AI could improve its own software, but the runaway would be damped by the hardware design and fabrication process- smarter code also needs faster hardware to run at faster subjective speed.
Granted, an advanced AI could also be a wizard chip designer, capable of pushing out a new design in a fraction of the current time, but chip compilers run no faster for a design generated by an AI than for one generated by humans with good design tools (my boss, Jeff Greason, has war stories of sleeping under his desk while monitoring the compilation of the Pentium chip layout).
Given that wafer fab processes, even for prototype runs, take several weeks for the wafers to wend their way through, a near-term AI driven singularity would likely double no faster than once a month. True, this is about 18 times faster than Moore's law, but it would make the singularity a bit less daunting, and allow more human preparation and adjustment during the process.
Of course, really clever AIs would seek to automate the entire fabrication process, but dramatic decreases in fab latency would likely require a whole new fab- with human labor in the loop keeping the doubling time long.
Eliezer S. Yudkowsky wrote:
> I think some people were extremely skeptical of their metrics -
> something along the lines of "You're counting each transistor as a
> floating-point operation."
-- Doug Jones, Rocket Plumber Rotary Rocket Company