Re: Geniebusters

Eliezer S. Yudkowsky (
Thu, 08 Apr 1999 20:46:00 -0500

I don't have time to read the Geniebusters site; I just swooped through chapters 18 and 19, since AI is my area of primary expertise.

In chapter 18, Lyle states:
"When I say, "When will we have..." I am looking for answers that take
into account the realities of writing software. What would be involved in writing a program that serves as a coder? How would that compare with writing Windows 2000? Would it be more complex? less? about the same? Is anyone working on such a project?"

I refer him to "Coding a Transhuman AI".

With the background of "Coding a Transhuman AI", I strongly disagree with chapter 19. Since an AI is "native" to the rich domain of code in a way that an unenhanced human never can be, I would expect even a
"coder drone", incapable of acting as architect or foreman - although I
have my doubts about that paradigm - to be capable of outprogramming the entire human race at present, and certainly of outprogramming an average programmer dollar for dollar, no matter how expensive it was.

The vast majority of coding would occur easily, almost reflexively, and thus very rapidly, since conscious thought is required only for a tiny fraction. In this sense, such an AI could be said to be a coding
"genie". But even with domain-specific functions, it could be that
coding is such a general task that a "coding drone" would be (1) independently motivated and (2) rapidly self-enhancing to the point of Powerdom. I personally believe (1) and (2).

My objection to chapter 19, then, is that focusing on human-equivalent coding ability misses the point entirely. We should be asking about the effect of incredibly fast but blazingly stupid coding, or of entirely transhuman coding.

--          Eliezer S. Yudkowsky

Disclaimer:  Unless otherwise specified, I'm not telling you
everything I think I know.