Geniebusters (was: RE: Yudkowsky's AI (again))

Lyle Burkhead (
Fri, 2 Apr 1999 17:41:01 -0800

There are three questions at issue here. In decreasing order of importance, they are:

  1. Whether my method of argument is valid and useful.
  2. Whether my conclusions are true.
  3. Who I am; whether I know anything about computer science, or anything else; whether I did a good job of applying the principle of calibration; etc.

Question #3 is of no relevance. Maybe I'm an idiot. That has nothing to do with it. As I said on the home page of geniebusters, Eric Drexler is not the issue here. Neither am I.

Question #2 is more important. Is someone going to design the entire "universal assembler," in advance, in such a way that once set in motion the whole thing suddenly comes into existence with explosive force? Are robots going to build skyscrapers for free? Is there going to be an event that is incommensurable with everything else? or not?

Question #1 is the most important one. How can we get a handle on question #2? Is there a general procedure for approaching such questions, so that they become matters of fact and proof rather than matters of opinion and speculation?

After writing the above, I finally looked at Billy Brown's comments and found, to my surprise, that they are rational. If he had made a rational comment in the first place, matters would have taken a different course.

> Ah. So you do still read the list. I wondered.

You wondered? In other words you were on the list two years ago? Anyway I don't read the list. Twirlip of Greymist pointed out to me that Anders had mentioned geniebusters on the list, so I looked at his comment and your reply. Then I read the earlier posts in that thread, and sampled some other threads in the archive. Apart from that, I haven't read the list since I retired from it. However, this afternoon I find myself writing yet another post to the list, the fourth one this week, in spite of my repeated efforts to stop. I may have to join some kind of 12-step program for postaholics. Maybe a Higher Power will help me do what I can't do myself.

> Certain sections of your site make it seem as if
> your method is to assume that any revolutionary claim
> must be false, and then look for an excuse to back up
> your assumption. This is just as wrong-headed as
> assuming that all such claims are true.

You don't say which sections you are referring to, or what "makes it seem" that way. I am concerned here with certain specific claims made by Eric Drexler in Engines of Creation. I don't even dispute all of the claims made in that book, let alone "any" revolutionary claim. If someone wants to claim that we will, in the foreseeable future, be able to make our bodies much more self-correcting than they are now, and that our cells and organs will become (asymptotically) perfect, I would not have a problem with that claim, even though it could be considered "revolutionary." If someone wants to claim that it is possible (and practicable) to make artificial organisms, I don't have a problem with that either. The assumption that I do make, and defend, is that new technologies, however revolutionary they may appear to be, will occur within a certain unchanging framework.

> Your assertion that any software capable of replacing a human
> would demand
a salary

I'm not asserting this for software capable of replacing *a* human, just for software that replaces humans who make unsupervised decisions requiring

human judgment. Some humans are not on "our" side of the line. Many employees are, for all practical purposes, automatons, and can be replaced by machines.
This happens all the time and will continue. My assertion is that the realm of automation will always exist within a larger realm that is not automated. The boundary between agents and automatons will always be there, and agents will always have to be dealt with as agents. They will always expect to get paid (unless you can find some other way to motivate them). I think what I'm going to have to do is illustrate this point with an animated diagram, where you can see something changing within a framework that does not change. I also need to put in a lot more interstitial material explaining what I'm doing. (In other words the revised version will be longer, not shorter. It isn't going to be an article, it's going to be a book. Sorry, Robin ;-)

> The comparison of a truly general-purpose nanomanufacturing system
> to a modern industrial nation is such an oversimplification that it has
> no predictive value. There are too many differences between the behaviors

> of the two systems, and the implications of these differences need to be
> assessed individually and in detail if you want to make a meaningful

That section isn't supposed to be read in isolation. It is part of the argument leading up to Exercise 5 in Section 7, and it also has to be read in conjunction with sections 12 and 13. The question is whether the Assembler Breakthrough is going to happen. To get a handle on that question, I'm trying to establish the level of complexity involved in "a system that can make anything, including copies of itself." If a nanosystem can make anything, in the sense required for the Assembler Breakthrough, then it will amount to the same thing as an industrial economy.

It would of course be possible to have a nanosystem that could make anything in a certain domain, maybe an extensive domain, without being able to make anything in general. But that's not what you need for the Assembler Breakthrough. If anyone has a proof (i.e. a structured argument, not some impressionistic remarks) that the Breakthrough can happen with a less general system, I'd like to see it.