Re: Geniebusters

Lyle Burkhead (
Tue, 13 Apr 1999 17:15:26 -0700

Eliezer writes,

> Discussions of nanotechnology and AI have advanced
> far beyond the popularized simplification you call the "genie"

The "genie" hypothesis is simply that AI systems with at-least-human intelligence will work for us for free. Discussions of AI have not gone beyond it, because it is a general idea that has nothing to do with the details of how an AI is created.

Here is what I wrote. If you disagree with any part of this, I would like to know what your disagreement is:

Project: I'm going to dig a tunnel and lay a pipeline (or grow a tunnel with a pipeline in it) from my back yard to the Saudi Arabian oil field, and pump the oil out.

This would seem to be beyond the capability of an individual. But suppose I have the Exxon Corporation at my command. I call the CEO into my office, and tell him: "I am going to lay a pipeline to Saudi Arabia, as described in this spec." At this point I give him a copy of the spec. He reads it attentively, then looks back up at me. I continue: "Have a meeting among your top executives this afternoon to plan the work and assign specific tasks to each division. Apply all your resources to this task, and have the pipeline completed in five years." He nods obediently.

The executives of Exxon do have the meeting, as instructed, then go back to their respective divisions, and start organizing themselves for their new task. They have to do a certain amount of retooling and retraining, since in the past they have worked on other things besides tunnels. They also have to hire some subcontractors. The subcontractors drop whatever else they were doing, and work on my pipeline. All the companies throw themselves into this task a hundred percent, and they do in fact have the pipeline completed on schedule.

Do you see anything implausible in this scenario?

Now, suppose there are mite-sized humans (or mite-sized robots with human intelligence). There is a one-to-one correspondence between the mite-sized people and the employees of the corporations discussed above. They have the same capability, on their level, that Exxon and its subcontractors have on our level. They have equivalent tools, the same ability to acquire or invent new tools, similar computers, the same access to outside consultants and subcontractors, the same corporate organization, everything. They are clones of Exxon and the subcontractors.

Let's go through the same scenario again, except this time I am not addressing a CEO of my own size, I am addressing a mite-sized CEO. I give him the same instructions as above, in the same peremptory tone, with the same result.

Is scenario #2 more plausible than #1? Why? Why would a mite-sized Exxon take orders from me, when the actual Exxon wouldn't?

If they were even smaller than mite-size -- if they were nanobots -- would that make it any more plausible? Why would a nanoscale Exxon take orders from me, when the macroscopic Exxon wouldn't?

Let's define a Genie as an entity with at-least-human intelligence and sensorimotor ability, who works for free. This is distinguished from a genie machine, which is defined as an entity that can make whatever it is told to make, including copies of itself, with no effort on your part. The two concepts are distinct but related -- a Genie is the AI system that makes genie machines possible. (By the way, the word "Genie" with a capital G does not occur in Engines of Creation -- that's just a rhetorical device I have introduced to emphasize the absurdity of the idea of an AI system that works for free. The expression "genie machine" does occur, however.)

Nanotechnologists assume that Genies will exist. That's what distinguishes Drexlerian Nanotechnology from ordinary technology. The nanotechnology meme isn't really about atoms. The key paragraphs are in the section called Accelerating the Technology Race on page 81:

> This transformation is a dizzying prospect. Beyond it, if we survive,
> lies a world with replicating assemblers, able to make
> whatever they are told to make, without need for human labor...
> Eventually, some AI systems will have both great technical ability
> and the social ability needed to understand human speech and wishes.
> If given a charge of energy, materials, and assemblers, such a system
> might aptly be called a "genie machine." What you ask for, it will

Size isn't the point here. Eric Drexler's "dizzying prospect" doesn't depend on moving atoms one by one -- that's a red herring. What makes the prospect dizzying is the idea that entities with at-least-human intelligence will do our bidding. The mere fact that the assemblers make what they are told to make isn't the point either. As I pointed out in an earlier section, we already have a system (the economy) that makes whatever it is told to make -- there is nothing wrong with that idea in itself. The key point here is that AI systems will design everything for us, and therefore design will be free. (This assumption isn't stated explicitly until page 95, but it's there all along.) The Genie saves us the trouble of focusing our imagination and then expressing our vision in an articulate language, including physical language, that brings about the events we want. That's the assumption that is made throughout Engines of Creation. That's what makes everything else possible. The implicit assumption is that a robot could take over the roles of architect and contractor and still be a passive machine without imagination or will -- that's what a Genie is, or would be, if such a concept were logically coherent.

Either you have to program the robots, or you don't. If you do, then using them to build a skyscraper will not be free of labor costs -- far from it. On the other hand if you don't have to program them, then they have crossed the line that separates agents from automatons. They have crossed over to our side, and the work _they_ do amounts to the same thing as "human labor." Either the robots are automatons, in which case you have to program them, or they are agents, in which case you have to pay them. Skyscrapers will never be free, because we will always have to make an effort to focus our minds and make things happen -- or the robots will have to make the same effort to focus _their_ minds and make things happen.

Without Genies, the ability to make things out of atoms is just an extension of present-day technology, not dizzying at all. Molecular manufacturing without Genies is just agribusiness.

Without Genies, there will be no sudden "assembler breakthrough." Instead of emerging in a sudden breakthrough, nanotechnology will emerge continuously from present-day technology, over a period of decades, step by laborious step, each step involving an effort of concentration in a human mind.

There are no Genies and never will be. This is a logical point, not a technical point. It's not a question of what can or can't be done with atoms, or what can or can't be done with computers. I'm not saying AI will never exist. What I'm saying is that it doesn't matter -- any entity with at-least-human intelligence (artificial or not) won't work for free. To the extent that a robot makes independent decisions, it will have to be dealt with as an entity that makes independent decisions. A group of robots that could build a skyscraper by themselves would be indistinguishable from a contractor, and would have to be dealt with as such.

I'm not saying that nanosystems will not exist, nor that they will not be able to create large structures such as pipelines or skyscrapers. I'm saying that a nanosystem (or any system) capable of creating a pipeline or skyscraper will contain many human-level intelligences, and they will not be at your command... unless you pay them, or find some other way to motivate them.

This is an instance of the interface problem. Any agent capable of making human decisions will have an interface comparable to the interface humans have with each other. This is independent of the size of the agent, and also independent of whether the agent is artificial or not.