Re: Goo prophylaxis

Nicholas Bostrom (
Fri, 29 Aug 1997 01:45:53 +0000

Carl Feynman wrote:

> >Isn't the
> >design work fairly tractable (Drexler has already produced some nice
> >designs) and it is mainly the lack of molecular tools that prevent us
> >from starting building things? Better CAM would help a lot, and it is
> >on its way.

> (1) The minimum self-reproducing device (mycoplasma genitalium) seems to
> require about a million bits of information. I don't think we'll be able to
> get much smaller than that with our artificial equivalents. That's about as
> much information as is embodied in a car or medium-size piece of software.
> To develop from having no experience in automotive technology whatever to
> the point where you can build a reasonably effective car took at least
> thousands of genius-years. Ditto software. The first piece of software
> that I'm aware of that was over a million bits long was OS/360, developed in
> 1964, at a cost of 5000 man-years, to say nothing of all the research that
> it took to bring software technology to the point where they could even
> start the project.

There is a difference between the three systems you mention, on the
one hand, and nano self-replicator on the other. Mycoplasma
genitalium, automotive vehicles and comersial software are all
required to be fairly optimized. To build an optimised nano
self-reproducing device would be much harder than simply to make
something useful that can replicate. For example, a universal Turing
machine has been constructed in Conway's Life world. The entity is
very big and it was hard, but nothing near a thousands of genius-year
task, to do it. The feasibility stems from the fact that you have
identical components that you can put together into bigger identical
components, and so on, and at each step you need only consider the
apparatus at a certain level of abstraction. If this is the right
analogy for nanotech, then the design work would seem tractable, once
the right tools are there. But I will take your opinion on this
issue into account in my future thinking. And debugging is also a

> (3) Drexler and Merkle are two very (very!) smart guys. They have labored
> for years, and designed what? Some bearings, a transmission, a PLA, and a
> Stewart platform? And you claim this shows how easy it is? I'd hate to see
> an engineering task you considered hard! If we keep working at this rate,
> we should see a complete assembler in around a millenium.

Those are again optimized designs. What about the rod logic
computer? And I didn't Drexler give it as an exercise to his students
to design an autoreplicator when he taught a course at some
university? I am not sure about how detailed these designs are,
though. (Could anybody tell us this, please?)

> > As you point out, each partial achievment would bring great benefits
> >to the power that makes it, so wouldn't this mean that it would have
> >a good chance of pushing further ahead, leaving the competition
> >behind?
> This assumes that it is possible to acquire the massive resources needed to
> develop the next generation without letting anyone know how the current
> generation works, or even that it works. I would suggest that this is
> impossible, since the resources have to be gathered either by selling
> products or extortion by threats of violence. Both of these are highly
> visible activities.

If the early generations of nanotech can be used to build things,
those things can be sold without giving away details about how they
are made. Or if the early generations can be used to build better
computers, then the military might benefit from that while keeping
the project secret.

>As soon as the world realizes that there is massive
> power to be had, everyone will work like crazy to catch up.

But the leading power will work like crazy to keep the lead. If they
all work equally hard, the one that starts out with an advantage
should get to the goal first. The main point we are discussing is
weather the other powers would by then have obtained enough nanotech
to effectively defend themselves against the leading power. I think
the major military advantages could differ dramatically between some
of the pairs of adjacent generations, so that the first power to
develop the later version would have an easy match against the power
who has the earlier version. This means that even if the whole road
to advanced self-replicators is long and slow, there would still be
some point where a slight progression yielded huge military payoff

> >If superinelligence (that could perform a thousand genius years in a
> >short time) comes before nanotech, or is developed at an early stage
> >of nanotech, then the bottle neck would almost certainly be the
> >hardware, the molecular tools, and in such a case the maturation
> >process would be almost instantaneous.
> I entirely agree.

How likely do you consider that scenario?

> >(2), given slow
> >maturation, will that mean that a multipolar world order can remain
> >stable?
> >
> >(2)
> >I believe the answer to the second question is No. I think there
> >would either be a negotiated merger, or the stronger power would
> >obliterate the weaker, and then immediately rebuild itself, and then
> >expand spherically at a good fraction of the light speed.
> There is at least one other possibility, and it is the one that keeps the
> world system stable: the cost of subduing the smaller power is more than the
> profit obtained from its submission. Please explain why an imbalance in
> nanotech is more likely to make this solution infeasible than does an
> imbalance in e.g. nuclear missiles, or iron swords. Nanotech makes the cost
> of conquest smaller, but also reduces the profit of conquest, since
> everything gets so much cheaper with nanotech. What are you going to get
> from your conquered enemies? Natuaral resources? Go to the asteroid belt!
> Lebensraum? Well, that's OK if you like scorched earth. I'd rather have my
> own O'Niell colony.

Forget about the earth, what is at stake is half the universe (the
chance to double the amount of value-structures you can create) and
the possibility of getting rid of an enemy that might decide to
destroy you at a later time. (Even more than this if there are more
than one rival power aspiring to nanotech.)

> I'm being sarcastic here, perhaps more than I should be, but I genuinely
> don't see the payoff to instantaneous extermination of foriegners through
> nanotech, espescially when your enemies will have at least nuclear weapons
> and at best nanotech only one generation behind your own.

Not all foriegners would need to be exterminated, only those that
refused a negotiated merger. But if genuinely hostile powers were not
extreminated (or disarmed) at this point, it would only be a matter
of time when they would be in a position where they could
exterminate us.

Whether nukes are a complication depends on how advanced nanotech we
are talking about. I agree that they would seem to retain some
deterrence at the early stages.

The outcome also depends on whether a superinelligence is in
controlled or whether the powers are ruled democratically by ordinary
people. In the latter case we would have a further complication.

Subduing the enemies might be more acceptable to the masses if it
could be done without bloodshed and harm to the other nation's
population. Advanced nanotech would make this possible.

What we are discussing lies at the very heart of the future. Maybe it
has never been done so incisively, anywhere, as we are now doing it,
on this thread.

Nicholas Bostrom