Re: Humanrintelligences' motivation (Was: Superintelligences' motivation)

Anders Sandberg (
Thu, 30 Jan 1997 22:49:00 +0100 (MET)

On Thu, 30 Jan 1997, The Low Willow wrote:

> On Jan 30, 2:40pm, Anders Sandberg wrote:
> } and you will end up with "Who watches the watchmen?". Another possibility
> } would be an automated "immune system" against nanotech, but even if it
> } would work well, there are always other dangers.
> } Most likely the only stable way of preventing widespread disaster is
> } dispersion.
> For defense against bio and nano immune systems probably are the best
> way to go.

I agree, but the main worry I have is that there are most likely other
ways of causing disaster ("cavitronics" a la Brin creating quantum black
holes, vacuum decay generators, invocations of Cthulhu). One idea
proposed in Brin's _Earth_ is that the silence in the sky is due to the
extreme simplicity for an advanced civilization to destroy itself; I
don't want to believe it, but he has a point.

> Best would be mutable systems that could be changed to small
> or large degrees. It would be unlikely that anyone could be certain
> they were protected against everything, but you wouldn't worry about
> plagues, because there woouldn't be a common system to infect.

Yes, the immune system would need to evolve to keep up with new threats
and changing conditions. The goal should not be 100% security (that
doesn't exist), but that all problems remain bounded and preferably small.

Someone challenged the readers of sci.nanotech to come up with designs
for a nanotech immune system so that he could try to find holes in them
to exploit; I think it would be a great idea if we could help this
important work.

I think a fruitful approach would be to look at biology and compare the
similarities and differences between the bio and nano problems, and then
try to exploit what we can learn from evolution.

> Security in the High Beyond is largely cryptology and
> immunology, especially combined.

If you look at them as finding entities that try to exploit your system
and then removing their access to it, then they become identical.

> I don't have any bright ideas for defense against Really Big Explosions.
> (Nukes, asteroids, solar flares, etc.)

Dispersion and prevention is the only way to deal with this; for every
defense you can come up with, I can come up with a bigger explosion.

Anders Sandberg Towards Ascension!
GCS/M/S/O d++ -p+ c++++ !l u+ e++ m++ s+/+ n--- h+/* f+ g+ w++ t+ r+ !y