Re: Singularity, Breaker of Dreams

den Otter (
Wed, 9 Sep 1998 12:09:10 +0200

> From: Eliezer S. Yudkowsky <>

> den Otter wrote:
> >
> > ----------
> > > From: Eliezer S. Yudkowsky <>
> >
> > > Sooner or later, some generation will face the choice
> > > between Singularity and extinction. Why push it off, even if we
> > > could? And besides, we might not die at all.
> >
> > But we can't rely on that, can we?
> Depends on what you mean by "rely". If you mean, "Can we assume that the
> probability is 90%?", the answer is "No". If you mean, "Can we behave as if
> the probability is 90%?", the answer is "Yes". Our world is dynamically
> unstable, and is being acted on by powerful forces and positive feedbacks
> which serve to destabilize it further. Under the circumstances, the only
> decision we can make effectively is whether we'll die in nuclear war or
> nanowar, or whether a Singularity will occur. These are the two stable
> states, and the Universe is filled with stable things.

The question is: how likely is it that there will be a world-wide disaster in the few extra years/months that it presumably takes to create SI from upgraded/uploaded humans (as opposed to creating SI from AI). This must then be weighed against the possibility of (super) AI terminating our existence. If you include the possibility of moving to space (surely quite feasible in a decade or two), then the scales tip towards waiting with SI until humans can be uplifted, IMO. Of course, this assumes that AI will be easier than uplifting, which is very likely but not completely certain.

> The forces inside a Singularity are powerful, complex, and far more dependent
> on external factors then the initial conditions. To the extent that initial
> conditions do have effect, they must use unstable forms of insanity to shield
> the AI from the external truth. In short, trying to control the Singularity
> would result in a world scoured bare and THEN Transcenscion. While I might
> find this outcome acceptable, I don't think anyone else would - and even from
> my perspective it's too dangerous; what if the Blight scours the Earth bare
> and then commits suicide?

So, if I understand you correctly, you would have peace with your own demise as long as your "brainchild", the Blight, lives? That's...traditional. Personally, I wouldn't be even satisfied with an uploaded copy of myself to continue "my" existence, let alone some completely alien form of life.

> I couldn't command the future even if I had the complete source code of a
> Macintosh-compatible seed AI in front of me right now. Choose between
> pre-existing possibilities, perhaps, but not add possibilities that weren't
> there before.

Of course, if you use AI to cause a Singularity you have effectively handed over control. Enhancing yourself (with chip implants, for example) to the point of SI (or at least significantly increased intelligence) would on the other hand give you a fair amount of control over things to come. No guarantees, but certainly a fighting chance.

> > The facts are simple:
> >
> > 1) at this time, only a handful of people really grasp the enormity
> > of the coming changes, and (almost certainly) most of them are
> > transhumanists (unfortunately, even in this select group many
> > can't/won't understand the consequences of a Singularity, but
> > that aside).
> Sounds a bit elitist to me.

The pot and the kettle? Yes, it's elitist in the sense that only a small group of people (will) see the Singularity coming, and even less will try to "surf its wave" (instead of trying to run away and getting washed over). On the other hand it isn't elitist at all since everyone is free to join this group of survivalists. I'm not exculding anyone, on the contrary: I've told hunderds of people (via the web) about the things to come, but (almost) no-one would listen.

> Are you sure *you're* one of the Chosen?

I have chosen myself, and so can you or anyone else. I know that Max for example has chosen himself too, so at least I'm in good company. ;-) At the moment it may mean little, but at least it's a beginning. I accept that the most probable outcome is failure (and death), but since there is nothing to lose, I might as well go for it (the same reason why I've signed up for suspension).

> Seriously, my perspective doesn't really allow for dividing humanity into
> groups like that.

I'm quite sure most of humanity will never know what hit them (some will still be in the stone age when it happens). Most people seem to even be in denial with regard to Y2K, which is *nothing* compared to the Singularity. A few people see it coming, some might even try to influence the event, but most won't do anything. Humanity is already devided. This is no personal preference, but a fact.

> Where thinking is concerned, you've got rocks, mortals, and
> Post-Singularity Entities. Sorting mortals by intelligence is as silly as
> separating rocks or PSEs.

As in nature, intelligence (in this case mainly foresight) will be the great discriminator. The "best adapted" mortals will become their own successors, the PSEs. The fate of the rest is very uncertain.

> > 2) This gives us a *huge* edge, further increased by the high
> > concentration of scientific/technological talent in the >H community.
> No, it doesn't. We ain't got no money. We ain't got no power.

Correct. A major flaw, which deserves more attention than any other. Money=power, money=life. It is time everybody woke up and realized that without money, all our future dreams are just hot air.

>_Zyvex_ might

> be said to have an edge because it's doing work in nanotechnology. The MIT
> labs might be said to have an edge. If you're really generous, I could be
> said to have an edge because of "Coding A Transhuman AI" or "Algernon's Law".
> What I'm trying to convey is that each individual has ver own "edge". Not
> only that, but I think that if all the Extropians worked together it would
> simply slow things down.

Individual edges are small, and useless if you don't see the big picture. To successfully create SI *from humans*, broad co-operation is needed. Besides, I don't trust Zyvex or MIT *with my life*, and neither should you. I want to be there when SI is born, nothing less. Co-operation, a "Singularity Club", would increase the personal chances of survival of all people involved. It is the rational thing to do for the egoist and altruist alike.

> > 3) If we start preparing now, by keeping a close eye on the
> > the development of technologies that could play a major part
> > in the Singularity (nanotech, AI, implants, intelligence
> > augmentation, human-machine interfaces in general etc.)
> > and by aquiring wealth (by any effective means) to set up a
> > SI research facility, then we have a real chance of success.
> Go ahead. Don't let me stop you.

If I could do it by all myself, I wouldn't be wasting my time on this list, would I? ;-)

> > Note: the goal should (obviously) be creating SI by "uplifting"
> > (gradually uploading and augmenting) humans, *not* from AIs.
> Too damn slow. Turnaround time on neurological enhancement is probably at
> least a decade and probably more - for real effectiveness, you have to start
> in infancy. I can't rely on the world surviving that long.

> Also, I trust humans even less than I trust AIs.

At least you know this devil (humans). The only (rational) reason why we would want to have the Singularity asap is our innate mortality. If science can find a way around that without the help of SI, then there is much less of a rush. You can move to space to keep safe from WW3 and like mishaps, and work on the slower but more reliable SI-from-humans-approach. Everyone keeps a close eye on the others, and when the technology is ready, everyone uploads and gets augmented at the same time. Then...all bets are off.

> Also, the first-stage neurological transhumans will just create AIs, since
> it's easier to design a new mind than untangle an evolved one.

The main question should not be "what's easier", but "what's safer". You may trust AI more than humans (I can get into that), but do you trust AI more than *yourself*?