Re: >H: The next 100 months

Eliezer S. Yudkowsky (sentience@pobox.com)
Thu, 21 Oct 1999 16:00:29 -0500

Xiaoguang Li wrote:
>
> here's my understanding of Eliezer's argument for actively
> pursuing the creation of a power:
>
> destruction is easier than construction: given humanity's
> lackluster record with past technological advances, nanowar (read:
> armagedon) is virtually certain.
> evolution is messy: even if humanity survives the advent of
> nanotech and gives rise to transhuman powers, these powers would have no
> more chance of being benevolent than powers designed de novo. the
> dichotomy -- if transhuman powers retained evolutionary baggage such as
> emotional attachments or moral inhibitions, then they are prone to
> inconsistencies and self-contraditions and are therefore untrustworthy; if
> they did not, then initial conditions are insignificant, and there's no
> functional difference between transhuman powers or powers who arise
> otherwise. the caveat -- AI is easier than IA, so this scenario would
> require active suppression of technology, the ills of which are
> well-known.
> power before nanowar: creation of a power designed de novo before
> nanowar will result in a singularity -- that is, all bets are off (read:
> no more house advantage against the survival of humanity).

So far, so good.

> now for my doubts. does the creation of a power really increase
> the chances of our survival? it seems that the odds of humanity
> surviving a singularity are significantly lower than a coin flip. given
> the enormous difference between the creation and its creator, it seems
> most likely (> 99%) that humanity would not be significant to a power.
> that is, perhaps less significant than virion particles to a human
> being.

I honestly don't know. I really don't. I certainly wouldn't put the probability at lower than 10%, or higher than 60%... but, ya know, I could be missing something either way. I think we're missing a fundamental part of the question. There isn't *any* outcome I can see for the Singularity, not one of your three possibilities, that could conceivably have happened a few hundred thousand times previously in the Universe... which is what I would regard as the absolute minimum number of intelligent species to expect, given that some galaxies are already so old as to be burned out.

What I am fairly certain of, and what my intuitions agree with, is this:

  1. If benevolent Powers can be created, it will be because there are no forces or tendencies, internal or external, that would interfere with their creation - not because we mounted a heroic effort and overcame the resistance. Humans and AIs are part of the continuum of "mortal minds", they are both made of fundamentally the same stuff, and there isn't any magical reason why humanborn Powers can be benevolent and designed Powers can't.
  2. We can't come up with an evidence-consistent account of what happens after the Singularity, or what Powers do, because we're not taking some basic factor into account. The twentieth century just isn't late enough for our worldview to be capable of phrasing the question properly.
  3. There is a massive momentum, backed by profit-motive, idealism, and all the forces of generalized technological progress, towards creating and using ultratechnologies like nanotech and AI. Neither can be suppressed indefinitely, because both will just keep getting easier and easier, until eventually one person can do it. We can't slow down. We can't pick and choose. We can only steer for one direction or the other. Something huge is going to happen on this planet; it's just a question of what.
  4. Uploading is extremely advanced nanotechnology *and* cognitive science. Any technology capable of uploading someone has been capable of frying the planet *or* building an AI for at least five years of Internet time.
  5. This *will* be settled, one way or another, fifty thousand years from now, no matter what happens in the meantime. One generation has to be the last, so it would be irresponsible to refuse to deal with it. -- sentience@pobox.com Eliezer S. Yudkowsky http://pobox.com/~sentience/tmol-faq/meaningoflife.html Running on BeOS Typing in Dvorak Programming with Patterns Voting for Libertarians Heading for Singularity There Is A Better Way