Re: Singularity: Generation gap

Eliezer S. Yudkowsky (sentience@pobox.com)
Fri, 26 Sep 1997 18:20:36 -0500

Eric Watt Forste wrote:
>
> It *is* useful to use notions of the Singularity as Vinge did and
> the Spike as Broderick did, when addressing a wider audience, the
> sort of people who try our patience at times. But these notions
> are not so useful on a list like this one. Most people here already
> know that things are getting weirder (and mostly better) and will
> continue to do so at an accelerating rate at least until the
> world has been changed beyond recognition.

I think of the Singularity as protection against Failures Of The Imagination.
Such failures can have *real* and *horrible* consequences. I have gotten
several letters from "Singularity skeptics" who think that enhanced
intelligence isn't all *that* powerful, and technology can't progress *that*
fast, and that it is therefore proper to IGNORE THE MOST ELEMENTARY
PRECAUTIONS when attempting to create a transhuman intelligence! One person
seriously proposed that you could just "pull the plug" if it started looking
dangerous! Another, on this list, said that we should just "keep it bottled
up" and only let it out "if it convinces us to". I'm sorry, but those
precautions would not suffice to keep *me* in, much less anything smarter.
Pull the plug? Hey, I'm already running on 200,000 computers around the
world. Only if it convinces us? We all know humans are impossible to fool.

If not for the "Singularity", I might not have realized that there are protein
synthesis machines which may be hooked up to the Internet... that locked-goal
Asimov circuits leak like sieves... and so on, and so on.

> > don't expect to be in charge, and it looks to me like the logic of
> > Libertarianism breaks down if you're omniscient For All Practical
> > Purposes.
>
> Sorry, I'm calling you on this one. "omniscient For All Practical
> Purposes" looks like a phrase with null semantic content to me.
> Could you please try rephrasing whatever it was you intended to
> say?

Omniscient FAPP means that you can visualize (and perfectly predict) all other
"agents" in a game-theoretic scenario. While multiple competing OFAPP agents
are questionable (halting problem?), a Power deciding whether to force-upload
humanity could well be OFAPP, game-theoretically speaking.

(Note: If external influences exist on a game scenario, OFAPP requires that
you be able to predict the external influences as well.)

-- 
         sentience@pobox.com      Eliezer S. Yudkowsky
          http://tezcat.com/~eliezer/singularity.html
           http://tezcat.com/~eliezer/algernon.html
Disclaimer:  Unless otherwise specified, I'm not telling you
everything I think I know.