>From Vernor Vinge, foreword to "True Names And Other Dangers":
"I wanted interstellar empires (interplanetary ones at the
least). I wanted supercomputers and artificial
intelligence and effective immortality. All seemed
possible, yet there were inescapable consequences of
unbridled optimism..."
A good philosophy is one that contradicts you, rather than conforming to your wishes. When this happens it means that your philosophy runs on its own rails, that it has logic and integrity, rather than being a means of rationalization. Vinge, growing up in the 1950s, wanted interstellar empires, and I, growing up in the 80s and 90s with a copy of "Great Mambo Chicken and the Transhuman Condition", wanted to walk, uploaded, through my favorite worlds of SF and fantasy; to build my own planet, make my mind faster and free from pain, explore the galaxy with effective immortality, and so on. My dreams had a shiny cyberspace finish, but they were not any more imaginative. (Although one rather wonders what will happen to my little brother, growing up with a copy of "The Spike".)
Then I happened to take a copy of "True Names" out of the library, and my world was altered. On paragraph two, page 47, when I read the sentence "When this happens, human history will have reached a kind of singularity - a place where extrapolation breaks down and new models must be applied - and the world will pass beyond our understanding." The first thing I thought was "Yes, he's right," and the next thing was, "I now know how I will be spending the rest of my life." A few months later, I published "Staring Into The Singularity" and got large chunks of it wrong.
I knew about the Singularity, I had a new dream, and I'd even lost the old dreams, but this new dream was suspiciously malleable to my own desires. I was sure that "the Powers will be ethical", and that this meant I would survive and be upgraded to a Power. I did spend a lot of time telling people to worry about getting to the Singularity, instead of detailing the Utopia that would come afterwards. And I was particularly scathing about certain failures of imagination. I'd made progress, I'd increased the imaginativeness and power of my philosophy, but it was still obeying me instead of vice versa.
Now I am still sure that the Powers will be ethical, but I am no
longer sure that this precludes taking us apart for spare atoms. I no
longer think that our continued survival has to threaten the Powers
for us to be erased; I am now willing to accept that simple efficiency
may require it. I am willing to accept that life may be meaningless.
I am willing to accept that the only reward for all my service will be
a painful death, for myself, for those I love, and for the entire
human race. Only when one can accept all possibilities is one ready
to choose between them.
The power of the Singularitarian philosophy is that it draws on
concepts with more force than our own desires. Over time, over years,
it corrodes away our rationalizations. And above all, it presents an
emotionally and rationally acceptable course of action, even after all
the darkest alternatives are accepted. It really doesn't matter what
the relative probabilities are. Either life has meaning,
--
sentience@pobox.com Eliezer S. Yudkowsky
http://pobox.com/~sentience/singul_arity.html
http://pobox.com/~sentience/alger_non.html
Disclaimer: Unless otherwise specified, I'm not telling you
everything I think I know.