Robin Hanson wrote:
>
> Asking Drexler has some obvious problems with bias. Better to evaluate the
> arguments he presents.
I did. As far as I can tell, they're OK. But I don't delude myself that I can understand them simply because I'm an extremely bright person with a grasp of basic physics and chemistry and because Drexler is a good explainer. There is a difference between being an amateur at something, no matter how well informed, and spending years at it. It's what you think of as My Specialty that you wind up with powerful intuitions in. I can sort of understand why quantum physics is ugly and General Relativity is beautiful, not because my father was a physicist, but because there are basic principles that relate to the nature of causality. But I don't think I understand in the same way Einstein did, and I don't try to deal with the problem in the way Penrose does. I have no intuition telling me that Drexler is right; I read his book and he doesn't sound like a crank, he didn't say anything I know is false, so I have no choice but to trust his conclusions - to default to Drexler on the matter of nanotechnology.
> >In short, I think there are big wins because I've looked a little beyond the
> >limits of my own mind ... these are ultimately the only reasons for believing
> >in a Horizon or a Singularity, and neither can be argued except with someone
> >who understands the technology. ...
> >Anyway, the primary point that I learned from the Singularity Colloquium is
> >that neither the skeptics nor the Singularitarians are capable of
> >communicating with people outside their professions. (Or rather, I should say
> >that two people in different professions with strong opinions can't change
> >each other's minds; ...
>
> Please don't attribute any disagreement to my failing to understand AI.
> I think you will find that I can match any credentials you have in AI (or
> physics or nanotech for that matter).
Oh, I'm sure you understand AI! Enough to come up original ideas, for that matter. Still, you've got more powerful intuitions in social science. I daresay that you may even be leveraging your understanding of AI with your understanding of psychology, social agent interactions, or just the intuitions you picked up in economics. Not that there's anything wrong with this, you understand.
My understanding of economics is based on facts about system overhead I learned while trying to program causal propagations. The idea of causal insulation and preventing information loss that lies at the heart of complex barter - I picked that up from a module I wrote in deep C++ over the course of a month. I'm going to have different intuitions about economics, and you're going to have different intuitions about AI, so we are inevitably going to disagree if we're discussing something as factually unknown as superintelligence and the future. It doesn't matter whether we both possess invent-level intelligence in the field, because our specialties are different.
> >> The question is *how fast* a nanotech enabled civilization would turn the
You can have a lot of small analogies hold, to things like the laws of physics
and the processing of information in the brain, rather than great big
analogies to entire civilizations. At that point, you're working with rules
instead of correspondences - with simulations instead of analogies.
> >> planet into a computer. You have to make an argument about *rates* of change,
> >> not about eventual consequences.
> >
> >If they can, if they have the will and the technology, why on Earth would they
> >go slowly just to obey some equation derived from agriculture?
>
> Economists know about far more than agriculture. And you really need
> a stronger argument than "I'll be fast because it's not agriculture."
>
> >I've tried to articulate why intelligence is power. It's your turn. What are
> >the limits? And don't tell me that the burden of proof is on me; it's just
> >your profession speaking. From my perspective, the burden of proof is on you
> >to prove that analogies hold between intelligence and superintelligence; the
> >default assumption, for me, is that no analogies hold - the null hypothesis.
>
> If no analogies hold, then you have no basis for saying anything about it.
> You can't say it will be fast, slow, purple, sour, or anything else.
> To me "superintelligent" means "more intelligent", and we have lots of
> experience with relative intelligence. Humans get smarter as they live longer,
> and as they learn more in school. Average intelligence levels have been
> increasing dramatically over the last century. Humanity as a whole is smarter
> in our ability to produce things, and in scientific progress. Companies get
> smarter as they adapt to product niches and improve their products. AI
> programs get smarter as individual researchers work on them, and the field
> gets smarter with new resesarch ideas. Biological creatures get smarter
> as they develop adaptive innovations, and as they become better able to
> adapt to changing environments.
>
> All this seems relevant to estimating what makes things get more intelligent,
> and what added intelligence brings.
The Big Bang was instantaneous. Supernovas are very bright. State-vector reduction is sudden and discontinuous. A computer crashing loses all the memory at once. The thing is, human life can't survive in any of these areas, nor in a Singularity, so our intuitions don't deal with them. The Universe is full of gigantic, tremendously powerful phenomena, but for some reason we don't seem to get close to them. If the planet was on a collision course with the Sun, I can see people saying: "We've seen ice melt, and the deserts are full of sand, and when things get extremely hot water evaporates, but there's no reason to suppose that the entire planet will vaporize. After all, the Earth's core is pretty hot and nothing happens except for a few volcanoes." Why doesn't our intuition say that things get heavier when they go faster? The extreme cases are scattered all over the Universe! But they don't support mortal life, so they don't appear in comforting analogies.
The phenomena I'm talking about would have destroyed life as we know it if they had appeared in any point in the past. How, exactly, am I supposed to find an analogy? It's like trying to find an analogy for Earth falling into the Sun! No, economic production doesn't go to zero no matter how bad the catastrophe, but that's all at the local scale - and besides, if the economic production had gone to zero, we wouldn't be discussing it. The same applies for economic production going to infinity. Different laws apply at other scales, whether it's microscopic interactions, velocities close to the speed of light, or superintelligence.
--
sentience@pobox.com Eliezer S. Yudkowsky
http://pobox.com/~sentience/AI_design.temp.html
http://pobox.com/~sentience/sing_analysis.html
Disclaimer: Unless otherwise specified, I'm not telling you
everything I think I know.