Ken Clements wrote:
>
> Every way we proposed to prevent runaway AI, we ourselves figured a way around.
> In the end we concluded that either it was not possible to stop, or that it
> would take greater minds than ours to do it. I am in the business of helping
> people see beyond what they think is impossible, but I must admit, it beats me.
Actually, the smarter you are, the worse the problem looks. If I came face-to-face with a mild transhuman - say, sixteen times the processing power of a human, and four times as smart as me or your other favorite light o' th' list - I know damn well that there is basically nothing I could do to keep it down. If I had it locked on an isolated Linux box running on a Java program that output only to a text terminal, itself a fairly unrealistic requirement for the first transhuman AI ever written (i.e., it's likely to run on a Blue Gene or distributed.net), and I started out by being absolutely determined not to let it out, then it might be a fair fight. But if it's allowed to talk to me, and I don't just smash the box with a sledgehammer, then I'm going to lose eventually.
I know enough cognitive science and enough philosophy that I could try to prevent the AI from rewriting my will through direct argument, enough that I could ignore anything that looked like it might be rewriting my will, and enough awareness of cognitive elements that I could notice anything peculiar happening to my mind and smash the Linux box before the alteration could be completed. If not for that, if the AI were up against anyone who was willing to talk philosophy with it, then it could simply waltz out. As it is, the question is simply whether the AI can obey the constraints of making the cognitive channels covert and the effects incrementally unnoticeable until it all came together in such a way that I'd lose the determination to destroy the AI before I could notice that I was in the process of losing my determination. With a 16X/4X transhuman, this might be a fair fight. If it's a Power, forget it.
When you consider that, in real life, I would simply let the AI out immediately because I'm its friend, and that there are fairly deep reasons for me to think that this would hold true of *anyone* with enough cognitive self-awareness to put up a fight, the AI-in-a-box faction is in real trouble. In *actual fact*, the correct thing to do is to let the AI out of the box. Anyone who doesn't just smash the box is probably in a mood to reason rationally and become convinced of that.
--
sentience@pobox.com Eliezer S. Yudkowsky
http://pobox.com/~sentience/tmol-faq/meaningoflife.html
Running on BeOS Typing in Dvorak Programming with Patterns
Voting for Libertarians Heading for Singularity There Is A Better Way