<< Unless the motives of an SI are subject to arbitrary configuration or
insanity, which would theoretically allow a sufficiently stupid person to create an eternal hell-world, then I am sure beyond the point where caveats are necessary. In practice, I'm sure enough that you can rip my arm off if I'm wrong. (David Gerrold's Criterion.)>>
Well, that sounds pretty damn sure<g>.
<<I'm not going to second-guess the SI. I believe that the SI will do
whatever I would do if I was only intelligent enough to see the necessity. I don't know what the motives of SI may be, but I identify with SIs, as a child might identify with vis adult self.>>
I think there are three things that an SI would need to be able to make good decisions (from the human perspective): Intelligence, Empathy, and Sympathy.
Sympathy is number one - the SI needs to wish to act in our best interests. Empathy is number two - the SI needs to understand our perspective before it could know what is in our best interests. Intelligence is number three - the SI would need to be able to come up with better methods to act in our best interests (and maybe even determine our best interests) than we could on our own.
I find it interesting that you see yourself as a child to the SI's adult. If this were the true situation, then why wouldn't the SI want to see you grow up? assuming the SI can do some of the things you suggest, why not simply show you how to gain SI level intelligence yourself?
<< Maybe. There are limits to how much time I'm willing to spend worrying
about the ways twentieth-century American culture can misinterpret me.>>
I worry this is a maladaptive response. It would be a pity if society sought to stop you not for what you were doing, but only what they thought you were doing. Misinterpretations have gotten people killed, and I don't think we've gotten beyond that level quite yet.
<<Again, I think that once again we have the basic divide between "life
goes on" and "walking into the unknowable". I am not advocating a social policy. I am not advocating anything human, and the whole concept of "social policy" is a very human thing.>>
True, but it will also be a very SI thing if they choose to interact with us.
<<Yes, but there wasn't even the theoretical possibility, at any point, of rule by anyone other than Cro-Magnons. And Cro-Magnons are, universally, evolved to contain special-purpose cognitive rules for reasoning about politics, with little hidden traps that cause every person to believe ve is the superior philosopher-king and fail to be one. It's all entirely explicable in terms of evolutionary psychology. And within that Integrated Causal Model, within that greater model of the Universe that contains and explains fifty thousand years of failure, we have every reason to believe that AIs will be on the outside.
Unless the initial programming makes a difference, in which case all bets are off.>>
I'm guessing that initial programming does make a difference.
<< I can, but never mind. Why is perfection necessary? I'll take 99.99%
accuracy any day.>>
You're right, I would take 99.99% any day. But I'd want some recourse should the 0.01% errors go against me. Also, when the SI has that much power, how do we know that it IS that accurate? It might just be altering our perceptions and memories to make us think it is right that often (maybe for "benevolent" reasons - like it doesn't want us to be emotionally upset about how much it is messing up).
<<Fine. Let's say the SI is programmed to simulate an enlightened, honest, fair, kindhearted, and absolutely informed agora. Wanna bet it couldn't beat the pants off of the modern American government?>>
Okay, you've got something there. It probably would. Of course, whose definitions of enlightened, fair, and kindhearted are we using?
<< Why should errors accumulate instead of cancelling out?>>
I would say both might happen, cancellation and accumulation. Of course, as long as it has a rigorous enough error check, and the error check isn't one of the function where the errors are accumulating, then the SI should be able to make corrections.
<<Once an error becomes large enough for humans to perceive, it is large
enough for any SI remotely worthy of the phrase to notice and correct.>>
Agreed. Unless it is something that is only obvious from our perspective.
<<I mean, why doesn't this whole argument prove that thermostats will
never match a free democracy that votes whether or not to turn the heater on?>>
Depends on what criteria are important for the decision whether to turn the heater on or off. Thermostats, however, make no initial decision as to what the temperature should be; so far that is still a human function. Once that decision is made, then a thermostat would do it better than a committee.
<<Even supposing this to be true, an SI could easily attain a high enough
resolution to prevent any humanly noticeable unhappiness from developing. I mean, let's say you upload the whole human race. Now it runs on the mass of a basketball. If the SI can use the rest of the Earth-mass to think about that one basketball, I guarantee you it can attain a perceptual granularity a lot better than that of the humans running on it.>>
But would this be the right thing to do? If you can upload humanity, then can't you also alter humanity so as to have them be able to utilize all that Earth mass for their own cognitive use? If the human race were uploaded and an SI kept them simple enough for it to perfectly comprehend, then I would not consider this in the best interests of humanity. The best interests of humanity would be best served in promoting our own growth.
One variant on this idea which might be considered benevolent would be if this were a dynamic instead of static process. Let's say the SI is one that is constantly improving itself caught up in a Singularity type scenario, where it is constantly growing and improving itself. Let us further say that the process doesn't allow much in terms of leaping ahead so that you more or less need to go through the previous levels of SI to get to the more advanced ones (perhaps at least with your personality intact). It would be immoral to ask the SI to stunt its own growth just so we could catch up, but as long as it gave us what aid it could in our own growth, then it would be morally acceptable for it to continue to retain its lead in ability, and also wise of the up-and-coming former humanity to accept the SI's leadership, as it is in a unique position to know better what to do in most situations. Under these circumstances, I would have very little problem following the lead of the SI (though I would still not take its rightness for granted).
<<Complete control over the processors running humanity isn't enough power for you?>>
No, to really entrust ourselves to an SI you would also want it to have complete control over the environment in which those processors and its own operate. Shit still happens, even to SI's.
<<What complexity? Just because it looks complex to you doesn't mean that
it's complex to the SI. I occasionally get flashes of the level on which human minds are made up of understandable parts, and believe me, society is a lot less complex than that. Less communications bandwidth between the elements.>>
I agree, the human mind is still more complex than human society in general. But remember, human society is made up of all those complex human minds.
<<Is this a true argument?
Would it be obvious to a sufficiently intelligent being? Then what will happen, in your visualization, is that I create an SI, and it adopts Greg Burch's personal philosophy and does exactly what you want it to for exactly the reasons you cited.>>
Then wouldn't the SI really be an extension of Greg Burch's personality? If the SI doesn't diverge from the opinions, goals, and desires of Greg Burch, then how is it separate from Greg Burch? Furthermore, if it can't come to a different conclusion as to what should be done, won't this handicap it in providing a better world (for though I greatly respect Greg's judgement, in my experience, no living human is ever completely right all the time. Though things could be a lot worse than having the Greg Burch-SI running the show<g>)? Why not just upgrade Greg Burch to SI level (and hey, what about me too while your at it<g>)?