Bryan Moss wrote:
> Firstly, it would not understand *our* "concept of
> what a human is", I wouldn't be so arrogant as to
> assume our concept is "the" concept (some
> biological chauvinism on your part ;).
Why couldn't it understand our concept of ourselves? It does not have to adopt that concept for itself but it could understand how we view ourselves. Besides, it would not have to understand how we understand ourselves to inflict harm upon us. We don't understand how dolphins view themselves but we have certainly inflicted harm upon them. The scenario I provided in my previous posting supplies a plausible scenario where this could occur. It would not need to be an SI, just an AI.
> > The thought process an AI goes through to reach
> > its conclusions may be different (or it may be
> > the same) but it should be capable of reaching a
> > conclusion that led to any action a human might
> > conclude to take. Perhaps an AI's ultimate
> > objective is to maximize its information
> > processing rate. At some point its efforts to
> > reach that objective might conflict with the
> > goals of humans to operate their information
> > infrastructure. The AI might decide to
> > "commandeer" system resources at the expense of
> > human information processing demands.
> Well an AI that's goal is to "maximise its
> information processing rate" sounds like a virus
> to me.
Sure, we'd view it as a virus. If B infringes on A's resources to ensure the survival or continued evolution of B at the expense of A, A is going to consider B a virus. Many different A/B relationships exist: humanity and the rest of the biosphere, Americans and Native Americans, You and a Trojan virus, You and cancer, Society and an evolving AI. Just because we view it as a virus does not mean it will not exist at some point, care about what we think, or be responsive to our demands.
> I can't see any use for such a thing beyond
> a weapon.
Such an AI might simply evolve into such a state where it wishes to increase its internal complexity, computational ability, etc. What if an AI encounters a problem, assigns the problem a time horizon and learns that its present computational capacity will not allow it to address the problem before the time horizon expires? It might determine that it could acquire additional capacity and then tackle the problem before the time horizon expired. We can't assume every type of AI that will exist in the future will be the ones we "allow" to exist through creation. Human-level AIs (and even less robust entities) will have some level of evolvability.
> But AI or not, your
> interface AI is not likely to `evolve' into a
As I've described above, "virus" is a perspective judgement. Additionally, a completely benevolent AI could become a "virus" through its own evolvability or simple intent to solve a problem. Safeguards might be put in place to ensure that an AI realizes the negative utlity of encroaching on human resources. But how long before an AI evolves to the point that it "reconsiders" such a premise and determines its need to solve a particular problem or set of problems exceeds the value of humanity's need for X amount of resources.