Re: Strong vs Weak SIs and the Incomprehensiblity Issue

Dennis Roberts (
Tue, 22 Sep 1998 13:56:01 -0400

Doug Bailey wrote:

> The central idea behind my post regarding the ultimate potential
> of SIs and HIs is to analyze the "unknowability" and
> "incomprehensibility" traits used by some to describe a post-
> Singularity environment. Using the "weak SI" and "strong SI"
> distinctions Nick mentioned, if the post-Singularity universe is
> populated by weak SIs then it will be comprehensible to us. If it
> is populated by strong SIs then it will be incomprehensible.
> The question of whether SIs will be "strong" or "weak" does not
> appear to be a question of SI capabilities but instead should be
> an inquiry into the nature of knowledge. Is there knowledge to be
> had that is forever beyond the grasp of human intelligence (even
> amplified HI)? Its a epistemological question which I doubt can
> be resolved easily. Such knowledge would be beyond our abilities
> of recognition, inference, deduction and modelling. We can not
> speculate on the feasibility of the existence of such knowledge
> since, by definition, we can not appreciate the validity of the
> knowledge. Why should we assume such knowledge exists except out
> of some ill-advised attempt at anthropic humility?
> If a SI is a strong/qualitative SI then it will be able to access
> these areas of knowledge that HIs can not. This type of situation
> parallels the turbo dog brain example Nick described. HI is
> qualitatively superior to >DI because of HI abilities of cognition,
> deduction, modelling, inference, etc. However, if we modified the
> >DI's software then it might achieve HI levels. If a SI is only
> a weak/quantitative SI then it might think faster and more
> efficiently, but it would not be able to access those areas of
> knowledge unknowable to HIs. The weak/quantitatively-superior SI
> would reach milestones faster but it would not reach more milestones
> overall.
> The Knowledge Question raises an interesting issue. Even if
> Strong SIs existed, how would we properly recognize them as Strong
> SIs? Since the knowledge they accessed that heralded their
> ascension to Strong SI status is beyond our ability to appreciate,
> how could we distinguish between a Strong SI and a Weak SI that
> has lost its marbles?
When faced with such a choice it appears
> that our only guidance becomes the faith we have in the SI. Either
> we say, "Gee, what the SI is doing now is completely beyond me.
> Since its a SI, it probably knows what its doing. Its a Strong SI"
> or we say, "This thing has gone nuts." Another problem is
> differentiating between knowledge that is forever beyond us and
> knowledge that is beyond us for now but will be understandable
> after another 200 years of scientific work.
> Doug Bailey

I think that the problem of determining the existance of a strong SI is a real one. The old line about, how do you explain what blue means to someone who is blind, seems pertainent. The main difference that I can see between SI's and between them and ourselves for that matter is one of the quality of perception, not the quantity of factual knowledge. A strong SI might start out being very ignorant of factual knowledge but due to its trancendentally superior perception would not stay in that state for long. A hare rather than a tortoise.

Dennis Roberts
 Boost your website with our start-up page!

"Do not be decieved by...some false secondary power, by which, in
 we create distinctions, then deem that our puny boundaries
 are things which we percieve, and not which we have made." ---