Strong vs Weak SIs and the Incomprehensiblity Issue

Doug Bailey (
Thu, 17 Sep 1998 10:25:19 -0400

The central idea behind my post regarding the ultimate potential of SIs and HIs is to analyze the "unknowability" and "incomprehensibility" traits used by some to describe a post- environment. Using the "weak SI" and "strong SI" distinctions Nick mentioned, if the post-Singularity universe is populated by weak SIs then it will be comprehensible to us. If it is populated by strong SIs then it will be incomprehensible.

The question of whether SIs will be "strong" or "weak" does not appear to be a question of SI capabilities but instead should be an inquiry into the nature of knowledge. Is there knowledge to be had that is forever beyond the grasp of human intelligence (even amplified HI)? Its a epistemological question which I doubt can be resolved easily. Such knowledge would be beyond our abilities of recognition, inference, deduction and modelling. We can not speculate on the feasibility of the existence of such knowledge since, by definition, we can not appreciate the validity of the knowledge. Why should we assume such knowledge exists except out of some ill-advised attempt at anthropic humility?

If a SI is a strong/qualitative SI then it will be able to access these areas of knowledge that HIs can not. This type of situation parallels the turbo dog brain example Nick described. HI is qualitatively superior to >DI because of HI abilities of cognition, deduction, modelling, inference, etc. However, if we modified the
>DI's software then it might achieve HI levels. If a SI is only
a weak/quantitative SI then it might think faster and more efficiently, but it would not be able to access those areas of knowledge unknowable to HIs. The weak/quantitatively-superior SI would reach milestones faster but it would not reach more milestones overall.

The Knowledge Question raises an interesting issue. Even if Strong SIs existed, how would we properly recognize them as Strong SIs? Since the knowledge they accessed that heralded their ascension to Strong SI status is beyond our ability to appreciate, how could we distinguish between a Strong SI and a Weak SI that has lost its marbles? When faced with such a choice it appears that our only guidance becomes the faith we have in the SI. Either we say, "Gee, what the SI is doing now is completely beyond me. Since its a SI, it probably knows what its doing. Its a Strong SI" or we say, "This thing has gone nuts." Another problem is differentiating between knowledge that is forever beyond us and knowledge that is beyond us for now but will be understandable after another 200 years of scientific work.

Doug Bailey