The central idea behind my post regarding the ultimate potential
of SIs and HIs is to analyze the "unknowability" and
"incomprehensibility" traits used by some to describe a post-
environment. Using the "weak SI" and "strong SI"
distinctions Nick mentioned, if the post-Singularity universe is
populated by weak SIs then it will be comprehensible to us. If it
is populated by strong SIs then it will be incomprehensible.
The question of whether SIs will be "strong" or "weak" does not
appear to be a question of SI capabilities but instead should be
an inquiry into the nature of knowledge. Is there knowledge to be
had that is forever beyond the grasp of human intelligence (even
amplified HI)? Its a epistemological question which I doubt can
be resolved easily. Such knowledge would be beyond our abilities
of recognition, inference, deduction and modelling. We can not
speculate on the feasibility of the existence of such
If a SI is a strong/qualitative SI then it will be able to access
these areas of knowledge that HIs can not. This type of situation
parallels the turbo dog brain example Nick described. HI is
qualitatively superior to >DI because of HI abilities of cognition,
deduction, modelling, inference, etc. However, if we modified the
>DI's software then it might achieve HI levels. If a SI is only
a weak/quantitative SI then it might think faster and more
efficiently, but it would not be able to access those areas of
knowledge unknowable to HIs. The weak/quantitatively-superior SI
would reach milestones faster but it would not reach more milestones
overall.
The Knowledge Question raises an interesting issue. Even if Strong SIs existed, how would we properly recognize them as Strong SIs? Since the knowledge they accessed that heralded their ascension to Strong SI status is beyond our ability to appreciate, how could we distinguish between a Strong SI and a Weak SI that has lost its marbles? When faced with such a choice it appears that our only guidance becomes the faith we have in the SI. Either we say, "Gee, what the SI is doing now is completely beyond me. Since its a SI, it probably knows what its doing. Its a Strong SI" or we say, "This thing has gone nuts." Another problem is differentiating between knowledge that is forever beyond us and knowledge that is beyond us for now but will be understandable after another 200 years of scientific work.
Doug Bailey
doug.bailey@ey.com