"Michael S. Lorrey" wrote:
>
> You create a blind spot. In the blind spot is the 'concience' kernel, which
> cannot be directly manipulated. It can only be programmed by the experiential
> data input which it analyses for useful programming content. It colors this new
> content by its existing meme set before it integrates the new content into its
> accumulated database of 'dos' and 'donts'. The entire database gets to vote on
> every decision, so new content cannot completely wipe out old content, except
> under extremely stressful circumstances (i.e. HOT STUFF HURTS!).
Inadequately specified cognitive elements. What is a "meme set"? What are "stressful circumstances"? When you say "it analyzes", what exactly is doing the analyzing, and how, and why do you trust it?
>From http://www.tezcat.com/~eliezer/AI_design.temp.html#pre_prime
The Rigidity Problem:
"So?" you say. "Don't let the AIs reprogram the goal system."
Leaving out the equivalence of the goal system and the reasoning
system, leaving out the ubiquitous and reasoned connections between
goal and action, leaving out the web of interdependencies that make
goal behaviors a function of the entire AI...
==
What is likely to happen if the whole AI gets upgraded except for the goal system? About the same thing that would happen if you took a gigantic program going through a major revision and suddenly made 10% of the code unalterable. Parts of the basic architecture would be frozen. Rewrites going past the surface would be almost impossible.
If the basic architecture of all the AI except the goal system changed, one of two things would happen. The goal system would fall out of synchronization with the rest of the AI, and would increasingly exhibit random behavior - in other words, the AI would go gaga. The other thing that might happen is that other parts of the architecture would have new and internal goal systems, to replace the useless and unalterable legacy code. Eventually, the new goal system would be both broader and more intelligent than the old code, and they would come into conflict. ==
This also sounds like a prime target for the Architecture Change Problem, the Unstable Definition Problem, the Goal Conflict Problem, the Entailment Problem, and the Redirection Problem. Depending on how you're enforcing the "blind spot", you could also run into the Combinatorial Hack Problem. On the plus side, as far as I can tell, the proposed architecture is too primitive to run into the Inconsistency Problem, the Resonant Doubt Problem, or the Arbitrary Goal Problem.
The Recursive Watcher Problem isn't an actual "problem", an anomalous AI behavior; it's just a technical impossibility.
--
sentience@pobox.com Eliezer S. Yudkowsky
http://pobox.com/~sentience/AI_design.temp.html
http://pobox.com/~sentience/sing_analysis.html
Disclaimer: Unless otherwise specified, I'm not telling you
everything I think I know.