(WAS: Failure of AI a prediction...)
Of late, the opinion has been expressed that it's okay to believe strongly in something as long as you have evidence for it - in other words, "dogma" describes a belief that is held strongly in the absence of evidence. I disagree. Any strong belief is a chain. Dogma is dogma even if the facts happen to be in correspondence with it. If an atheist would refuse to change vis mind if presented with contrary evidence, ve's being dogmatic.
I give credence to an opinion if it is caused by the facts, if a causal linkage exists between that opinion and the evidence, which requires that the removal of the evidence result in the removal of the opinion. Any strong belief causes the opinion to persist in and of itself and slows reaction to new information. Translation: It makes you stupid and slow.
Consider: A parent, born a Communist, becomes a Republican on the strength of the evidence. Vis child picks up the parent's reasoning and repeats it verbatim. The parent is rational, the child is not - even though they hold exactly the same beliefs. The parent would become a Libertarian if given a copy of Reason magazine; the child would remain a Republican.
Look at CSICOP and sTARBABY! Are there psychic powers? Very likely not. You shouldn't need to resort to distorting the evidence to prove it. But CSICOP still managed to screw up thoroughly, solely by believing too strongly in skepticism. They forgot that "scientific investigation" isn't in the answer, it's how you ask the question. They believed so strongly in a particular outcome that their experiments got sloppy, which is bad, and then they tried to cover it up, which is hideous. Moral: There may be strong rational evidence and excellent moral reasons for believing, but believe too strongly and you're screwed. Belief rots the brain.
A strong belief prevents you from factoring the negation of that belief
(at however low a probability) into your actions. I do not hold as a
point of dogma that a Singularity is possible; in fact, I factor the
(admittedly low) probability that it is *impossible* into my planning.
Strong beliefs mean being unprepared for anything you didn't believe in.
You must be able to emotionally accept both P and ~P before you can
rationally decide which is true. It is certainly acceptable to care
very strongly about the _decision_ - but it is not acceptable to care
too strongly about P.
Learn to operate without strong beliefs. Everything is a guess. I don't care who you are; at least one thing you strongly believe is false. Be prepared.
--
sentience@pobox.com Eliezer S. Yudkowsky
http://pobox.com/~sentience/AI_design.temp.html
http://pobox.com/~sentience/sing_analysis.html
Disclaimer: Unless otherwise specified, I'm not telling you
everything I think I know.