From: Peter C. McCluskey (pcm@rahul.net)
Date: Tue Aug 12 2003 - 10:13:17 MDT
sentience@pobox.com (Eliezer S. Yudkowsky) writes:
>Don't open-source an AI that can potentially be corrupted and that can
>accomplish more damage than uncorrupted AIs can correct. If you're in a
>state where a corrupted version of your AI can do serious global damage,
>then don't open-source that AI.
One problem with this is that you probably won't have a good estimate
of the damage that could be caused by a corrupted version, so this
reasoning will lead you to keep the source secret even if that slows
down development enough that someone else produces the first AI, and/or
hinders the debugging to a degree that you end up relying on an AI that
isn't as friendly as you think it is.
Another problem is that if you keep it secret and claim it's powerful
enough to save the world, it would seem sensible for anyone who believes
your claims about its power to suspect that you will use it to acquire
personal power over the rest of us.
If, as I suspect, releasing the source code only enables corrupted versions
to become powerful after years of learning (i.e. much of the value is in
a database produced by the AI that takes a lot of cpu time to duplicate),
then I recommend open-sourcing the AI even though that causes some hard-to-
evaluate risks of malicious AIs being produced at some uncertain time.
Remember that keeping it secret can't eliminate the uncertainty - once
someone shows that AI is possible, there will be lots of people trying to
write their own AI. It's just a matter of time before they succeed.
-- ------------------------------------------------------------------------------ Peter McCluskey | "To announce that there must be no criticism of http://www.rahul.net/pcm | the President, or that we are to stand by the | President right or wrong, is not only unpatriotic | and servile, but morally treasonable to the | American public." - Theodore Roosevelt
This archive was generated by hypermail 2.1.5 : Tue Aug 12 2003 - 10:22:24 MDT