Re: Is this safe/prudent ? (was Re: Perl AI Weblog)

From: Anders Sandberg (asa@nada.kth.se)
Date: Wed Aug 13 2003 - 11:50:06 MDT

  • Next message: John Grigg: "Re: protectionism (For Damien)"

    On Wed, Aug 13, 2003 at 09:45:55AM -0700, Samantha Atkins wrote:
    > On Wednesday 13 August 2003 02:48, Anders Sandberg wrote:
    > >
    > > If the seed AI exponential self-amplification scenario is true,
    > > then this approach will of course be too late. In that scenario,
    > > the only policies seem to be to call for draconian restrictions,
    > > centralising the project (into whose hands?) or have a
    > > free-for-all to become Mother of God.
    >
    > True. But locking the barn door before the horse exists hardly seems like
    > reasonable policy. Too much paranoia now about some theoretical scenarios
    > could very well rob us of the many benefits of all levels of AI, benefits
    > that are very much required if we are to have the intelligence to chart our
    > course wisely.

    Exactly. The problem is to keep a balance between worrying about
    possibilities and waiting too complacently. If we show that we care about
    the risks and want to minimize them, many people react favorably - but too
    much disaster scenarios may make people fearful. Showing that many worries
    are groundless is important to help research develop, but we should not be
    polyannas assuming everything automatically turns out the best.

    > > The precautionary principle in its active form suggests that we
    > > should actively find ways of protecting ourselves from the risks
    > > of a bad AI even if it is an unknown factor; friendliness theory
    > > is a start, building institutions is another. But that
    > > shouldn't distract us from actually getting a better knowledge
    > > of the problem.
    >
    > The precautionary principle is one of the worst notions I have ever
    > encountered as it proposes to base policy on the ability to prove a negative.

    Actually, in its simple and correct form it is just common sense. What you
    react to is how it is *usually* used - "don't do anything until you have
    proven it safe!" which is actually a misinterpretation of it (just look at
    how it is stated in the Rio Protocol, as an example). The corrupted
    version is unfortunately the one that is used most widely by conservatives
    of all colors. The active form I mentioned (absence of scientic proof of a
    danger is no reason not to proactively minimize dangers) can of course be
    used to stifle development too, but judiciously used it makes a lot of
    sense when venturing into the unknown.

    -- 
    -----------------------------------------------------------------------
    Anders Sandberg                                      Towards Ascension!
    asa@nada.kth.se                            http://www.nada.kth.se/~asa/
    GCS/M/S/O d++ -p+ c++++ !l u+ e++ m++ s+/+ n--- h+/* f+ g+ w++ t+ r+ !y
    


    This archive was generated by hypermail 2.1.5 : Wed Aug 13 2003 - 11:57:55 MDT