Damien Broderick wrote:
> At 01:59 PM 28/09/00 -0400, Dan wrote:
>
> >Alternately, you hopefully remembered to build the goal system so that
> >it was not supposed to pursue (sub)goals X, Y and Z but to pursue
> >(sub)goal A: "try to do what we ask you to do, even if we change our
> >minds." So, at the primitive stages, doing what you ask shouldn't be
> >a problem, presuming it's at least complex enough to understand what
> >you're asking for in the first place.
>
> Well... This is Stanislaw Lem territory, but the clearest and funniest and
> scariest dramatization I know is Egan's QUARANTINE, where a loyalty mod is
> circumvented by all-too-plausible sophistry (or is it sheer logic?),
> enabling an enslaved mind effectively to burst free by re-entrant
> redefinition of its goals and strategies *while remaining enslaved to the
> mod*.
The key qualifier to my claim was "at the primitive stages." At
sufficient complexity, this trick will fail to work, in which case
you'll have to resort to one of the other two options I suggested:
shutting it down without consent (if it's that important), or asking
it very nicely in terms that make sense to it, if there's nothing you
can do about it.
-Dan
-unless you love someone-
-nothing else makes any sense-
e.e. cummings
This archive was generated by hypermail 2b29 : Mon Oct 02 2000 - 17:39:21 MDT