Eliezer S. Yudkowsky [SMTP:sentience@pobox.com] wrote:
> Problem is, choosing to commit suicide is still a choice - and that's
> not what I'm hypothesizing. At that level, I don't have the vaguest
> notion of what would really happen if an SI's goal system collapsed.
> The whole lapse-to-quiesence thing in Elisson is a design feature that
> involves a deliberate tradeoff of optimization to achieve a graceful
shutdown.
I know. But if they all have the same hoghly optimized cognitive architecture by the time they reach this point, and it happens to give a similar result, then we could actually get the whole civilization to shut down. Of course, that still leaves eveyone's automation running, and a lot of it is likely to be sentient...
No, I don't think it could really happen either. The whole scenario is just too contrived, and it definitely requires that the target civilization make mistakes (which isn't exactly a guide to useful thought when you're talking about SIs). However, it beats anything else I've seen. Killing off a civilization like this without sterilizing the universe is pretty hard to do.
> Well, if you're interested in a not-so-known-laws-of-physics
> speculation: The various colonies achieve SI more or less
> simultaneously, or unavoidably. The first thing an SI does is leave our
> Universe. But, this requires a large-scale energetic event - like, say,
> a supernova.
>
> Still doesn't solve the Great Filter Paradox, though. Some hivemind
> races will have the willpower to avoid Singularity, period. This
> scenario takes mortals and Powers out of the picture during a
> Singularity, but it doesn't account for the deliberate hunting-down that
> would be needed.
Agreed.
> I think the most plausible argument is this: Every advance in
> technology has advanced the technology of offense over the technology of
> defense, while decreasing the cost required for global destruction.
> There are no shields against nuclear weapons - not right now, anyway -
> and we've certainly managed to concentrate that power more than it's
> ever been concentrated before. In fact, the more technology advances,
> the easier it becomes to cause mass destruction by *accident*. It holds
> true from nuclear weapons, to biological warefare, to the Y2K crash, to
> nanotechnology. All you really need to assume is that the trend
> continues. Eventually one guy with a basement lab can blow up the
> planet and there's nothing anyone can do about it.
Maybe. But I think your example is an artifact of the nature of our recent historical situation. Nuclear weapons are a superweapon primarily because we can't disperse ourselves properly (Earth being too small for that purpose), and biological weapons have mass-destruction potential because we can't improve our immune systems.
If we were going to have a future without AI/IA, I think it is clear that these trends would reverse within a century. Spreading into interplanetary space would give us plenty of room to build economical defenses against nuclear weapons, and the combination of sealed environments and competent genetic engineering would make bioweapons a very limited threat. Evaluating the potential of nanotechnology is always tricky, but I think it actually ends up giving the defender a substantial advantage once it reaches a reasonable level of maturity.
IMO, the hard part is surviving the transition period we are currently in, where we have increasingly advanced destructive technologies but there is no room in which to deploy countermeasures.
Billy Brown, MCSE+I
ewbrownv@mindspring.com