Ander Sandberg (email@example.com) said:
>The problem seems to be that it is impossible to test very complex
>systems for all possible contingencies, and this will likely cause
>trouble when designing ufog. How do you convince the customers it is
I program medical and avionic systems, and the general criteria are pretty straight-forward.
You test the thing for designed behavior, and then you test it for environmental
benignity (e.g. operating-room equipment has saline solution pured on it, and
stell rods poked into open orfices, and various shorts on teh pweor plug)
>You get the same problem with AI: what testing would be required
>before an AI program was allowed to completely run a nuclear power
Well, speaking as a professional safety engineer, I think this would be an easy
argument. Just test the program with the same simulation used to train the
human operator. If it does ok, then it's ok.
However, most regulatory environments require that no single fault should
be able to induce a harmful failure (this is simple common sense,
Therefore, one might have a much easier time with certification if there was a second AI, with a different design, to second-guess or cooperate with the first. This makes the system far less prone to fail the first time an AI makes
a mistake, And, it _will_, right?