Re: Open Peer review on Internet

Robin Hanson (hanson@hss.caltech.edu)
Wed, 25 Sep 96 13:27:50 PDT


Eric Watt Forste writes:
>it's starting to look pretty obvious to me that sooner or later we are going to
>have a recognized profession of credential-creators. People whose ability
>to judge work within a given field is fully recognized by practitioners of
>that field and is generally recognized by people outside that field who
>need to consume information produced by workers in that field. ...
>The potentials for abuse are stupendous. Here is a field ripe for those who
>are interested in designing alternative institutions. As far as I can tell,
>it'll be a matter of getting the design of the protocols right and leaving
>room for the future improvement of those protocols. What do you think,
>Robin? You've got a lot more expertise in this matter than I do.

Yes it is a ripe field for innovation. Therefore it is not obvious to
me that your solution of professional credentialors will win or is best.
Until you can describe an attractive plausible design along your
lines, I will think the game is still quite open.

Davin C. Enigl writes:
>An ordinary journal signals quality (or lack of) by its history. With time,
>this on-line publication system will be recognized as quality (or not). I
>think the requirement of application to join is a safeguard. Applications
>can be verified. "Recognized" names of people doing the peer review might
>help if they join.
>The peer review I have been involved with is not as good as this proposal
>because the reviewers were chosen poorly. They did not even know my subject.

Poor choice of reviewer is not intrinsic to the current process, and
evolutionary selection of journals should prefer those with
editors who pick reviewers who will pick articles their readers like.

I have since learned that this online journal idea lets reviewers post
numerical ratings, which are averaged into a paper rating. But since
anyone can review anything, I don't have great hopes for the average
quality of high-rated articles. But it is an empirical question.

Robin D. Hanson hanson@hss.caltech.edu http://hss.caltech.edu/~hanson/