Stanley Fish has been around the academic block more than most of us.  During and after service as a dean at the University of Illinois Chicago he wrote a wonderful column for the Chronicle of Higher Education, “All in the Game”, in part morality tales about academic faculty behaving badly.  A favorite, entitled “Somebody Back There Didn’t Like Me“[1], speaks of senior faculty whose work was found wanting at an earlier stage and thereafter “spend a lifetime trying to wash away the taint by being … ungenerous to others …invoking standards much more rigid than the standards they know you think they didn’t quite meet…”[2]

It has been my privilege to observe and participate in peer review of manuscripts, grant applications, and people.  For the most part, it is an effective process for allowing ‘the cream to rise to the top‘, and a demonstration of the wisdom of crowds in action.  My fellow reviewers have been, for the most part, diligent, insightful, and fair.  Indeed, for the vast majority of peer review meetings I’ve attended, those being reviewed could have listened to an anonymized recording and concluded that justice was done.  For the most part…

Every so often, however, peer review goes horribly awry, and can resume its appropriate course only with great effort.  In addition to the behavior Fish describes, reviewers play out political agendas in the guise of peer review, and compete to see who can be the most devastatingly critical.[3]

Why fishy[4] behavior?

Peer reviewers are people first and distinguished academic faculty second.  As people, they are always of two minds, one driven by threat and opportunity and jumping to conclusions, and the other capable of rational but slow deliberation.  Importantly, while only the second (“System 2”) can perform appropriate peer review, System 2 can undertake only what the first (“System 1”) will allow:

• In academia, vulnerability to threat and access to opportunities and resources, while ideally only a function of academic merit, is also a function of stature and reputation.  System 1 promotes competition for stature and reputation. Peer review is an opportunity to augment or consolidate stature and reputation, and this opportunity if taken may trump thoughtful deliberation.
• System 1 is attentive to what others in a group think and do.  In peer review, System 1 is highly averse to appearing stupid, ignorant, or wrong — sometimes leading to silence when challenges should be raised if not confirmation bias or groupthink. Innovation or activity outside the cultural norm can be disrespected for this reason regardless of its merit (see ingroup-outgroup bias).
• “Fairness” is a key value of academic faculty.    ‘Everyone else had to go through this so it is only fair to continue it’ is a powerful rationale for the perpetuation of bad practice unrelated to actual academic merit.  As with hazing, this consolidates group coherence and social stature, which System 1 values.
• Academic faculty also afford stature to those with high standards and critical ability[5], and System 1-driven competition for stature can game this by elevating standards to unrealistic levels[6] and through excessive criticism

In rare[7] cases these tendencies can result in fishy behavior. Three primary manifestations are snobbery, envy and retribution. They are natural and automatic.[8]

Snobbery is the unjustified belief that one (and/or one’s tastes, accomplishments, stature, importance) is superior to others (see illusory superiority). If everyone is above average or anyone can equal the snob, the belief is refuted. Therefore, the snob can support the belief by denying others what is believed to make the snob superior.

Envy is “a painful emotion of wanting an advantage that another has and/or wishing the other did not have it or, simply put, ‘pain at another’s good fortune’”.[9]   In peer review, one (but not the only) form is envy of having come by success ‘the easy way’; i.e., without the trials and tribulations to which the envier was subjected.[10]

Retribution is a tendency to punish others “because they deserve it” as an end in itself. [It differs from utilitarian punishment intended to deter future bad behavior. These are not mutually exclusive.][11]

On these bases, peer reviewers may oppose, cheat, or even intimidate or sabotage those being reviewed.[12] Research shows that even when a group member’s own situation is fixed (e.g., full professors who have already been promoted and cannot be promoted to a higher rank), members will undermine the assessment of others or seek to inflate their own ranking.[13]

The impact of these rare but fishy behaviors can be enormous. When they lead to the denial of funding, publication, or appointment/reappointment/promotion/tenure to worthy candidates, those candidates may abandon worthy projects, experience stress disorders, leave the institution, or henceforth underperform and/or engage in these fishy behaviors themselves[14]. This behavior is also corrosive to those who engage in it.[15] When peer review loses its credibility as an objective arbiter of academic worth, the credibility is difficult or impossible to regain. That is, academia is damaged.

What to do?

As noted, these bad behaviors are natural and automatic. No one is immune to them. Rather than pretend they don’t exist, devise countermeasures.

√  One of System 1’s greatest strengths (the ability to take quick action based on the behavior of other people) can also be used to subdue it. As noted, System 1 is enormously deferential to what other people think. It will go out of its way to avoid embarrassment and public shaming. When it knows others are the lookout for bad behaviors that are the moral equivalent of academic fraud, it will do its best to avoid creating a pretext for accusation. I suggest that every peer review meeting begin with the chair stating: “All of us need to scrutinize one another for snobbery, envy, retribution, implicit bias, and so on, and so perform rational and objective assessment. We will be on guard.” [16]

√  Use circuit-breakers and pauses. Leaders of peer review committees can systematically pause discussions and ask: “Is there any possibility that unconscious behaviors are distorting our assessment?” Or, better still, “How have unconscious behaviors distorted our assessment?”[17]

System 2 is quite competent to conduct high-quality peer review when it is free to do so.

[18]

[1] http://chronicle.com/article/Somebody-Back-There-Didnt/46188/

[2] For a scholarly treatment of how past events may provoke such responses, see: Veiga, JF, Baldrige, DC, Markoczy, L. 2014. Toward greater understanding of the pernicious effects of workplace envy. Int. J. Human Resource Management 25: 2364-2381

[3] If you ever have an opportunity to see the CRTL Players enact a peer review session (http://www.crlt.umich.edu/crltplayers/fence) , take it! If you can’t, another depiction is at https://www.youtube.com/watch?v=H0_EHJN_TDs . A shout-out to the Purdue ADVANCE program for producing the latter.

[4] http://www.merriam-webster.com/dictionary/fishy , second definition

[5] In my institution’s lore, for example, is the following quote:

Chicago has developed a celebrated—some would say notorious—brand of academic civility. It is a place where one is always in principle allowed to pose the hardest question possible—of a student, a teacher, or a colleague—and feel entitled to expect gratitude rather than resentment for one’s effort. The trait is frequently noted (not always approvingly) by scholars from other institutions who visit us. …When Weber wrote about the scholar’s obsession with devil’s advocacy, he could have been talking about the University of Chicago.

[Report of the Faculty Committee for a Year of Reflection. The University of Chicago Record 32: 2-13, 1998.]

[6] In institutions is usually a tension between standards and pragmatism. On the one hand, the higher the standard, the better the personnel and the higher the quality of the product, all else equal. Personnel who don’t meet the standard can be turned away. On the other hand, regardless of the standards you ‘need to field a team’ [in academia, teach students; in academic medicine, take care of patients]. This can lead either to the acceptance of personnel who don’t meet the standard and/or can be recruited inexpensively, or default on responsibilities. Some institutions resolve this tension better than others (cf. record of championships for New England Patriots vs. Chicago Cubs).

[7] I do not wish to exaggerate the frequency of such instances. They are rare. We should nonetheless strive to reduce their incidence to zero, which is the rationale for this post.

[8] Menon, Tanya, and L. Thompson. 2010. Envy at work. Harvard Business Review. https://hbr.org/2010/04/envy-at-work

[9] See footnote 2, and references cited therein.

[10] Bokonon tells us: “Beware of the man who works hard to learn something, learns it, and finds himself no wiser than before. He is full of murderous resentment of people who are ignorant without having come by their ignorance the hard way.” Vonnegut, K. 1963. Cat’s Cradle. Holt, Rinehart and Winston. http://amzn.com/038533348X

[11] Tan, Fangfan and E. Xiao. 2014. Third-party punishment: retribution or deterrence? Max Planck Institute for Tax Law and Public Finance Working Paper 2014-05. http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2425522

[12] See footnote 8 and footnote 11

[13]Charness, G, Masclet, D, Villeval, MC. 2012. The dark side of competition for status. University of California at Santa Barbara, Economics Working Paper Series. http://escholarship.org/uc/item/1vr4g446

[14] A prior post (https://decannomics.com/2015/06/26/13-yes-and/) quoted WH Auden (http://www.poets.org/poetsorg/poem/september-1-1939):

“I and the public know
What all schoolchildren learn,
Those to whom evil is done
Do evil in return.”

[15] See footnote 8

[16] This is the equivalent of posting eyes above an ‘honor box’ where one voluntarily pays for coffee, a newspaper, etc. https://www.newscientist.com/article/dn9424-big-brother-eyes-make-us-act-more-honestly/   See: Bateson M., Nettles D., Roberts G. 2006 Cues of being watched enhance cooperation in real-world setting. Biol. Lett. 2, 412–414.

[17] As noted in the footnotes to https://decannomics.com/2015/07/19/16-curve-ball/ , a a former Provost at my institution would routinely ask in tenure cases: Has the implicit bias in letters of assessment affected judgment?

[18] I am grateful to Tanya Menon and Fangfan Tan for correspondence on this topic, and to Tanya Menon for sharing her unpublished work.