“There’s no such thing as a stupid question.[1]” Big Bird[2]

The problem

An earlier post discussed reluctance to seek help, advice, and feedback, especially when to do so we appear ignorant. In Think Like a Freak, Levitt and Dubner[3] state “I don’t know” are the “three hardest words in the English language. People in general have issues with admitting ignorance. As children we have a voracious appetite for knowledge and no reluctance to ask for it. Why do many[4] of us lose this, and why is its loss so pronounced in academics?

• Some elders, parents, and educators teach us never to admit ignorance[5]
• Fear
• Perceived or genuine intimidation by ones higher in the institutional or professional hierarchy; high Power Distance Index
• Embarrassment; reluctance to “look stupid”; insecurity; shame
• Reluctance to “bother” those with knowledge and expertise to share
• Various emotional ‘triggers’[6]
• Threat to self-image and stature

Let’s explore the last a little bit. As academics we strive to become the authority or the leader in something. This is the foundation of our professional reputation and identity, professional advancement, and ability to attract the best colleagues, trainees, and funding and other resources essential to our academic productivity. Admissions of ignorance seemingly pose a grave risk to this foundation.

Unfortunately, there is an even graver risk to this foundation: failure to advance. As fields move forward, authorities and leaders must advance with them or get left behind.

Typically our colleagues (be they already in our circle of professional acquaintances, outside it, our trainees, or higher-ups in our institutional or professional hierachies) are the solution for failure to advance. Moreover, most of us are eager to share our knowledge and expertise with colleagues needing it, and derive great satisfaction from helping others in this way. Even if we can’t, we typically know of colleagues who can and will. But the transmission of knowledge and expertise is utterly defeated if the ones needing to learn can’t bring themselves to say: “I don’t know. Will you teach me or show me how?”

Solutions

The first step of twelve-step programs is to concede there’s a problem.[7] [But see WWKD’s comment at the end of http://wp.me/p5LUkN-26.}

After that, for those who cannot admit ignorance to more knowledgeable colleagues, knowledge and expertise is available in other ways. These days short courses and training seminars are available on practically every topic, academic and otherwise, as are professional coaches and tutors. In biomedical research in particular, many vendors of equipment or reagents offer infomercials on their products and even personalized training. Most states require a certain number of continuing medical education credits for renewal of medical licensure.[8]

But what about those for whom these work-arounds are insufficient, who must implicitly or explicitly admit ignorance to colleagues in order to advance?

Some prior posts

3. Why it’s easy to commit to faculty development activities, and difficult to follow through on these commitments [Happy New Year!]

3.5 Devices that help us keep commitments [A Bestiary of Commitment Devices]

4. Why we are averse to feedback and what to do about it [Difficult Conversations]

7. Fear of interaction with others impedes faculty development. It can be countered. [The Only Thing We Have To Fear]

14. Judicious application of forcing functions can yield faculty development [Almost The End of Civilization As We Know It]

23. Do we need ‘career resource management?  [Fly the intimidating skies: ‘The Ethnic Theory of Plane Crashes’ applied to faculty career development in academic health centers]

may contain helpful advice, as will future ones.

Email may have its problems, but often is less intimidating to the sender than requests via telephone or in person.

Cognitive behavioral therapy, which can be self-administered, can be effective in overcoming avoidance. It involves graded exposure to anxiety-provoking situations. Indeed, we have used this effectively in junior faculty, assigning them progressively more anxiety-provoking “asks” in a supportive peer group setting.

When anxiety is overcome, ignorance is admitted, and help is obtained the first time, subsequent admissions of ignorance and requests for help may become progressively easier.

Be a sympathetic colleague. When you see a colleague who is struggling to admit ignorance, share your personal stories and offer to facilitate:

ROBIN: I realize what my research needs is the new quantum technique that Lakisha Jones is using in her lab, but I don’t know the first thing about quantum techniques and would just look stupid. Besides, Professor Jones is world-famous, doesn’t know me, and I hate to bother her.

YOU: Yes, I remember feeling like that once. I know Lakisha well [whether you do or don’t]. How about I sound her out about helping you?

============

YOU: Professor Jones, I have a junior colleague, Robin Smith, who is very interested in learning the new quantum technique you’re using. Would you be willing to help?

LAKISHA: Who? Oh, never mind. Sure! Ask Robin to get in touch.

Finally, know that we all face this issue. Here are some of UChicago’s finest confronting it:

[1] http://sesamestreet.freeservers.com/bbird_and_snuffy.html

[2] https://en.wikipedia.org/wiki/Big_Bird

[3] Levitt, S.D., Dubner, S.J. 2016. Think Like a Freak: The Authors of Freakonomics Offer to Retrain Your Brain. HarperCollins. http://a.co/cwZS0LL

[4] This tendency varies by gender [Babcock, L., Laschever, S. 2007. Women Don’t Ask: The High Cost of Avoiding Negotiation–and Positive Strategies for Change, Bantam. http://a.co/8cJvmGc ] and national culture [see ‘The ethnic theory of plane crashes’ , Chapter 7 in Gladwell, M. 2008, Outliers: the story of success. Little, Brown and Company. http://a.co/3yZDwhA ], moreover.

[5] Cohen, Leah Hager 2013. I don’t know: In Praise of Admitting Ignorance (Except When You Shouldn’t). Riverhead Books. http://a.co/fw9V1kS

[6] Difficult Conversations: How to Discuss What Matters Most, by Douglas Stone and Sheila Heen. Penguin Books. http://www.amazon.com/Difficult-Conversations-Douglas-Stone/dp/0143118447

[7] https://en.wikipedia.org/wiki/Twelve-step_program

[8] https://www.fsmb.org/Media/Default/PDF/FSMB/Advocacy/GRPOL_CME_Overview_by_State.pdf

“Be happy in your work.”[1]

Janet Bickel likens academic career development to a long-distance hike[2], requiring sustained effort. In the tenure track, advancement from assistant professor to associate professor takes 5-9 years of strenuous effort at many institutions, and advancement from associate professor to full professor often requires 5-10 more. Problem is: as a species we’re not good at such sustained effort. Consider physical exercise and weight-loss diets. A minimum of 50% of us who begin these give up, with some sources reporting up to 90% attrition[3]. Why? We don’t find these efforts to be enjoyable, and they cause stress.

One approach to sustaining academic effort is using commitment devices (also here), forcing functions, and social pressure — all external. Although these can be effective, they are not always pleasant and can add to stress.

With respect to exercise, in her book No Sweat[4] Michelle Segar has an effective and ingeniously simple alternative solution to the problem of giving up: find a form of exercise that you enjoy performing.

According to Segar:

• Unsustainable exercise regimes typically result from negative motivations (too unfit, too obese, too unattractive, too unhealthy, too embarrassed) for unpleasant exercise (an unwanted chore, a necessary evil, punishment, self-torture or, at the very least, taking time away from more important activities) with no immediate gratification (i.e., fitness takes time to develop). Sustained will power can overcome this combination, but is seldom sufficient.

• A first step is to reframe the activity as a gift to one’s self, something one wants to do autonomously, a reward.

• A next step is to find pleasurable forms of physical activity that are consistent with this reframing. Enjoyability trumps effort because activities that are not enjoyable are probably not sustainable. This is very much an individual preference. Some individuals enjoy intense work-outs, long runs, and sweating; some do not or lack the time for them. For the latter individuals, if the ‘medical model of exercise’[5] is set aside, opportunities for physical activity become obvious and activity more feasible.

[These two steps require resisting common societal and advertising messages that only those forms of exercise that are unpleasant or the ‘medical model’ are effective. To be sure, some forms of activity improve fitness and weight more than others, but if they are unsustainable because they are unpleasant, their overall impact will be small and temporary.]

Applying Segar’s approach to career development:

• If one’s academic activity is unpleasant, it is probably not sustainable. Find forms that are pleasant or rewarding, and hence sustainable.

• Within one’s job description is almost always an opportunity for choice or flexibility — of research project, of classroom approach to a topic, of how to contribute to one’s academic community, etc. Choose the options that are pleasurable or rewarding. They then become not ‘just your job’, but opportunities for enjoyment or reward. Gratification is both delayed (the eventual promotion) and, more importantly, immediate (you are doing something you enjoy).

[Presumably we entered academia because we enjoy the processes and challenges of creating knowledge, convincing peers, and educating or training others. Remember the enjoyment. Or, if it is no longer pleasant or rewarding, consider transition to a career that is.]

Admittedly, this prescription may be more feasible in some systems than others. At my institution we have a standard tenure track in which assistant professors who find their primary joy in teaching (and detest research) would be well advised to find another institution. For faculty who care for patients, however, we (and a growing number of academic health centers) have a second track that credits diverse contributions. In this track faculty can advance through any contribution that sufficiently enhances the distinction of the academic health center. We liken this to the Olympic or Paralympic Games, wherein individuals of diverse abilities and talents can excel. What matters for promotion is less the sport, and more that the competitor medals. In such a system, Michelle Segar’s prescription becomes: Figure out where your passion and institutional needs overlap, and then pursue your passion.

Such freedom of choice, however, can pose its own problems of decision paralysis. But one can’t have everything.

No sweat!

[Here is Bassam Shakashiri, Professor of Chemistry at the University of Wisconsin and former Assistant Director of the National Science Foundation for Science and Engineering Education from 1984 to 1990, reminding us that what we do is and should be fun:

][6]

[1] http://www.imdb.com/character/ch0219243/quotes

[2] Bickel, J. 2008. Career development as a long-distance hike. J Gen Intern Med 24(1):118–21. DOI: 10.1007/s11606-008-0834-3. http://www.janetbickel.com/docs/09hike.pdf

[3] See, for example, https://www.bodybuilding.com/fun/2013-100k-transformation-contest-press-release.html , https://www.creditdonkey.com/gym-membership-statistics.html , https://www.linkedin.com/pulse/why-90-you-quit-gym-early-2015-paul-elsass-msm , https://link.springer.com/article/10.2165/00007256-199417010-00004 .

[4] Segar, M. 2015. No Sweat: How the Simple Science of Motivation Can Bring You a Lifetime of Fitness. Amacom. http://a.co/5yEvp94

[5] e.g., http://www.heart.org/HEARTORG/HealthyLiving/PhysicalActivity/FitnessBasics/American-Heart-Association-Recommendations-for-Physical-Activity-in-Adults_UCM_307976_Article.jsp – .WZMxYq3GxPk; https://www.cdc.gov/cancer/dcpc/prevention/policies_practices/physical_activity/guidelines.htm

[6] http://scifun.chem.wisc.edu/xmaslect/liquid-oxygen.jpg

“Culture eats strategy for breakfast.” Attributed to Peter Drucker[1]

Academic health centers (AHCs) are strange places. Unlike some academic institutions, which are more collegial and less hierarchical, AHCs have a chain-of-command culture as in the military. Deans have command authority and real power over chairs, chairs over chiefs, chiefs over sub-chiefs, and sub-chiefs over rank-and-file faculty. This makes sense; after all, human life is at stake and AHCs are especially complex and difficult to manage[2]. But a military-like culture has unintended consequences for faculty career development: Those higher in the hierarchy have the wisdom and knowledge that facilitate career development and are typically eager to share it with those lower in the hierarchy. Those lower in the hierarchy, however, can be reluctant to benefit[3] or even engage[4].

We have other examples of hierarchy impeding productive interaction:

This is what became of Korean Air Flight #801[5]

This is what became of Avianca Flight 052[6]

This is what became of United Airlines Flight 173[7]

As Malcolm Gladwell explains in “The Ethnic Theory of Airplane Crashes”[8], such crashes occurred because in teams of aviation personnel, those lower in a hierarchy were reluctant to engage those higher up, and those higher up did not effectively encourage such engagement. Similar disasters can ensue in diverse situations (e.g., hospital emergency rooms, intensive care units, and operating rooms; nuclear reactors).

An explanation for such disasters, as Gladwell contends, lies in the “Power Distance Index” (PDI)[9], developed by Geert Hofstede and defined as “the extent to which the less powerful members of organizations and institutions accept and expect that power is distributed unequally.” In this dimension, inequality and power is perceived from the followers, or the lower level. PDI can be calculated from subordinates’ scoring (on a scale of 1 to 5) of pairs of statements such as:

In high PDI cultures, subordinates and superiors can be less likely to interact appropriately in situations in which teamwork is critical. Airplane cockpits can be high PDI cultures. Gladwell describes such an instance:

“they died because… when the copilot asked questions, his implied suggestions were very weak. The captain’s reply was to ignore him totally. Perhaps the copilot did not want to appear rebellious, questioning the judgment of the captain, or he did not want to play the fool because he knew that the pilot had a great deal of experience flying in that area. The copilot should have advocated for his own opinions in a stronger way…”

The air crews of Korean Flight 801 and Avianca Flight 052 were from cultures in which the PDI is especially high. Gladwell writes:

“As one former Korean Air pilot puts it, the sensibility in many of the airline’s cockpits was that ‘the captain is in charge and does what he wants, when he likes, how he likes, and everyone else sits quietly and does nothing.’

So, when the first officer says, ‘Don’t you think it rains more? In this area, here?’ we know what he means by that: Captain. You have committed us to visual approach, with no backup plan, and the weather outside is terrible. You think that we will break out of the clouds in time to see the runway. But what if we don’t? It’s pitch-black outside and pouring rain and the glide scope is down.

But he can’t say that. He hints, and in his mind he’s said as much as he can to a superior. The first officer will not mention the weather again. “

The United 173 crash (in which flight team members failed to inform one another that fuel was about to exhaust) was among those that motivated the development of crew resource management (CRM), which includes training in overcoming the hierarchical differences among aviation team members when necessary[10]. Korean Air then used CRM training to improve its safety record remarkably. CRM has since been applied to health care, firefighting, and other situations in which emergencies occur.

If academic health centers have high PDI cultures in which “subordinate” faculty with career development needs are reluctant to engage or approach “superior” colleagues who want to provide wisdom and knowledge, do they need “career resource management” training? If so, (a) how can an institution implement this practice, and (b) what should the content be?

With respect to implementation, the US Federal Aviation Administration advises: “It is essential to get buy-in from the entire organization, from the top down and from the bottom up. Every member of the organization from management to crew must be vested in the program. Any broken link or disinterested party in the organizational chain will affect the success of the CRM program.”[11] Despite their own advice, ironically the FAA actually implemented CRM by mandating it.[12]

With respect to content, three suggestions as a start:

• Those higher in the hierarchy can learn to be less intimidating. Kevin Grigsby[13] makes some useful suggestions

• Those lower in the hierarchy can learn to be less intimidated. Janet Bickel[14] makes some useful suggestions.

• All positions in the hierarchy can learn to interact more productively in the context of career development.[15]

More to come in future posts.

[1] https://www.quora.com/Did-Peter-Drucker-actually-say-culture-eats-strategy-for-breakfast-and-if-so-where-when

[2] http://www.cfar.com/sites/default/files/resources/Leading_Planning_AMC.pdf ; R. Kevin Grigsby (2015) Enhancing the Behavioral Science Knowledge and Skills of 21st-Century Leaders in Academic Medicine and Science, Journal of Organizational Behavior Management, 35:1-2, 123-134. http://dx.doi.org/10.1080/01608061.2015.1031428

[3] 7. The Only Thing We Have To Fear. https://decannomics.com/2015/05/18/7-the-only-thing-we-have-to-fear/

[4] A senior and respected faculty member and friend recently avoided multiple requests to suggest improvements in faculty development. Eventually I insisted on a meeting, in which I asked: “Why are you so so reluctant to meet, even though we are friends and you know your suggestions could benefit faculty and would be warmly received?” The response: “In high school nobody wants to go to the Principal’s Office. It feels like I’m going to the Principal’s Office.”

[5] http://www.airlinepilotchatter.com/2013/08/land-short-accidents-part-iii-korean-801.html

[6] https://en.wikipedia.org/wiki/Avianca_Flight_52 – /media/File:Avianca-Flight-52-Wreckage-1.jpg

[7] http://www.planecrashinfo.com/w19781228.htm

[8] Chapter 7 in Gladwell, M. 2008, Outliers: the story of success. Little, Brown and Company. http://a.co/3yZDwhA

[9] https://en.wikipedia.org/wiki/Hofstede%27s_cultural_dimensions_theory

[10] Vanderbilt University Medical Center has an excellent and concise summary of CRM in the context of medicine at https://ww2.mc.vanderbilt.edu/crew_training/

[11] https://www.faa.gov/tv/?mediaId=447

[12] https://www.gpo.gov/fdsys/pkg/FR-1995-12-20/html/95-30449.htm

[13] R. Kevin Grigsby (2015) Enhancing the Behavioral Science Knowledge and Skills of 21st-Century Leaders in Academic Medicine and Science, Journal of Organizational Behavior Management, 35:1-2, 123-134. http://dx.doi.org/10.1080/01608061.2015.1031428

[14] http://www.janetbickel.com/docs/APSBoss.pdf and http://www.janetbickel.com/docs/APSbossImp.pdf

[15] Bickel, Janet MA; Rosenthal, Susan L.  2011. Difficult Issues in Mentoring: Recommendations on Making the “Undiscussable” Discussable.  Academic Medicine 86:  1229-1234.  doi: 10.1097/ACM.0b013e31822c0df7

I thank Janet Bickel and Kevin Grigsby for comments.
Continue reading →

When my children were young and we took them on long road trips, I would pass the time by inventing stories to explain common expressions.  We were once in Colorado, outside Vail, when I told them the story of Charles Vail, the highway engineer who routed U.S. Highway 6 through the Vail Valley in 1940, which eventually became Interstate 70.  Charles had three sons, Ebenezer,  Japheth, and Noah, the youngest.  Such being the times, they volunteered to serve in the Army, the Air Force, and the Navy, respectively.  Ebenezer served with distinction in Army Intelligence.  He was a key player in the capture of the Enigma device.  [Hereupon ensues the fantastic and prolonged tale of his derring-do.]  And, of course, Japheth’s role in providing air support for the Normandy invasion will never be forgotten.  [This too was recounted at length.]  Both were decorated for their exploits, and returned home to heros’ welcomes when World War II ended.  But I digress.   Noah, his youngest son, was assigned to a submarine, the USS Toothfish, in the Pacific.   He served with distinction beginning in the Battle of Midway. [More exploits…  this was a long car ride.]   Unfortunately, the Toothfish was reported missing after the Battle of Roratunga.

Throughout all this, Charles Vail was a very devoted father.  Every day when his sons were in away in the service and until they returned home, he would write a letter to each of them.  Indeed, for the rest of his life he would write a daily letter to his youngest son.  They were never answered, of course.

And that is the origin of the expression:  “…writing letters to no avail.”

Sometimes it feels like these blog posts are written for Noah Vail too.  Their intent was to elicit discussion among faculty and those who develop them on novel ways of achieving faculty development.  But discussion was sparse, leading me to take two years off from writing.

Maybe, as one critic suggested, I am too far ahead of the curve.  Diffusion of innovation theory posits that novelty, no matter how justifiable, takes hold only when it is embraced by early adopters, the early majority, the late majority, and so on.  And universities and academic health centers are notoriously resistant to change.

[from https://innovateordie.com.au/2010/05/10/the-secret-to-accelerating-diffusion-of-innovation-the-16-rule-explained/ ]

For now, it seems like just me and Noah.  If you have more than a passing interest in these writings, I’d love to hear from you.  Please feel free to reach out to me according to my standard contact information, or if you’d prefer via an alternative address:  noahvail4@gmail.com.  Or, at least, sign up as an official ‘follower’.

I am indebted to several colleagues in the Group on Faculty Affairs, especially Kim Skarupski, who encouraged me to take up the keyboard again.

On a mission

Karen Kelsky is on a mission to save the academic world from incompetent and/or irresponsible graduate advisors. She wrote:

Dear faculty members: I sell Ph.D. advising services on the open market. And your Ph.D. students are buying. Why? Because you’re not doing your job.[1]

Kelsky is both a former professor and chair and a PhD in cultural anthropology, which together make her eminently qualified to observe, interpret, and help the strange tribe we call academic faculty, faculty developers, and faculty wannabes. For years she’s written a blog, The Professor Is In. Just recently she’s published a book[2] of the same name, subtitled “The Essential Guide to Turning Your PhD Into a Job”.

Justifiably laudatory reviews of this book are available online, and I won’t repeat them here. As the title implies, it targets PhDs (and, to my mind, emphasizes the natural and social sciences); we could benefit from comparable works for physician-scientists and clinician-educators. Importantly, it fills a significant gap in the academic self-help literature, between ‘how to get a PhD’ and the ‘how to get promoted – how to help others get promoted’ continuum.   Some personal favorites pertinent to my world are:

• At the Helm: Leading Your Laboratory, Second Edition, by Kathy Barker[3]
• Academic Scientists at Work, Second Edition, by Jeremy Boss and Susan Eckert[4]

Why good advice is necessary but not sufficient

The most interesting part of The Professor Is In is what it almost doesn’t say or doesn’t say. One thing it almost doesn’t say is: “If you insist on a likely unsuccessful quest for an assistant professorship, this advice will keep you from becoming unsuccessful immediately.”  [You need to read carefully between the lines to see it, but it is there.  But let me make it more obvious.]

Neither The Professor Is In nor this blog is about whether to pursue a career as an academic faculty member or how to choose a training program that will best achieve this goal.  Imagine if they did.  Further imagine if, like medical centers and business schools, graduate programs were to disclose the fraction of matriculants with this goal who actually achieve it, and by when.  Right now, the disclosure for all biology PhD programs lumped together would look like[5]:

[Notice the framing in the upper right; an alternative statement would be >92% of entering PhD students will not become tenure-track faculty.]   Of course, past results are no guide to future performance; your mileage may vary; do not attempt at home.  This is an average for one area of graduate study; actual statistics differ among and within broad areas of study and among graduate programs.  Yet for many if not most areas and programs, the number of matriculants seeking assistant professorships vastly exceeds the number of assistant professorships.  Given these odds, why would any rational being count on becoming an assistant professor and not also prepare for an alternative career [except in those rare areas and programs in which the odds are good]?

Any rational being would say not, but we are people and thus not rational. Academic faculty and their trainees[6] collude in the collective delusion training only to become an assistant professor is often feasible if properly executed. Why? Our cognitive bias is: what you see is all there is[7]. All around us are people like ourselves who have ‘made it’. We tell ourselves stories in which the route to success is straightforward and we are the heroes.

My vain hope for the second edition: A sure-fire prescription for combatting this delusion, and its counterpart in faculty anticipating promotion despite evidence to the contrary.  Or, is it more important to maintain confidence and morale?

Job interview illusions

One of these stories we tell ourselves is that assistant professorships go to the best applicant. Indeed, much of The Professor Is In is about the obvious missteps to be avoided so that one is likely eliminated in due course instead of immediately. For example, Part V: Techniques of the Academic Interview, covers what to expect, what to wear, what to eat and drink, how to fend off inappropriate questions, etc. While its focus is on job interviews, it is broadly applicable to the many explicit and implicit interviews that academic life comprises.

Clearly, with >100 doctorates competing for a position, the selected applicant will often be very good by chance alone. But ‘the best’? The techniques we use to rank applicants are deeply flawed. According to Laszlo Bock, Google’s Senior Vice President of People Operations[8]

• Typical interviews explain only 14% of subsequent performance
• References explain only 7% of subsequent performance
• Number of years of experience explain only 3% of subsequent performance

The numbers speak for themselves.

Why are interviews, in which academic faculty place such stock, so problematic? For better or worse, the interviewers are human and will be subject to the biases of the human mind. They will, for example, prefer or disfavor the interviewee because of:

• The interviewee’s accent
• The interviewee’s names, both given and surnames
• Whether the interviewee is physically attractive
• The height, age, race, and apparent gender of the interviewee
• The time of day of the interview
• The weather on the day of the interview
• Whether the interview is the first, last, or in the middle of a series

in addition to the ‘usual suspects’ (prestige of training institutions, prestige of journal or press of publications). They will very quickly (within the first 10 seconds, according to work that Bock cites) form an impression, and spend the rest of the interview seeking information that confirms the first impression.

All of these preferences are unrelated to the ability to perform an academic job relative to other candidates, and most are beyond the control of the interviewee.

Moreover, a key component of many academic interviews is ‘the Macon Test’, as described by a faculty member in Atlanta, Georgia:

The Macon Test …is the mental calculation a faculty member employs to determine if he or she would like to have a faculty candidate as a companion on a three-hour road trip to some town like Macon, Georgia (scientists in the Northeast might use Poughkeepsie as the destination). This quick test is based on the first impressions of the people we meet. In this case, we, the faculty, ask ourselves if the prospective candidate would be good company over the long haul of an academic lifetime, or if he would be an annoying or boring passenger on the trip. In the latter case, the candidate flunks the Macon Test, and might not get a job offer.[9]

The Macon Test has some justification. Faculty members clearly have an interest in not appointing those who will be highly disruptive and poor academic citizens, or who will be unhappy in their ranks. But it is also a prescription for social conformity, groupthink, prejudice, and lack of diversity. Regardless, it typically will be administered in academic interviews – and a high score on the Macon Test can trump many a defect in an interviewee.

[Can these biases be defeated? Yes, but not by the interviewees. Google, which has more financial and human capital to invest than many search committees and even many universities, does this, as Bock describes in his book’s fifth chapter, entitled “Don’t Trust Your Gut: Why our instincts keep us from being good interviewers, and what you can do to hire better.” Google eschews unscripted interviews, and uses combinations of tests that they validate against actual job performance (including the performance of applicants they reject). Interestingly, their studies have led them to exclude managers (in academic-world, the faculty of the hiring department) from the hiring decision.]

Karen Kelsky’s next book

A larger question is: why do we persist in such delusional behavior when we should know better? [And if you are wondering ‘why a post on graduate advising’, the question is applicable to the entire faculty life cycle.]

As Woody Allen’s character puts it in the conclusion to Annie Hall[10]:

It reminds me of that old joke- you know, a guy walks into a psychiatrist’s office and says, hey doc, my brother’s crazy! He thinks he’s a chicken. Then the doc says, why don’t you turn him in? Then the guy says, I would but I need the eggs. I guess that’s how I feel about relationships. They’re totally crazy, irrational, and absurd, but we keep going through it because we need the eggs.

Books such as The Professor Is In advance our understanding and should be part of the armamentarium of every competent faculty member and faculty developer. But they get us only so far. We are academic trainees, academic faculty, and faculty developers because we need the eggs. We also need books on how to cultivate imaginary chickens.

Which brings me to Karen Kelsky’s next book. She writes (https://www.facebook.com/TheProfessorIsIn/posts/960887117291169):

It’s about the “Ph.D. Brain”: powerful, analytical, critical, skeptical, productive, logical, goal-oriented, but also obsessive, dismissive, self-critical, narrow, competitive, cynical and judgmental. The Ph.D. Brain is both wonderful and terrible, our best asset and worst enemy. With it, you end up with overdeveloped analytical skills, and underdeveloped intuition and self-care. Those of us who have it are seeing our native habitat collapse (the university, RIP) and we – both those on the tenure stream in the corporatized university, and those who never make it in – must learn how to thrive in a hostile environment. I’ll talk about academic productivity (and yes, getting tenure). But the larger gist is: in a post-apocalyptic world, how can you harness your Ph.D. aptitudes, and when necessary overcome them, to make your way forward in a healthy, balanced, financially secure way?

Very sketchy ToC:

I. Intro: The Endangered Ph.D. in a Post-Acapocalyptic World
II. The Ph.D. Brain: Greatest Asset or Worst Enemy?
III. Systems Under Stress: Productivity and Self-Care in a Contracting Academy
IV. How to Get Tenure Without Losing Your Mind
V. The Crux of the Matter: Healthy Productivity
VI. Learning to Value Yourself (and Get Paid)
VII. Activating Your Whole Mind

(… how our thinking and values have to change to survive and thrive as hyper-specialized species when our habitat is being razed.)

Welcome, cultural anthropologist, to the worlds of ecology, psychology, and decannomics. We look forward to your next book.

And she is right:  Her book/blog/service (and others like them on behalf of faculty) wouldn’t be necessary if we all did our jobs as developers of current and future faculty.

To-do list:

√  If you advise or develop others (including yourself), ask yourself: Have I fully disclosed to my client the extent of my expertise, what I am prepared to do (and not do) in the course of a relationship, and any conflicts of interest or commitment? Have I fully explored my client’s tolerance for pain and truth, and am I prepared to deliver these accordingly? If I cannot meet my client’s needs, have I disclosed this so that my client can make other arrangements? Have I clearly communicated the odds of success and failure? That is, have I obtained informed consent for what is about to ensue? If not, do so.

√ If you are advised or developed by others (or yourself), have these questions been answered for you? If not, ask them, find another advisor, or accept the consequences.

√   If you are involved in appointing faculty, consider Laszlo Bock’s observations, ask ‘is the way we’ve always done it the best we can do?’, and – if not – be a change agent.

√  Should we formally obtain informed consent from anyone who enters a PhD program?

[1] http://chronicle.com/article/To-Professors-Re-Your/129121/

[2] Kelsky, K. 2015. The Professor Is In: The Essential Guide to Turning Your Ph.D. Into a Job. Three Rivers Press, New York. http://theprofessorisin.com/buy-the-book/

[3] Cold Spring Harbor Laboratory Press, 2010. http://amzn.com/0879698667

[4] Springer, 2006. http://amzn.com/0387321764

[5] http://www.ascb.org/where-will-a-biology-phd-take-you/

[6] Karen Kelsey indicts graduate advisors as solely responsible for this delusion. I disagree; advisor and advisee are jointly responsible. Indeed, as she says, “unfortunately, PhD students are largely resistant to professionalization.” The subordinate status of the advisee is no excuse; Moreover, although advisor and advisee are clearly hierarchical, ‘managing upward’ is feasible and often required.

[7] Kahneman, Daniel. Thinking Fast and Slow. Farrar, Straus and Giroux. http://amzn.com/0374275637

[8] Bock, Laszlo. 2015. Work Rules! Insights from Inside Google That Will Transform How You Live and Lead, Twelve. http://amzn.com/1455554790

[9] Academic Scientists at Work, Second Edition, by Jeremy Boss and Susan Eckert. Springer, 2006. http://amzn.com/0387321764

[10] http://www.imdb.com/title/tt0075686/

©Martin E. Feder 2015

Permalink: https://decannomics.com/?p=566

Shortlink: http://wp.me/p5LUkN-98

Source[1]

Dear faculty member seeking promotion or funding:

First, a disclaimer.   This is not advice to deceive academic assessment. Do not “put lipstick on a pig”. Be certain to do your best to create a body of accomplishment so unambiguously strong and a narrative so compelling that promotion or funding will be a foregone conclusion. Seek advice from knowledgeable colleagues or the literature on how best to do this; this advice will not be re-summarized here.

Second, know your enemy! Even if you create such a body of accomplishment and narrative, the forces of predictable irrationality[2] will conspire to defeat you. Hopefully the peers who assess you will keep these forces at bay, and your peers are usually successful at this. But – leave as little to chance as possible.   Help your allies. Plan to disable predictable irrationality. The good news is that it is predictable. Because you can predict it, you can devise countermeasures to these forces. In short, razzle dazzle ’em.

PROLOGUE: FUNCTIONALLY DISFUNCTIONAL

An invisible hand is at work. When academics exhibit talent, judgment, wisdom, accomplishment, and stature, they are asked to do more (including to assess other academics).   The more they do, the more they are asked to do, and the busier they become. In particular, the more effective they are as peer reviewers, the more they are sought as peer reviewers. Bottom line: those who review you are likely to be very busy people. The busier they become, the more they need to work efficiently and delegate some of their work to others. How? They will take usually-reliable shortcuts (heuristics). Bottom line: create shortcuts, and put them in the path of those motivated to take them.[3]

ACT 1: BEING JOHN MALKOVICH[4]

We all suffer from egocentric bias, which in present circumstances means that we believe what makes sense to us makes sense to everyone else. We all suffer from fear of looking stupid. These two biases can combine to undermine the best of promotion cases: You think you’ve explained your work adequately, but you haven’t; ‘they’ either think ‘they’ understand your work, but they don’t – or ‘they’ say ‘the work is outside my area’, which is code for ‘I’m afraid to express a judgment for fear of looking stupid’.

Most likely your training did not include communications as a formal topic, so here’s the elevator version: Presuming you have done work worth supporting, it is your job to sell it to those who will assess it – strike that, true but too cynical – it is your job to make your work and its significance comprehensible to those who will assess it. In the real world, when the stakes are high (e.g., in political campaigns, advertising, cinema releases) there are exhaustive test-marketing, focus groups, and surveys in which alternative forms of messages are compared[5]. Shouldn’t you do the same?

My point is simply to be certain that your sales pitch is compelling. Because of egocentric bias, your judgment of what is compelling is likely to be flawed; never trust it. How, then, can one be certain?

• Study the masters (and the failures[6]). Every research presentation, lecture, elevator speech, textbook, teaching moment, etc. (and, for that matter, all political and sales campaigns) is an object lesson in what works and what doesn’t in a sales pitch.
• Find reliable and appropriate critics.   Reliable: Many potential critics will sugar-coat their criticisms, or will feign understanding to avoid looking stupid. Instead of them, rely on those who can be brutally honest. Appropriate: In job applications and promotions you will need to please two audiences. The first, content experts in your field, you probably already know how to please (and whether you’ve succeeded). The second is both less recognized and more important: those who are not content experts in your field. Except in unusual cases, the second audience will be involved in assessing your product. Therefore you also need critics who are non-experts (and probably the more non-expert, the better). Use them to perform the Vonnegut test: “…any scientist who couldn’t explain to an eight-year-old what he was doing was a charlatan.”[7] Unless she is a professor in your field, your mother is probably an ideal critic. Professional communications coaches can be very helpful.[8] Critics: That’s plural. Few messaging tactics are effective with all audience members (and those tactics are probably too offensive to use).  Test-market your sales pitch with sufficiently diverse critics until you are certain that it will be successful with the known or unknown ‘peer’ reviewers who will judge you.
• Allow sufficient time and effort to optimize your message. Many academics mistakenly believe that content sells products, and focus insufficiently on crafting a message that convinces consumers to consider their product. Given our meager training, successful messaging is likely to require many successive approximations and consultations, each taking time. Hint: Make each iteration comparative. That is, try communicating in two ways and ask your critics which is better? This defuses fear of looking stupid, and does not put your critics on the spot to deliver advice.

ACT 2: THE POWER OF BRANDS

Branding works because once a product’s brand comes to signify an attribute (e.g., high quality, high standards, reliability, good taste, etc. – or the opposite), consumers will favor or disfavor the product on the basis of its brand. Some academic assessors are consumers who will favor or disfavor the ‘products’ they assess in part on the basis of brands such as current or past institutional affiliations, the impact factor of the scholarly journal in which the work appears, an honorific award, or a particular type of funding. That is: If one’s short on time, just look at the brand and jump to the obvious conclusion.

If you have a choice of a brand for your personal academic product, choose the brand with the highest consumer approval. If you don’t know which scholarly institution, journal, honorific, or funding has the highest consumer approval, that’s what the Internet is for.

Having said this, do not ‘go overboard’. For example, some scientists will insist on repeatedly revising and resubmitting work to a particular journal in the belief that acceptance will guarantee tenure, and in so doing undermine their own productivity and squander the opportunity to place their work in equally appropriate journals. Perfect can be the enemy of good enough.[9]

ACT 3: THE MALLEABILITY OF CROWDS

People (and they include even faculty) are hardwired to follow other people, often for good reason. Running with (and not against) people fleeing danger normally enhances survival. In academic assessment, which is typically not done in private, the counterpart is agreement with majority opinion, dominant paradigms, and those perceived as leading scholars. A minority of you will create heterodox scholarship so compelling that it will overcome these prejudices. But if you are not in this minority, you should anticipate groupthink and manage it. Malcolm Gladwell’s The Tipping Point[10] is an entertaining popular account of how this is done in the real world. In your world, it is prudent to count on several factors:

• Deference to “experts”. In biomedicine, these are members of the National Academy of Sciences, investigators of the Howard Hughes Medical Institute, named chairs at prestigious institutions, and so on. A few minutes’ thought should yield the names of those in your area. Chances are: confidential assessments will be solicited from them when your promotion is under consideration, and your institution will be hard-pressed to ignore overwhelmingly positive and enthusiastic endorsements from them. It is much easier for your ‘experts’ to provide such endorsements if they know and respect you personally. How?   (a) Although the rule is to avoid committee service while a probationary faculty member, the one committee you want to be on is the one that organizes departmental colloquia or seminars. Ask to join it, and use it as an opportunity to invite your ‘experts’ to speak. [Be patient: Some of the big acts book up years in advance.] When they arrive, host them, introduce them, wine them, and dine them[11]. Show them a good time – but do talk some research, including yours. This will also associate them with you in the minds of your colleagues, and legitimize your area of research. (b) Go to professional meetings and conference they attend, and make their acquaintance. Offer them coffee, see if you can join them for a meal, and/or sit next to them on the bus[11.5].
• Deference to brands. If your curriculum vitae includes many invitations from brands your assessors respect, they will be impressed. You can suggest to a prestigious journal that it invite you to produce a review, editorial, or thought piece. You can suggest to a program officer that you be invited to serve as a peer reviewer for a prestigious funding organization. You can suggest to a significant professional society that you be invited to serve in some visible capacity or stand for election as an officer. The worst they can say is ‘no’ and, if they do, no one will know. There is also what the politicians call logrolling; it works like this: You befriend someone at a prestigious institution. You say: If I invite you to give a talk at my institution, will you invite me to give a talk at yours? You put the talk on your CV.   Or, at a professional meeting you run into someone from a prestigious institution and mention: “I’ve always wanted to visit your institution…”
• Deference to leaders at your own institution. Typically the people who assess you will be senior to you and have greater institutional stature. Treat them as you’d like to be treated; the rest is commentary.   Your institution may instruct them to set behavior aside and assess only your work; they may well be unable to do this, for they are only human. And when they speak, they’ll be heard.
• Deference to paradigms. As implied above, our minds are biased to prefer what confirms our pre-existing knowledge and disfavor what challenges it. The more your work challenges existing paradigms, the more your patrons and assessors will be averse to endorsing it. If your academic work can become the existing paradigm in time to be recognized as such, you’re in excellent shape! But: “You come at the king, you best not miss.”[12] It may be prudent to postpone a paradigm shift work in progress to after tenure – except if this leads you to be scooped. [In that case, publish away and it may then be necessary for you to claim your named chair at another institution.]

In summary, expect groupthink, and at least do not create conditions that will lead groupthink to oppose you.

EPILOGUE:

You may complain that the activities I suggest shouldn’t be necessary. OK, you’re right. Feel better now?

You may complain that the activities I suggest aren’t appropriate. I disagree. Know the 3 Ps of academics[13]? Academics produce knowledge that is

• Public
• Peer-reviewed
• Platform on which others can build

If a tree falls in the middle of the forest and there’s no one around to hear it, does it make a sound? If an academic produces work that is unknown to or unrecognized by peers, is it an academic product? Product yes – academic no. These days we all compete for attention; academic products far outnumber our ability to notice them all. Nonetheless, many of us assume that because we are academics, we can rationally assess and compare all relevant products such that the cream will rise to the top automatically. Believe this at your peril!   If perchance the manipulations I suggest are un-necessary, you will at least have had a good time in their execution.

Finally, the largest obstacle to successfully following my advice is likely you yourself; i.e., your own cognitive biases and tendencies. Stage fright is difficult to overcome. Brutally honest advice from critics can hurt. Aiming for the best brands, the biggest names as patrons, and the most prestigious of invitations risks rejection. These all can be unpleasant – but far less than rejection of your manuscript, grant application, or promotion case. Nonetheless, our minds lead us to avoid present pain and risk even when this sets the stage for much greater future pain and risk. Fortunately, as detailed in prior posts, there are ways to overcome the aversions that can impede our advancement.

Good luck! Give ’em the old razzle dazzle. And once you are promoted, work to make all this un-necessary.

[1]  Razzle Dazzle.   From Chicago (1975).   Music by John Kander, lyrics by Fred Ebb. https://en.wikipedia.org/wiki/Chicago_(musical)  Lyrics: http://www.stlyrics.com/lyrics/chicago/razzledazzle.htm  http://amzn.com/B006RXQ1L6

[2] Ariely, Dan. 2008. Predictably Irrational. HarperCollins.

[3] This is a version of BJ Fogg’s first law of captology: Put hot triggers in front of motivated people.

[4] A film in which the principal plot device is the ability to look at the world through another’s (John Malkovich’s) eyes: Being John Malkovich. 1999. Polygram USA Video. http://amzn.com/6305807086

[5] http://www.winningcampaigns.org/Winning-Campaigns-Archive-Articles/Polls-Focus-Groups-in-Political-Campaigns.html https://en.wikipedia.org/wiki/Focus_group https://en.wikipedia.org/wiki/Test_market

[6] https://www.youtube.com/watch?v=uhiCFdWeQfA

[7] https://en.wikiquote.org/wiki/Cat%27s_Cradle

[8] For example: https://www.linkedin.com/pub/julie-peterson/3/976/96a

[9] After Voltaire. https://en.wikipedia.org/wiki/Perfect_is_the_enemy_of_good

[10] Gladwell, M. 2000. The Tipping Point. Gladwell. http://amzn.com/0316346624

[11] I tried for years without success to get a distinguished professor to speak at my institution. What finally worked was when he mentioned that he always wanted to dine at Charlie Trotter’s, then the leading restaurant in my city. I told him: “I can do that.” Once the arrangements were made, it turned out that they also included a visit to my institution.

[11.5] Karen Kelsey offers additional instructions in Chapter 20, “How to Work the Conference”, of The Professor Is In, Three Rivers Press, 2015.  http://amzn.com/0553419420  See also: http://theprofessorisin.com/?s=how+to+work+the+conference&x=0&y=0

[12] https://en.wikiquote.org/wiki/The_Wire

[13] Attributed by   Simpson D, Fincher RM, Hafler JP, Irby DM, Richards BF, Rosenfeld GC, Viggiano TR. Advancing Educators and Education: Defining the Components and Evidence of Educational Scholarship. Proceedings from the Association of American Medical Colleges Group on Educational Affairs Consensus Conference on Educational Scholarship, 9-10 February 2006, Charlotte, NC. Washington DC: AAMC 2007, to Hutchings P, Shulman LS. 1999. The scholarship of teaching: New elaborations, new developments. Change 31:10–15.

Permalink: https://decannomics.com/?p=543

Shortlink: http://wp.me/p5LUkN-8L

©Martin E. Feder 2015

Countless books and websites advise on ‘how to advance in academia’. Often their advice is good, necessary, important, and well worth following. On the other side is an enormous written and unwritten body of work on how to perform academic assessment[1], and each institution’s process is elaborated often in excruciating detail. By and large, however, this work ignores a critical factor in academic decision-making:

People assess the academic work and make the decisions, and the assessments and decisions are not private. Not private means that the assessments and decisions can or will be shared with other people.

Ideally the academic work is so good or so poor that the summary judgment is a foregone conclusion and immune to the issues I’ll describe. Sound advice is: strive to be that good if you are a candidate, or assess only cases that good or bad if you are a judge or juror. The vast majority of instances are in between, however. For them, if you are insistent on excellence in academic performance, either as a performer or a judge, sound advice is: Know your enemy![2]

Who’s your enemy? As developed elsewhere, he is us[3]. Academic promotion involves people judging people. While human minds are capable of dispassionate, consistent, and fair assessment and decision-making in academic promotion, they are also irrational – and that irrationality can bedevil academic promotion. Importantly, however, the irrationality is predictable[4]. As such, it can be avoided if need be, or intentionally triggered if need be.   “If you know the enemy…you need not fear the result.”[5]

If you were a wealthy defendant or litigant in a legal proceeding or a legal judge seeking due process, you might well hire consultants to cope with the cognitive psychology of the participants. As an academic assessor or assessee, you are probably on your own. Nonetheless, as Yogi Berra said, “You can see a lot by just looking.”[6] The next few posts in this series (see below) will get you started. Legal psychology is an open book online, as is behavioral economics, cognitive psychology, social psychology, and cognitive bias. Just look. If you’re smart enough to be an academic, you’re smart enough to understand this.

But will you? Answering this question is itself prone to cognitive bias, so you may not.   But cognitive tendencies can be restrained. It’s up to you.

In this series:

• No matter how much ‘they’ seem like ‘us’, ‘they’ aren’t; design academic assessment and faculty development programs and communications in anticipation of this reality [The Dancing Fool Meets John Malkovich].
• An enlightened attitude towards scholarship does no good unless the gatekeepers of publication, funding, and academic appointment and promotion buy in to it [Curve Ball].
• Our normal behavior can interfere with high-quality peer review. Devise countermeasures. [Fishy behavior]
• Cognitive biases can influence academic peer review, more so when review is done in groups. Even if this is rare, prudence dictates we anticipate it and combat it. [The Jury Is Still Out]
• To faculty seeking promotion: Use the power of cognitive bias for good, not for evil [Give ’em the old razzle dazzle]

[1] For example, see http://www.aaup.org/sites/default/files/files/Good%20Practice%20in%20Tenure%20Evaluation.pdf

[2] Sun Tzu, The Art of War http://www.artofwarquotes.com

[3] http://en.wikiquote.org/wiki/Walt_Kelly

[4] Ariely, Dan. 2008. Predictably Irrational. HarperCollins.

[5] Sun Tzu, The Art of War http://www.artofwarquotes.com

[6] http://www.quoteworld.org/quotes/12139

©Martin E. Feder 2015

Permalink: https://decannomics.com/?p=515

Shortlink: http://wp.me/p5LUkN-8j

FOREMAN. . . . Nine . . . ten . . . eleven . . . That’s eleven for guilty. Okay. Not guilty? (EIGHT’s hand is raised.) One. Right. Okay. Eleven to one, guilty. Now we know where we are.
THREE. Somebody’s in left field. (To EIGHT) You think he’s not guilty?
EIGHT (quietly). I don’t know.
THREE. I never saw a guiltier man in my life. You sat right in court and heard the same thing I did. The man’s a dangerous killer. You could see it.
EIGHT. He’s nineteen years old.
THREE. That’s old enough. He knifed his own father. Four inches into the chest. An innocent nineteen-year-old kid. They proved it a dozen different ways. Do you want me to list them?
EIGHT. No.
TEN (to EIGHT). Well, do you believe his story?
EIGHT. I don’t know whether I believe it or not. Maybe I don’t.
SEVEN. So what’d you vote not guilty for?
EIGHT. There were eleven votes for guilty. It’s not so easy for me to raise my hand and send a boy off to die without talking about it first.
SEVEN. Who says it’s easy for me?
EIGHT. No one.
SEVEN. What, just because I voted fast? I think the guy’s guilty. You couldn’t change my mind if you talked for a hundred years.
EIGHT. I don’t want to change your mind. I just want to talk for a while.
Twelve Angry Men[1]

We operate under a jury system in this country, and as much as we complain about it, we have to admit that we know of no better system, except possibly flipping a coin.[2]

Generally speaking, we can observe that the scientists in any particular institutional and political setting move as a flock, reserving their controversies and particular originalities for matters that do not call into question the fundamental system of biases they share.[3]

When legal juries convene, judges and attorneys are (or, at least should be[4]) highly aware of the cognitive biases that affect rational decision-making. Competent attorneys seek to exacerbate or mitigate these biases to benefit their clients, sometimes aided by jury consultants, and competent judges strive for due process. These are highly skilled professionals who have developed considerable knowledge [just search online for ‘jury decision making psychology’, ‘litigation psychology’, ‘legal psychology’, and so on].

When academic juries convene – as in review of funding applications or promotion cases – the deliberation is often uninformed by any consideration of possible cognitive bias. To the contrary, the jurors assume that – because they are academic faculty, having been selected for their training, accomplishment, and critical ability – they are capable of completely rational decision-making and immune to cognitive biases. To accuse an academic peer reviewer of bias is a serious charge indeed. As for knowledge about the impact of cognitive biases in peer review, aside from implicit bias with respect to the typical axes of diversity[5], scholarly works are few and far between. van Arensbergen et al. is the best recent review[6].

What do we know about group decision-making, such as in legal juries, that might[7] also interfere with academic peer review, and what might be done about it?

In both cases, the ideal is consistently to apply a standard (the law, academic contribution and promise) to data[8] in the same way that trained graders of gemstones or judges of animal breeds can in isolation arrive at identical assessments. In both cases we stray far from the ideal because (a) we are judging people, and (b) the jurors are people, and the judging is not private.

‘Judging people’ means that, because no two people are the same, the data under consideration are never identical (cf. ‘comparing apples and oranges’). As the Physics Department at MIT correctly states, “ There are as many different successful paths to tenure as there are tenured faculty members…”[9]

‘The jurors are people’ means that they have normal human minds, which often jump to conclusions in predictable ways (or, in techspeak, use heuristics). Often these jumps to conclusions are useful shortcuts, but they can bias assessments in characteristic ways, including:

• Framing. The way in which information is presented influences assessment. For example, “70% of her papers appeared in high-impact journals” and “30% of her papers did not appear in high-impact journals” are quantitatively identical, but the latter is interpreted more negatively than the former.

• Anchoring (‘if it costs that much, it must be good’). A form of framing in which the first information given influences the interpretation of subsequent information. In real estate or automobile sales, if the asking price or list price is given first, subsequent negotiations will proceed from this price rather than a true estimate of value. If peer reviewers first learn that a candidate is highly/poorly cited, a full professor vs. a lower rank, from a prestigious vs. second-tier institution, the assessment can differ accordingly.

• Hindsight bias (‘anybody could have seen it coming’): For both promotion and funding applications, reviewers may alter their assessment according to whether flaws were/are foreseeable (and can be anticipated if not avoided or prevented), or not. We are biased to overestimate foresight and blame others for not foreseeing what, in hindsight, appears obvious (cf. ‘Monday morning quarterbacking).

• Confirmation bias (‘my mind is made up; don’t confuse me with the facts’): We overemphasize data that confirm preliminary conclusions, and tend to ignore data that contradict them.[10]

• Egocentric bias (‘I get it; therefore everyone else should’). We assume what is comprehensible, easy, or obvious to us is comprehensible, easy, or obvious to others. Therefore we neglect to communicate appropriately to others, who are baffled or reach wrong conclusions. [This is more a problem for those being assessed than those doing the assessing.]

• Recency bias or Primacy effect (‘but that was then’): in assessment, we tend to overweight recent accomplishments and information, and underweight prior accomplishments and information, all else equal.

• Presentation bias (looks good, therefore is good; looks bad, therefore is bad)[11]: As Chia-Jung Tsay begins in her PNAS paper: “We do judge books by their covers. We prefer the nicely wrapped holiday gifts, fall in love at first sight, and vote for the politician who looks most competent.”[12]   The way in which academic accomplishments are presented can influence our assessment of our merit.

Not private, as in ‘The jurors are people, and the judging is not private’ means that judges and jurors are just as much on trial as those being assessed [well, maybe not as much, because the judges and jurors have typically already been promoted, but on trial nonetheless.]   The academic judges and jurors are under pressure to seem fair, reasonable, knowledgeable, critical, and deferential to high academic standards in the eyes of their colleagues. They are members of communities in which stature and access to resources depend in part on their perceptions by other members. They are automatically and naturally susceptible to implicit biases, snobbery, envy, and a desire for retribution, but often must behave before their colleagues as if these tendencies don’t exist.   These pressures can deform academic due process and judgment. For example:

A man walks into a room. Seven others are already seated there, and all are told they are to engage in a judgment. They are shown two sets of lines

and asked: “Which line on the right is the same as the one on the left?”   All give the correct answer. The judgment is repeated several times with different sets of lines until it is routine. Then, by prearrangement, the seven already in the room start giving a conspicuously wrong answer. At first, the man disagrees. Eventually, however, the man joins the majority in giving the wrong answer.

This is the famous Asch experiment[13]. 75% of subjects agree to at least one wrong answer.[14]

Our minds are highly attentive to what others think and do, might think and do, and might think of us and do to us. This tendency can exacerbate existing biases or introduce additional biases when peer review is not completely private. These include:

• Anchoring is described above.       In any group, typically the first statement made anchors the discussion. That is, if the first speaker supports or opposes promotion/funding, the ensuing discussion ensues in relation to this first statement. If on the old NIH rating scale the first reviewer proposes a score of 2.0, others might disagree by proposing scores of 1.7 to 2.3. If the first reviewer had instead proposed 3.0, others might disagree with scores of 2.7 to 3.3.   Many promotion reviews occur in multiple steps (e.g., department, promotions committee, dean, and provost); the early steps tend to anchor the later steps.

• Social Influence/Groupthink/The Bandwagon Effect/The Abilene Paradox: The desire for concurrence, consensus, and cohesion in a group can temper or even overwhelm independent assessment. That is, participants in peer review will alter their independent assessments so as to conform to a majority opinion.[15]   This undermines ‘the wisdom of crowds’. As in the Asch experiment, those holding a minority judgment can be tempted to mute or abandon it because they are in minority rather than because the judgment is incorrect. Those in the majority can be overly resistant to challenges to their conclusion and prone to confirmation bias.

• Social anxiety; fear of displaying stupidity, low standards, limited expertise, poor judgment, lack of confidence. Such displays undermine status in academia, and with it access to resources and resistance to threats. When their judgments can become known, assessors may express no judgment or pretend to agree with the majority or seemingly assertive, authoritative assessors to avoid ‘looking bad’ in the eyes of their peers.

Admittedly these impacts are largely extrapolated to academic assessment from work in other contexts and, as noted, studies of academic peer review are few. These works suggest that extrapolation is not unwarranted, however.[16]

What can be done to minimize the influence of these cognitive biases and tendencies?

√ Education. As has been stated, in academia reputation and stature are linked to the demonstration of sound judgment and wisdom free of bias, and simply knowing that fellow judges are on the lookout for bias ought to deter it. A precondition is that the fellow judges must know what biases to look for, and individuals must know what biases to avoid displaying.

√ Leadership of peer review groups. Those who chair such groups should be painfully aware that, for example, those who speak first and those who speak last will be influential (serial position effect), that an assertive high-status person who speaks first will anchor the discussion, that social anxiety will squelch valid points of view, and so on. Chairs can routinely ask: “How may cognitive biases and tendencies have influenced our judgment in inappropriate ways?” [If this is routine, it is not an accusation – and accusations can backfire.]   At my institution chairs are coached to call on committee members in random order and to call on members who are reticent to speak spontaneously. [This has a collateral benefit of inducing all members to study each case carefully and discourages social loafing.]

√ Avoid framing and anchoring. In an ideal world, all framing (identities and affiliations of the candidate, authorship position and co-authors, titles of publications outlets, honorifics, etc.) would be redacted from review materials so that reviewers would be forced to focus on the unadorned work itself. This is likely impossible [although at my institution one ambitious department did its initial screen of applicants for a position by considering only the abstracts of exemplary publications – with bibliographic and other identifying information redacted]. But some framing/anchoring can be withheld. For example, in promotion discussions at my institution we used to begin with the de-identified preliminary scoring of the members; we no longer do.

√ Empower all reviewers. Many disasters are due to groups not working together and members’ reticence to express concerning observations to their superiors. To combat this in aviation, crews practice cockpit resource management or CRM.

CRM aims to foster a climate or culture where the freedom to respectfully question authority is encouraged. It recognizes that a discrepancy between what is happening and what should be happening is often the first indicator that an error is occurring. This is a delicate subject for many organizations, especially ones with traditional hierarchies, so appropriate communication techniques must be taught to supervisors and their subordinates, so that supervisors understand that the questioning of authority need not be threatening, and subordinates understand the correct way to question orders. These are often difficult skills to master, as they may require significant changes in personal habits, interpersonal dynamics, and organizational culture.

The way this might work in peer review is:

• Opening or attention getter – Address the individual. “Hey Chair,” or “Professor Smith,” or “Jane,” or however the name or title that will get the person’s attention.
• State your concern – Express your analysis of the situation in a direct manner while owning your emotions about it. “I’m concerned that we are being unfair to this candidate,” or “I’m worried that the discussion is going off track.”
• State the problem as you see it – “We are overemphasizing the letters of reference, which are subject to implicit bias,” or “We are being unduly influenced because the paper is in Science, and are not fully examining its contribution.”
• State a solution – “Let’s look carefully at the adjectives in the letters,” or “Can someone explain why the work in Science is a significant contribution?”
• Obtain agreement (or buy-in) – “Does that sound good to you, Ms. Chairperson?”

√ Voting: Always vote by private if not secret ballot or its equivalent. The rationale is obvious.

Conclusion

Cognitive bias in peer assessment hopefully is rare and of little influence. Prudence dictates that we expect it, combat it, and are pleasantly surprised when it doesn’t occur. The only always ineffective strategy is to pretend it doesn’t exist.

[1] Rose, Reginald. 1955. Twelve Angry Men. http://amzn.com/1417812656 https://docs.google.com/document/d/1irVXTuMAQESSwtoqOtQiC_-5dZa59LCmOxA_IQzlxww/edit

[2] Dave Barry. http://articles.chicagotribune.com/1996-01-07/features/9601070283_1_reasonable-doubt-lawyers-jury-system

[3] Gunnar Myrdal, Objectivity in Social Research, as cited in Klein, DB and Stern, C. 2009. Groupthink in academia: majoritarian departmental politics and the professional period. Independent Review 13: 585-600.

[4] Benforado, A. 2015. Unfair: The New Science of Criminal Justice. Crown Publishers. http://amzn.com/0770437761

[5] An outstanding source is http://kirwaninstitute.osu.edu/wp-content/uploads/2014/03/2014-implicit-bias.pdf

[6] van Arensbergen, P, van der Weijden, I, and van den Besselaar, P. 2014. The selection of talent as a group process. A literature review on the social dynamics of decision making in grant panels. Research Evaluation 23: 298-311. doi: 10.1093/reseval/rvu017   http://www.vandenbesselaar.net/_pdf/2014%20Prpic.pdf

[7] “Might”≠”will”. In my experience, the vast majority of peer assessment leads to sound judgment. This does not excuse us from increasing the size of this majority and reducing the incidence of misjudgments rooted in cognitive biases, etc., however. Informing ourselves about these biases is a necessary first step.   My institution is committed to reducing if not eliminating the influence of cognitive bias on promotion decisions.

[8] http://www.aaup.org/sites/default/files/files/Good%20Practice%20in%20Tenure%20Evaluation.pdf

[9] http://web.mit.edu/physics/policies/dept/AdviceForNewFaculty.pdf

[10] The stages of truth:

• It’s not possible
• It’s possible but either impossible to test or not worth doing
• It’s obvious and we knew it all along

https://cs.uwaterloo.ca/~shallit/Papers/stages.pdf is an exhaustive and masterful review of the concept. Confirmation bias is the third stage.

[11] Although there must be a pre-existing name for this bias in the literature, I do not know it.

[12]   Tsay, Chia-Jung. 2013. Sight over sound in the judgment of music performance.   Proceedings of the National Academy of Sciences USA 110: 14580-14585. More general discussion in http://www.theatlantic.com/business/archive/2013/09/the-science-of-snobbery-how-were-duped-into-thinking-fancy-things-are-better/279571/

[13] https://en.wikipedia.org/wiki/Asch_conformity_experiments. The Candid Camera counterpart is at https://vimeo.com/61349466

[14] Perceptive readers will recognize this statement as an example of framing and anchoring. Humans are not that susceptible to normative social influence, but are susceptible nonetheless.

[15] https://en.wikipedia.org/wiki/Groupthink
https://en.wikipedia.org/wiki/Abilene_paradox
https://en.wikipedia.org/wiki/Conformity
https://en.wikipedia.org/wiki/Social_influence
https://en.wikipedia.org/wiki/Bandwagon_effect

[16] Some quotes from van Arensbergen et al. (endnote 5; citations removed):

“…when group members know who the experts are in reference to a specific task, they will adjust their group decision to the decision of the experts…   …in general high-status members talk more and receive more attention from other members. Low status members generally talk less or even do not talk at all when their opinions deviate from those of high-status members. This can harm decision making processes because not all true opinions are expressed and high-status people will not be contradicted often… members of cohesive groups may want to preserve the group’s relationships and therefore avoid any kind of behavior considered to be harmful. This could mean that people agree to group decisions while they actually do not agree with it individually… …people paid more attention to preference-consistent information than to information that conflicted with their preferences. This effect was even stronger when confirming information was introduced by the person himself than by other group members. Whether people adjust their initial preference based on new information that is contributed to the discussion is strongly influenced by social validation. …people defend their initial preference and in order to convince others they mention more information that supports their preference.   But it can also be the result of more unconscious processes: people consider preference-consistent information as more accurate and relevant and therefore pay more attention to it. …panelists may use different strategies or social tactics in processes of decision making, e.g. consultation, pressure, personal appeals, and coalition tactics. The use of social tactics to influence one another is affected by status differences. …Often, opening statements serve as point of reference for all statements being made thereafter. With regard to panel review, this implies the comments of the first reviewer are very influential and set the tone for further discussion. Knowledge of these cognitive heuristics can be implemented as social tactics when panelists actively use them to influence negotiation outcomes.   …Groupthink is more likely to occur in groups where any degree of accountability is absent. Making individuals accountable is found to be more effective on reducing groupthink tendencies than making them collectively accountable as a panel…   …The combination of the large scope of applications to be evaluated and the restricted time available reduces the ambitions of panelists to execute very rigorous reviews. When panels experience strong time pressure, reviewers pay more attention to shared information and less attention to alternatives, consequently resulting in a closing of the mind. People tend to rely more on cognitive heuristics …and are more focused on reaching (cognitive) closure Therefore, high time pressure is considered an important antecedent for groupthink .”

Klein and Stern (see endnote 3) treat very generally how groupthink may devalue dissenting views, even if well-founded, in academic units. Langfeldt shows that peer review groups may autonomously establish priorities or practices that influence outcomes, or as she puts it “The guidelines given to the panels had little effect on the criteria they emphasized… Put more clearly, panels do as they like…” Recently Park et al. have modeled how peer pressure in review – herding – may stifle innovation (or penalize innovators).

Langfeldt, L. (2001). The decision-making constraints and processes of grant peer review, and their effects on the review outcome. Social Studies of Science, 31(6): 820-841.

Park IU, Peacey MW, Munafò MR. 2014. Modelling the effects of subjective and objective decision making in scientific peer review. Nature 506:93-6. doi: 10.1038/nature12786.

Permalink: https://decannomics.com/2015/08/01/18-the-jury-is-still-out/

Shortlink: http://wp.me/p5LUkN-80

©Martin E. Feder 2015

Stanley Fish has been around the academic block more than most of us.  During and after service as a dean at the University of Illinois Chicago he wrote a wonderful column for the Chronicle of Higher Education, “All in the Game”, in part morality tales about academic faculty behaving badly.  A favorite, entitled “Somebody Back There Didn’t Like Me“[1], speaks of senior faculty whose work was found wanting at an earlier stage and thereafter “spend a lifetime trying to wash away the taint by being … ungenerous to others …invoking standards much more rigid than the standards they know you think they didn’t quite meet…”[2]

It has been my privilege to observe and participate in peer review of manuscripts, grant applications, and people.  For the most part, it is an effective process for allowing ‘the cream to rise to the top‘, and a demonstration of the wisdom of crowds in action.  My fellow reviewers have been, for the most part, diligent, insightful, and fair.  Indeed, for the vast majority of peer review meetings I’ve attended, those being reviewed could have listened to an anonymized recording and concluded that justice was done.  For the most part…

Every so often, however, peer review goes horribly awry, and can resume its appropriate course only with great effort.  In addition to the behavior Fish describes, reviewers play out political agendas in the guise of peer review, and compete to see who can be the most devastatingly critical.[3]

Why fishy[4] behavior?

Peer reviewers are people first and distinguished academic faculty second.  As people, they are always of two minds, one driven by threat and opportunity and jumping to conclusions, and the other capable of rational but slow deliberation.  Importantly, while only the second (“System 2”) can perform appropriate peer review, System 2 can undertake only what the first (“System 1”) will allow:

• In academia, vulnerability to threat and access to opportunities and resources, while ideally only a function of academic merit, is also a function of stature and reputation.  System 1 promotes competition for stature and reputation. Peer review is an opportunity to augment or consolidate stature and reputation, and this opportunity if taken may trump thoughtful deliberation.
• System 1 is attentive to what others in a group think and do.  In peer review, System 1 is highly averse to appearing stupid, ignorant, or wrong — sometimes leading to silence when challenges should be raised if not confirmation bias or groupthink. Innovation or activity outside the cultural norm can be disrespected for this reason regardless of its merit (see ingroup-outgroup bias).
• “Fairness” is a key value of academic faculty.    ‘Everyone else had to go through this so it is only fair to continue it’ is a powerful rationale for the perpetuation of bad practice unrelated to actual academic merit.  As with hazing, this consolidates group coherence and social stature, which System 1 values.
• Academic faculty also afford stature to those with high standards and critical ability[5], and System 1-driven competition for stature can game this by elevating standards to unrealistic levels[6] and through excessive criticism

In rare[7] cases these tendencies can result in fishy behavior. Three primary manifestations are snobbery, envy and retribution. They are natural and automatic.[8]

Snobbery is the unjustified belief that one (and/or one’s tastes, accomplishments, stature, importance) is superior to others (see illusory superiority). If everyone is above average or anyone can equal the snob, the belief is refuted. Therefore, the snob can support the belief by denying others what is believed to make the snob superior.

Envy is “a painful emotion of wanting an advantage that another has and/or wishing the other did not have it or, simply put, ‘pain at another’s good fortune’”.[9]   In peer review, one (but not the only) form is envy of having come by success ‘the easy way’; i.e., without the trials and tribulations to which the envier was subjected.[10]

Retribution is a tendency to punish others “because they deserve it” as an end in itself. [It differs from utilitarian punishment intended to deter future bad behavior. These are not mutually exclusive.][11]

On these bases, peer reviewers may oppose, cheat, or even intimidate or sabotage those being reviewed.[12] Research shows that even when a group member’s own situation is fixed (e.g., full professors who have already been promoted and cannot be promoted to a higher rank), members will undermine the assessment of others or seek to inflate their own ranking.[13]

The impact of these rare but fishy behaviors can be enormous. When they lead to the denial of funding, publication, or appointment/reappointment/promotion/tenure to worthy candidates, those candidates may abandon worthy projects, experience stress disorders, leave the institution, or henceforth underperform and/or engage in these fishy behaviors themselves[14]. This behavior is also corrosive to those who engage in it.[15] When peer review loses its credibility as an objective arbiter of academic worth, the credibility is difficult or impossible to regain. That is, academia is damaged.

What to do?

As noted, these bad behaviors are natural and automatic. No one is immune to them. Rather than pretend they don’t exist, devise countermeasures.

√  One of System 1’s greatest strengths (the ability to take quick action based on the behavior of other people) can also be used to subdue it. As noted, System 1 is enormously deferential to what other people think. It will go out of its way to avoid embarrassment and public shaming. When it knows others are the lookout for bad behaviors that are the moral equivalent of academic fraud, it will do its best to avoid creating a pretext for accusation. I suggest that every peer review meeting begin with the chair stating: “All of us need to scrutinize one another for snobbery, envy, retribution, implicit bias, and so on, and so perform rational and objective assessment. We will be on guard.” [16]

√  Use circuit-breakers and pauses. Leaders of peer review committees can systematically pause discussions and ask: “Is there any possibility that unconscious behaviors are distorting our assessment?” Or, better still, “How have unconscious behaviors distorted our assessment?”[17]

System 2 is quite competent to conduct high-quality peer review when it is free to do so.

[18]

[1] http://chronicle.com/article/Somebody-Back-There-Didnt/46188/

[2] For a scholarly treatment of how past events may provoke such responses, see: Veiga, JF, Baldrige, DC, Markoczy, L. 2014. Toward greater understanding of the pernicious effects of workplace envy. Int. J. Human Resource Management 25: 2364-2381

[3] If you ever have an opportunity to see the CRTL Players enact a peer review session (http://www.crlt.umich.edu/crltplayers/fence) , take it! If you can’t, another depiction is at https://www.youtube.com/watch?v=H0_EHJN_TDs . A shout-out to the Purdue ADVANCE program for producing the latter.

[4] http://www.merriam-webster.com/dictionary/fishy , second definition

[5] In my institution’s lore, for example, is the following quote:

Chicago has developed a celebrated—some would say notorious—brand of academic civility. It is a place where one is always in principle allowed to pose the hardest question possible—of a student, a teacher, or a colleague—and feel entitled to expect gratitude rather than resentment for one’s effort. The trait is frequently noted (not always approvingly) by scholars from other institutions who visit us. …When Weber wrote about the scholar’s obsession with devil’s advocacy, he could have been talking about the University of Chicago.

[Report of the Faculty Committee for a Year of Reflection. The University of Chicago Record 32: 2-13, 1998.]

[6] In institutions is usually a tension between standards and pragmatism. On the one hand, the higher the standard, the better the personnel and the higher the quality of the product, all else equal. Personnel who don’t meet the standard can be turned away. On the other hand, regardless of the standards you ‘need to field a team’ [in academia, teach students; in academic medicine, take care of patients]. This can lead either to the acceptance of personnel who don’t meet the standard and/or can be recruited inexpensively, or default on responsibilities. Some institutions resolve this tension better than others (cf. record of championships for New England Patriots vs. Chicago Cubs).

[7] I do not wish to exaggerate the frequency of such instances. They are rare. We should nonetheless strive to reduce their incidence to zero, which is the rationale for this post.

[8] Menon, Tanya, and L. Thompson. 2010. Envy at work. Harvard Business Review. https://hbr.org/2010/04/envy-at-work

[9] See footnote 2, and references cited therein.

[10] Bokonon tells us: “Beware of the man who works hard to learn something, learns it, and finds himself no wiser than before. He is full of murderous resentment of people who are ignorant without having come by their ignorance the hard way.” Vonnegut, K. 1963. Cat’s Cradle. Holt, Rinehart and Winston. http://amzn.com/038533348X

[11] Tan, Fangfan and E. Xiao. 2014. Third-party punishment: retribution or deterrence? Max Planck Institute for Tax Law and Public Finance Working Paper 2014-05. http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2425522

[12] See footnote 8 and footnote 11

[13]Charness, G, Masclet, D, Villeval, MC. 2012. The dark side of competition for status. University of California at Santa Barbara, Economics Working Paper Series. http://escholarship.org/uc/item/1vr4g446

[14] A prior post (https://decannomics.com/2015/06/26/13-yes-and/) quoted WH Auden (http://www.poets.org/poetsorg/poem/september-1-1939):

“I and the public know
What all schoolchildren learn,
Those to whom evil is done
Do evil in return.”

[15] See footnote 8

[16] This is the equivalent of posting eyes above an ‘honor box’ where one voluntarily pays for coffee, a newspaper, etc. https://www.newscientist.com/article/dn9424-big-brother-eyes-make-us-act-more-honestly/   See: Bateson M., Nettles D., Roberts G. 2006 Cues of being watched enhance cooperation in real-world setting. Biol. Lett. 2, 412–414.

[17] As noted in the footnotes to https://decannomics.com/2015/07/19/16-curve-ball/ , a a former Provost at my institution would routinely ask in tenure cases: Has the implicit bias in letters of assessment affected judgment?

[18] I am grateful to Tanya Menon and Fangfan Tan for correspondence on this topic, and to Tanya Menon for sharing her unpublished work.