01 Mar 2016
Research Brief: The Benefits of Biasby Erin PetersonTopics:
Every year, specially chosen committees help the US National Institutes of Health (NIH) decide how to allocate massive—totaling more than $24 billion in 2014—competitive federal grants for medical research. But the evaluators on the various committees come from disparate fields and bring highly specialized expertise to the process—which some worry might create subject-area bias in the decision-making process and affect the quality of research.
Assistant Professor Danielle Li takes a mathematical approach to examining this issue in a recent HBS working paper, crunching data from nearly 100,000 NIH grant applications from 1992 to 2005. She plotted links between applicants and evaluators—via professional publication citations—to determine potential bias. Then, to establish the quality of the funded projects, she tracked the number of publications and citations that the research produced over time.
Her findings indicate that expert evaluators were indeed biased in favor of projects in their own area, which increased the application’s funding chances by 3.2 percent. But there was an unexpected benefit to their bias. Evaluators’ expert ability to separate the exceptional proposals from the lackluster ones led to projects with a measurably larger academic impact than projects selected without significant specialized expert input. The takeaway was clear, says Li. “Strong conflict-of-interest policies to reduce bias come at the expense of reduced quality of the experts you are actually consulting. And that [leads to] funding research that is worse.”
While Li’s paper doesn’t advocate doing away with conflict-of-interest policies entirely, she says it may be acceptable—in some contexts—to loosen the reins. “You don’t want to eliminate all conflicts of interest, because there can be such a big information tradeoff,” she says. For the NIH, the benefits of more expertise at the judging tables are likely to far outweigh potential bias.
“Expertise vs. Bias in Evaluation: Evidence from the NIH,” by Danielle Li, HBS Working Paper.