Image by Mike McQuade

Duncan Gilchrist (PhDBE 2015) wants to experiment with your next meal delivery. Don’t be alarmed: Gilchrist, who runs multiple data science and analytics teams at Uber Eats, isn’t planning to put sardines on your burger or kimchi on your fries. Instead, he’s interested in tweaking the algorithm that matches your order with a courier to ensure your food is delivered as quickly and efficiently as possible. (In his former capacity as head of rider pricing and marketplace experimentation, Gilchrist helped develop the algorithms that drive the rideshare side of the business.)

After running the modified algorithm for a while in one or more markets where Uber Eats operates, Gilchrist will eventually switch back to the original version. He and his colleagues will then pore over the data, comparing the performance of the two algorithms across thousands upon thousands of orders so that Uber executives can weigh the evidence for making a change in operations.

The key word in the preceding sentence is evidence. And it is central to what HBS faculty members Michael Luca and Max Bazerman call the “experimental revolution”: an ongoing movement among organizations of all kinds—for-profit and nonprofit, private and public—to rely on experimental findings when making decisions about products, services, and policies.

As Luca and Bazerman recount in their recent book, The Power of Experiments, it’s a trend that originated in government circles more than a decade ago. Drawing on insights gleaned from social psychology and behavioral economics, government wonks began conducting randomized control trials—the gold standard for experimentation, in which participants are randomly assigned either to an experimental group or a control group—to advance policy goals and demonstrate the value of new initiatives.

Perhaps the most famous example, which dates to 2010 or thereabouts, involved an effort by the British government’s Behavioural Insights Team (aka BIT, or the Nudge Unit), on whose advisory board Luca and Bazerman now sit, to increase back-tax collection by Her Majesty’s Revenue and Customs. By making subtle changes to the reminder letters and mailing each new version to a randomly selected group, the team was able to home in on the most effective way of prodding laggards, raising millions of pounds in tax revenues that would otherwise have been left on the table. Inspired by BIT’s success, hundreds of similar nudge units quickly emerged around the world.

Tech companies were also quick to seize on the idea of running randomized, controlled trials when testing new ideas and products, partly because they were already swimming in data and partly because randomization was relatively straightforward for them. If Facebook wants to know whether a different font will lead users to spend more time on their feeds, for instance, the company can simply present that change to a random group that will serve as the experimental arm of the trial. Everyone who continues to see the same old font becomes part of a de facto control group against which the experimental group can be compared.

According to Luca and Bazerman, the major tech companies currently run thousands of such experiments every year. They have even established formal infrastructures for doing so, such as the teams of economists, computer scientists, statisticians, and machine-learning experts that Gilchrist leads at Uber Eats. These experimental units have made millions for big tech by improving products and services, and they’ve saved millions more by cutting waste. Luca notes that an early experiment by eBay’s economics research team allowed the company to identify about $50 million in unnecessary spending on what turned out to be utterly worthless Google ads.

Experimentation can also reveal, and therefore help to cure, unethical practices. Several years ago, Luca and co-authors Ben Edelman and Dan Svirsky, both then at HBS, ran an experiment that showed Airbnb hosts were discriminating against Black guests, ultimately leading the company to make platform changes that Luca had proposed in his research.

Yet perhaps the greatest advantage of this shift toward evidence-based decision-making lies in the concomitant move away from an overreliance on gut instinct. “We’re overconfident in our limited intuition,” says Bazerman—a situation that often leads to poor decision-making, especially given the basket of cognitive biases (see: overconfidence) to which human beings are prone.

“In the absence of measurements, you tend to rely on seasoned folks to make decisions without actual data behind them,” elaborates Gilchrist. “What’s great about experimentation is that you can really flip that on its head in a valuable way. You can say, Hey, let’s take a number of different teams, let them try things out and measure them, and make decisions that way.”

Which is not to say that conducting good experiments is easy. The market-level experiments that Gilchrist typically runs, for example, involve complex spillover effects: Make a change to one product or service across a city in which Uber operates and you will likely affect all the others, too. Compensating for those effects requires rigorous experimental design, powerful statistical techniques, and careful interpretation of results.

What’s more, for an experiment to bear fruit, all the stakeholders—ranging from the data scientists running the test, to the executives who hope to learn something from it—must agree on what they are trying to measure, how they are going to measure it, and how they will use those metrics to arrive at a decision. That, in turn, requires leaders who are willing to admit that there are things they don’t know and who are ready to invest in the skills, expertise, and organizational structure that successful experimentation requires. “It takes good management to have effective experimental infrastructure,” says Luca. He has developed a course, From Data to Decisions, to help HBS students develop the skills they will need to leverage experiments within their own organizations.

While big tech was quick to capitalize on the power of experimentation, traditional businesses—which may find it more difficult to implement randomization, or lack the sheer number of participants or the right kinds of data to run successful experiments—have been slower to grasp the nettle. Too often, says Bazerman, “convincing an executive that an idea is good is a whole lot easier than convincing an executive to slow down and test on a smaller sample to actually find out.”

Still, given the proven power of experiments and the advent of a new generation of business leaders who are familiar with the benefits of experimentation, Bazerman is hopeful that the revolution will continue to gather steam. “We’re in the early stages,” he says. “I think that the rest of the business world will come along.”

Post a Comment