01 Mar 2013

Sizing Up Social Impact

Quantifying results in the social sector comes down to the tricky work of measuring social good
Re: Maureen Harrington (MBA 2001)
by Julia Hanna


As the aftereffects of the 2008 global economic crisis continue to be felt, nonprofit organizations are competing for fewer and fewer dollars. Donors, also feeling the squeeze of a sluggish economy, just don't have as much to go around—and when a grant is awarded, it comes with increased expectations for hard, measurable results. Which seems only fair, right?

The problem: Nonprofits can lose sight of their mission if a major funder's performance measurement priorities don't align with the necessary actions to achieve an organization's mission. Add to that the fact that government and individual donors may have different priorities and demands, and soon a nonprofit trying to please too many (including the general public) pleases no one—least of all those constituents they were hoping to help.

"Accountability and impact are the two big topics of discussion among nonprofit leaders today," says Associate Professor Alnoor Ebrahim, a member of the School's General Management Unit and Social Enterprise Initiative who focuses on the challenges of performance management, accountability, and governance facing social-sector organizations. As one branch of his work, Ebrahim is delineating a framework that takes into account the inherent uncertainty and complexity of measuring results in the social sector.

"Performance measurement in nonprofits and in NGOs requires an assessment of social change instead of profit-making; unlike financial performance, there is no common currency of measurement and aggregation," Ebrahim explains. "The value created by social-sector organizations accrues to society rather than to a set of owners or shareholders, which raises issues of control—there's a limit to how much influence an organization can have on social factors beyond its boundaries." Most research on organizational performance focuses on cause-and-effect situations that assume degrees of linearity and control that exist only in limited circumstances, he adds. For example, it's straightforward to measure the number of temporary housing units an organization has created in post-earthquake Haiti. But how does one measure the effectiveness of a training program in boosting farmer incomes in Ghana?

This was the question facing the Millennium Challenge Corporation (MCC), one of several organizations Ebrahim has studied or written about in cases that examine approaches to performance measurement. (See "A Case for Performance Management" below; others include the Robin Hood Foundation, Action-Aid International, the Aga Khan Rural Support Programme India, and Acumen Fund.) Created by Congress, the MCC is an independent foreign aid agency with the mission to reduce poverty through sustainable economic growth in some of the world's poorest countries.

"Farmer training was at the heart of the Ghana program," says Jonathan Bloom (MBA 1972), MCC's deputy vice president, West Africa. "We set an objective of training over 50,000 farmers; the final number was just under 67,000."

For some organizations, performance measurement would begin and end with that data point alone. But the MCC takes a much longer view. "Training farmers is an input," says Bloom, who joined the MCC when it was founded in 2004 after stints at the World Bank and at his family's industrial distribution business. "Then you have to consider the outputs: Did they use this new technology and training? Did it produce more bananas or mangoes or pineapples? Did they sell those additional crops? And ultimately, what was the change in household income? At the end of the day, did we reduce poverty?

"We measure quantified objectives beforehand, monitor them during implementation, and evaluate them afterward," Bloom continues. "It's all very public, very explicit, and very rigorous. Sometimes we reach our objectives. And frankly, sometimes we don't."

In effect from 2007 to 2012, the five-year, $547 million Ghana "compact" (MCC's name for long-term grant commitments) encompassed investments in irrigation and roads, community services, and the farmer-training program cited by Bloom. To qualify for funding, MCC partner countries must meet a majority of 20 economic, governance, education, health, civil liberties, legal, and anticorruption benchmarks as determined by third-party data from organizations such as the International Finance Corporation, the World Health Organization, and the Heritage Foundation, among others. "It's become a valuable Good Housekeeping seal of approval that countries value, because it can bring in private investment," says Bloom. "Fortune 500 companies have told us that they read our scorecards every year."

In countries where the MCC operates, a government entity implements the program agenda, in addition to monitoring progress and results. "It is a formal organization established at the country level," notes Ebrahim. "It doesn't disappear into other government agencies. The reasoning is that you can't achieve results without an accountable, locally embedded authority directly championed by the country's executive leadership. In the case of Ghana, it's called the Millennium Development Authority, and it's locally based in Accra, with a governing board made up of top figures from government, business, and civil society. That, I think, is a very serious advance."

"We have enormous respect for a principle we call country ownership," Bloom remarks. "That is, people get done what they want to get done. It's really the country's decision to determine which problems to solve, although we will say when we can't be helpful in a particular area."

Ebrahim's case on the MCC (coauthored with V. Katsuri Rangan) describes the internal debate by the Ghana team—which then included managing director Maureen Harrington (MBA 2001)—around how to respond to the government's insistent request, relatively late in the planning process, for increased community services in areas such as education, water and sanitation, and electrification of rural areas. All worthy causes, to be sure, but also more difficult to clearly integrate with MCC's precise directive of focusing on activities that lead to poverty reduction through economic growth.

"The MCC has been explicit in setting boundaries around initiatives that it believes will lead to increases in farmer incomes," Ebrahim says. "On one level that means core elements such as water, transportation, and roads, as well as access to seeds, fertilizers, and new training and technology. But any funder will be confronted with a situation where a potential grantee says, 'We'd like to pursue this activity—such as improving schools—which isn't explicitly linked to your mission. But we believe that unless you do it, your mission will be difficult to achieve.' "

Spoiler alert: The MCC did commit $46 million to community services, ultimately reducing the average distance from a rural water source from 1,190 meters to 522 meters and decreasing the average time required to collect water from 44 minutes to 32 minutes. Electrification of rural areas and the computerization of rural banks improved the flow of goods and services and decreased check-clearing times at banks. A government-training program designed to increase procurement capacity was adopted by the World Bank and other institutions. Finally, in an effort that posed unique performance measurement challenges, 250 schools were either constructed or rehabilitated.

"The outcome of investments in education, such as those we made in Ghana, can be a bit more difficult to evaluate," Bloom admits. "It's not difficult conceptually, but as a practical matter, it's challenging to conduct baseline surveys before an investment and to track and measure students for the years that follow."

With that understanding, decisions to undertake projects that don't fit as neatly with an organization's goals should be made with careful consideration. "An organization can endlessly dilute its boundaries around what it does, or it can determine if an unexpected opportunity might help advance its mission," says Ebrahim. "If so, the onus is on the organization to measure results. If it's difficult, that doesn't mean it shouldn't be done; if anything, we should devote even more attention to the task so that outcomes can be improved."

Too often, Ebrahim notes, organizations view performance measurement the wrong way around, allowing what they can measure and communicate easily to drive strategy; instead, the organization's strategy should drive what it measures. "Many nonprofits measure performance because they want to send a message to the outside world that they're doing good work, in order to secure the resources needed to do that work. But the real challenge is to start with the problem that you're trying to solve, figure out the right strategy or strategies for achieving it, and then build measures that help you determine if you're actually hitting the mark. The greatest learning arises when you find that you're not achieving the results you hoped for. It's an opportunity to refine your hypotheses about what works and what doesn't."

Even as the MCC moves forward in evaluating the impact of the Ghana compact's education and social-services projects, Bloom recognizes that years will pass before a final benefit analysis can determine if the investment resulted in significant return. In the meantime, he and his MCC colleagues continue to keep a tight focus on those activities that will contribute to its mission.

"I think the MCC is doing the right thing, for the right people, at the right time," Bloom says. "This is the best job I've ever had. We're really trying to implement the principle of teaching a man to fish in order to give people the basis for sustainable economic growth…which is, fundamentally, private enterprise. That is the engine that will keep development moving ahead."

A Case for Performance Management

In addition to the Millennium Challenge Corporation, HBS associate professor Alnoor Ebrahim has researched and written cases on several other nonprofits taking on performance measurement challenges in creative ways:

"The Robin Hood Foundation" (with Catherine Ross). Created by hedge fund and investment managers, the Robin Hood Foundation fights poverty through grants to nonprofit organizations in New York City. As the global financial crisis continues to impact the poor disproportionately, the foundation's senior vice president has developed a benefit-cost (BC) approach to analyze the performance of program grants. How effective is the method? Is funding programs with the highest BC ratios a good way to fight poverty? In three or five years' time, how will Robin Hood know if it is succeeding?

"ActionAid International: Globalizing Governance, Localizing Accountability" (with Rachel Gordon). As a global NGO working in 45 countries, ActionAid International aims to eradicate poverty by addressing its underlying causes such as injustice and inequality. This case follows a series of radical transformations, including a power shift from its headquarters in London to an international secretariat in Johannesburg; a new federated governance structure that increases the influence of units in Africa and Asia; and innovations in accountability and transparency to the poor communities with which it works.

"Acumen Fund: Measurement in Impact Investing (A & B)" (with V. Kasturi Rangan). Acumen Fund is a global venture capital firm with a dual purpose: It looks for a return on its investments, and it also seeks entrepreneurial solutions to global poverty. The (A) case examines Acumen's potential new projects in Kenya and associated decision-making factors, including political risk. The (B) case considers critical measurement problems for the organization at large. What should its performance tracking system look like when its interest is not just the bottom line but also social impact? And how can Acumen build industrywide benchmarks on measuring returns in order to attract investors to the field?

"I'm developing frameworks that can serve as diagnostic tools for organizations to help managers and front-line staff have a conversation around what they are trying to achieve and how to measure it. This depends on two critical factors: how much they know about causality in their interventions, and how much of their environment they can reasonably control," Ebrahim says. "Based on these two conditions, they can figure out what to measure."

Countdown to Change

In addition to Ghana, the Millennium Challenge Corporation has formed partnerships with 38 countries around the globe, from Albania to El Salvador to Zambia. Some of its recent five-year projects, or compacts, include:

Honduras, 2005–2010. This $205 million compact, one of the first signed by the MCC, resulted in the training of over 7,000 farmers; more than 10,000 loans for the purchase of seeds, equipment, and tools; and 563 kilometers of improved and completed roads.

Benin, 2006–2011. A $307 million compact with Benin addressed obstacles to economic growth by expanding the Port of Cotonou, investing in technology to promote land registration and security, improving access to capital, and creating a more efficient judicial system.

Tanzania, 2008–2013. Investments of $698 million (the largest MCC commitment to date) will improve roads, thus increasing commerce and connecting communities with markets, schools, and health clinics; improve the reliability of electric power and extend service to communities not currently served; and expand the availability and reliability of potable water to increase health and productivity.

Philippines, 2011–2016. This $434 million compact will support reforms and investments to modernize the Bureau of Internal Revenue, expand and improve small-scale, community-driven development projects in rural areas, and rehabilitate a secondary national road in Samar province to increase commerce and improve safety.


Post a Comment