Skip to main content

An evaluation framework and empirical evidence on the effect of pay-for-results programs on the development of markets for welfare-enhancing agricultural technologies

Abstract

Background

Donors and international development organizations increasingly recognize that private sector investment and creativity are needed to enhance global food security. Pay-for-results schemes are receiving greater attention as a means to catalyze private sector investment in sustainable, inclusive markets for goods and technologies that achieve food security and agriculture development goals. In pay-for-results schemes, the development organization promises prizes to private sector actors for achieving pre-specified goals.

Method

We describe an evaluation framework to help development organizations learn from both successful and failed pay-for-results projects to achieve agriculture and food security outcomes. Applying the evaluation framework, we describe the findings from four pay-for-results projects sponsored by AgResults, a multilateral initiative funded by development organizations from four countries (Australia, Canada, the UK, and the US) and the Bill & Melinda Gates Foundation.

Results

The lessons highlighted from these examples illustrate the importance of structuring the prize to encourage the creation of competitive agricultural markets; aligning the prize structure with the development goal of improving smallholder farmers’ food security; and constructing a theory of change that reflects a thorough understanding of the baseline market, enabling environment, and underlying assumptions about competitors’ response to the prize.

Conclusions

Our work has several policy implications:

  • Under certain conditions, pay-for-results mechanisms can help develop competitive, smallholder-inclusive agricultural markets and reduce food insecurity.

  • Prize competitions offering multiyear, proportional prizes are more conducive than grand prizes to fostering the development of competitive agricultural markets.

  • The enabling environment plays a significant role in pay-for-results mechanisms’ success or failure.

  • Private sector-led actions alone may not be sufficient to adequately address the targeted development challenge.

Background

In their 2010 meeting, G20 leaders called for the development of pay-for-results (PfR) mechanisms to achieve development goals in the food security and agriculture sector harnessing “the creativity and resources of the private sector” [1]. Two years later, the governments of Australia, Canada, the United Kingdom, and the United States, in partnership with the Bill & Melinda Gates Foundation, pledged funds to establish AgResults. AgResults is an initiative to design, implement, and evaluate PfR “challenge projects” to develop markets for welfare-enhancing agricultural technologies. Compared to traditional approaches which try to “prime the pump” for market development by raising awareness of and demand for welfare-enhancing agricultural technologies through subsidized distribution of those technologies, AgResults seeks to stimulate private sector investment in the development of smallholder farmer-inclusive commercial markets for these technologies. Insofar as these markets imply increased adoption of these technologies by smallholder farmers and poor households, they are expected to enhance their food security outcomes. This approach is also expected to have a more sustainable impact and cost less, because it develops markets.

Although AgResults is the first to test PfR schemes in the food security and agricultural sector, there is emerging evidence that the use of PfR schemes has broken down implementation barriers and driven progress on intractable social challenges in diverse sectors (see Meuth Alldredge et al. [2] for an evidence review). Mendelson et al. [3] and Suthar et al. [4], for example, document the success of PfR efforts in the health sector. Payment for environmental services has been used to pursue climate goals with evidence of efficacy in increasing forest cover [5, 6]. In the energy sector, PfR schemes are also common ways to incentivize the adoption, sale, and use of climate-friendly energy technologies, and to promote innovation across the energy supply chain [7].

PfR schemes for agriculture market development are relatively new, and there is still much to learn from both successful and unsuccessful attempts to use them. To investigate the promise of PfR schemes for agriculture development, AgResults provides a rich portfolio of independent challenge projects in a variety of countries. As of 2021, AgResults is a US$152 million initiative with four concluded challenge projects, four additional challenge projects underway, and two projects in the design phase. Each AgResults challenge project has a common goal: to incentivize the private sector to develop and disseminate welfare-enhancing agricultural technologies in environments where technology uptake was limited, for example, due to uncertain or risky operating environments, credit constraints, lack of information, an unsupportive enabling environment, or other factors. To incentivize the private sector to overcome these market constraints in each of these contexts, each AgResults challenge project introduced cash prizes to targeted market players (or “competitors”) that achieve pre-specified outcomes. The theory is that, if a prize is well designed and attractive to private sector competitors, they will creatively find solutions to constraints that otherwise inhibit the development of a market for the technology. Like other PfR schemes, AgResults provides payments to competitors only if and after they achieve the pre-specified outcomes; these outcomes are often expressed in terms of volume of output or sales of the technology or the output that it allows the farmer to produce. The expectation is that, if the private sector is successful, the resulting market for the technology will be self-sustaining after the competition ends. These prize competitions fall in the class of what McKinsey and Company [8] call “market stimulation prizes,” which includes grand prize competitions, advance market commitments, and social impact bonds (SIBs).

To the best of our knowledge, there are several papers that have evaluated PfR approaches but there is no evaluation framework for PfR schemes other than one proposed for evaluating grand challenges [9]. The PfR literature includes evaluations of grand prize challenges, advance market commitments, and social impact bonds. Grand prize challenges typically incentivize technology development, such as vaccine R&D [10], and there are few very evaluations of these types of challenges, perhaps because the prize is paid out only when the challenge is met which is considered as adequate evidence. SIBs mostly exercised in high-income countries, reward delivery of welfare services [11]. The literature on SIB makes an argument about the importance of evaluating social impact bonds, but does not put forth a formal framework to evaluate them [12, 13].

Similar to SIBs and grand challenges, the AgResults projects are focused on achieving a development goal such as improved health through reduced aflatoxin contamination (Nigeria Aflasafe) and enhanced nutrition (Zambia biofortified maize and Uganda quality legume seeds), reduced post-harvest losses through improved on-farm storage (Kenya on-farm storage), and improved farmer incomes (Nigeria Aflasafe, Uganda quality legume seeds, Kenya on-farm storage) [14,15,16,17]. However, as noted above, these projects also aim to develop an inclusive market for the technologies.

We propose an evaluation framework for such market development PfR mechanisms and applying it to AgResults. We demonstrate how the evaluation framework provides insights about both completed and unsuccessful projects regarding markets for aflatoxin-reduced maize in Nigeria, improved legume seeds in Uganda, improved on-farm storage devices in Kenya, and biofortified maize in Zambia. All four projects were ultimately intended to impact smallholder farmers’ food security outcomes by increasing their farm incomes and/or by increasing consumption of nutritious (vitamin-A fortified maize and legumes) and safe (aflatoxin-reduced maize) foods. Although the Uganda and Zambia projects were terminated due to lack of competitors’ success in achieving pre-specified award criteria, all four AgResults projects discussed in the paper provide important insights into the unique challenges and lessons for designing PfR for market development. We synthesize and discuss lessons learned from our evaluations of four AgResults technology adoption projects. These lessons will help inform future PfR mechanisms to foster development of markets for welfare-enhancing technologies by answering questions about the development goals that PfR is best suited for, private sector actors that the PfR should incentivize and how the PfR’s prize should be structured.

Method: evaluation framework

The overarching goal of our evaluation framework is to learn about the three key components of PfR mechanisms that aim to increase adoption of innovations by market creation: the development goal, the prize design, and the theory of change. The development goal of such PfR mechanisms is typically to enhance household well-being by the creation of a self-sustaining market. A PfR’s prize structure rewards competitors that achieve pre-defined outcomes toward the development goal to develop markets for smallholder welfare-enhancing technologies. The prize design defines the specific outcomes that will trigger payout, the criteria against which those outcomes are judged, and the timing and value of the prize payments. The design also indicates the third-party verification process that confirms whether the competitors achieved the outcomes before the prize is awarded. Underpinning the prize design is a clear theory of change that articulates the expected causal linkages between (1) the prize structure, (2) the competitors’ expected investments and activities in response to the prize structure, (3) the outcomes—both firm- and market-level—of those investments and activities, and (4) impact on the development problem.

We offer an evaluation framework designed to assess whether the development goal was achieved, whether the prize structure was appropriate, and whether the theory of change was correct. The evaluation framework proposes a market analysis, a causal impact analysis, and a cost-effectiveness analysis to answer several evaluation questions. Table 1 presents these evaluation questions, and the associated evaluation metrics that capture all elements of the PfR. Several of the metrics, marked “FS” for “fail-safe” are useful even in the case where no prizes are paid out due to the failure of any competitor to progress. In the next section of this paper, we apply this framework to share the evidence from four AgResults PfR projects. After, we illustrate how the framework lends itself to synthesizing lessons learned by looking at the patterns of results across projects.

Table 1 Evaluation metrics for PfR projects focused on market creation, by analysis method

Market systems analysis

An in-depth qualitative market systems analysis is critical to assess whether or not the PfR initiative catalyzed development of a sustainable market for the targeted technology and/or its product. Market development, by nature, generally produces changes across many levels or entities within a value chain. With many systemic changes—some exogeneous and others directly affected by the development project—it can be difficult to attribute any specific change to the development project itself. Therefore, the market systems analysis should encompass a qualitative industrial organization framework that systematically assesses relationships between the market’s underlying conditions, firms’ strategic behavior, and the market’s structure and performance, before the PfR and after it to determine if PfR altered these elements. To capture the impact of the project on these elements, we propose metrics that describe market structure, performance, and sustainability drawing on structure-conduct and performance paradigm to characterize markets [18]. To gather data for these metrics, we suggest ex-ante, periodic, and ex-post interviews with private sector competitors, their suppliers and customers, and relevant market and regulatory experts; along with prize award verification data and results of the farm- or consumer-level impact analysis. A comparison of these metrics before and after the PfR helps assess if the PfR engaged private sector actors, if it incentivized them to address the underlying the constraints limiting market development, and if it created a market that is inclusive and sustainable. In the case of AgResults, the comparison of these metrics before and after the projects helped us to assess whether the PfR helped to establish markets for smallholder produced aflatoxin-free maize in Nigeria, for improved legume seeds in Uganda, for pro-vitamin-A maize grown by smallholders in Zambia, and for improved on-farm storage for smallholder predominant areas in Kenya.

Impact assessment

To the extent that PfR’s ultimate aim is to develop markets that improve livelihoods and well-being of rural households, the evaluation should focus on the ultimate intended beneficiary: such as the consumers (including farmers) whose maize meal is contaminated with aflatoxins or who are Vitamin-A deficient, the farmers whose legume harvests are poor due to a lack of access to quality legume seed, or the poor farmers whose food security is undermined by excessive post-harvest losses of the grains they grew. To evaluate PfR’s impact on the intended beneficiary, we propose metrics about the PfR’s effect on farmers’ knowledge about the innovation, adoption of an innovation, and farmers’ well-being and the differential impact on women and other marginalized groups, since by their very nature, market systems development initiatives have the potential to exclude women given that women typically have lower levels of market integration than men [19].

Cost-effectiveness analysis

Cost-effectiveness analysis is useful to compare the efficiency of different program strategies, though it is important to consider the differences in implementation contexts, challenges, and complexity. If traditional development projects intend to achieve the same results in the same geographic regions, the associated projects’ expected costs per targeted result could be compared with those of AgResults. Direct comparisons of cost-effectiveness findings are best focused on alternative approaches to promoting uptake or market development of similar technologies in similar environments, rather than across implementation contexts and technologies. For the AgResults evaluations, we used the cost-effectiveness analysis to estimate projects costs per pre-specified outcome directly attributable to the project, within the timespan of the project. For example, as the goal of the Kenya project was to promote adoption of improved on-farm storage, we divided project cost by the number of smallholder farmers whose adoption of on-farm storage solutions was attributable to the project, based on findings from the impact analysis. To the extent that technology adoption continues to increase after the program, and that increase is due to the program, the cost-effectiveness measure calculated at the close of the project is an underestimate of its true cost-effectiveness. Therefore, we also recommend conducting forward-looking analysis to assess cost-effectiveness analysis under different scenarios of market development and sustainability.Footnote 1

Results: empirical application and evidence

The first four projects of the AgResults initiative targeted key food security crops in Africa. Two of the projects were completed (Nigeria and Kenya), and two were terminated early (Uganda and Zambia). Table 2 summarizes the development goal, prize design, and theory of change for each project.

Table 2 Key elements of AgResults challenge projects

We apply the proposed evaluation framework to provide evidence of the effectiveness of AgResults’ PfR approach. The balance of this section summarizes our findings on the evaluation metrics in the market analyses, impact analyses, and cost–benefit analyses to answer the evaluation questions there-in. The Uganda and Zambia projects have fewer data, owing to the fact that no competitor made sufficient, timely progress toward the award criteria.

The findings from our market analyses of the first four AgResults projects are summarized in Table 3 [20,21,22]. The positive findings are that the prizes can motivate a significant number of private sector actors to engage in a new market, and that the new market can be inclusive of women and smallholder farmers. The market is more likely to be sustained after the project ends if the private sector actors have a business interest in continuing to engage in the market. While initial supply- and demand-side constraints can be daunting and not all are addressed by competitors, the lack of a supportive environment is the biggest threat to the market success. Requirements for an enabling environment include a general trust between buyers and sellers, access to credit, and the absence of government investment in markets for substitutes.

Table 3 Findings from the market analyses

The “R” in PfR, i.e., the results triggering award payment, are rarely the full measure of progress toward a development goal. A causal impact analysis is an opportunity for prize sponsors to examine a broader set of outcome measures describing progress toward a development goal. In addition, causal impact analysis compares the results achieved to the results that would have been achieved in the absence of the PfR project (rather than assuming that no results would have been achieved absent the PfR project).

Despite increased attention to random assignment evaluations to demonstrate the effectiveness of new development strategies, causal impact analysis of market-level interventions remains rarely practiced, because it is especially challenging. Consumers, farmers, or intended beneficiaries are only one component of a market systems approach: a systems change, by nature, generally produces changes across many levels or entities within the market system. The impact of market systems approaches on households, consumers, or other vulnerable groups is not easily evaluated using field experiments, because it is difficult or impossible to define and protect a comparison group that is not affected by some level or entity within the market system. In Nigeria, we compared farmers involved in AgResults to a comparison group of farmers selected by us after the intervention was under way and used propensity-score weights to improve comparability of the treatment and comparison groups on stable characteristics not affected by the intervention. In Kenya, we used a difference-in-differences impact evaluation design of farmers who adopted the technology and matched farmers who did not adopt the technology. We did not conduct a causal impact analysis for the Zambia or Uganda projects that ended early. Table 4 summarizes the findings from our causal impact analysis for the projects in Nigeria and Kenya. The causal impact analysis showed that adoption of the technology was considerable, though less than otherwise suggested using the project monitoring data. In Nigeria, technology adoption improved farmer income, while in Kenya, it did not. Although, in Nigeria, PfR did not raise farmers’ awareness about the full health benefits of the technology.

Table 4 Findings from the causal impact analysis

The cost-effectiveness analysis from the four projects revealed that, on average, prize award costs roughly equaled management costs. This suggests that, if a sponsor hosts a portfolio of projects, the management and verification costs alone will be a significant proportion of the budget. The two successful projects, Nigeria and Kenya, spent vastly different proportions of their budgets on the prize award: this is due to high verification costs in Nigeria and also raises the question of whether the prizes were too generous in Kenya. The cost per farmer reached in Kenya ($39) is high compared to the cost of an average storage solution (near $2.50); the high cost per farmer reached in Nigeria ($134) may be warranted owing to the difficulty in changing farmer cultivation methods if verification costs are very high; a failed PfR could cost more than a traditional technical assistance project if no competitor realizes the prize criteria or if the newly created market is not sustained beyond the life of the project. Table 5 summarizes our cost-effectiveness results for the two completed projects.

Table 5 Findings from the cost-effectiveness analysis

Lessons learned

The evaluation metrics summarized in the previous section directly answered the research questions posed for the market, impact, and cost-effectiveness analyses. Synthesizing findings across these analyses, the evaluator has an opportunity to identify some lessons learned for the research community regarding the design of PfR. For example, integrating findings with respect to the key constraints for market development (market analysis) and impacts on farmers (causal impact analysis), researchers are able to reflect on whether the pathways in the theory of change proved correct, allowing the full set of development goals to be met. In this section, we discuss our synthesis of the evaluation findings from AgResults in the form of lessons learned. These lessons are organized by PfR component: development goal, prize design, and theory of change.

Development goals

Based on the evidence from these four projects, we summarize the circumstances in which prize competitions may be more likely to achieve their development goals.

The enabling environment must be adequate to support development of a market

Two AgResults PfR mechanisms failed at least in part as a result of enabling environment constraints that existed before the PfR mechanism was put in place. Uganda lacked an effective seed quality certification system. Without the ability to distinguish quality of seeds or to control counterfeit labels, seed companies lacked incentive to invest in producing or marketing high-quality legume seeds. In Zambia, extensive government involvement in the market for white maize depressed investment in the market for PVA maize, which was a close substitute for white maize. On the other hand, in Kenya and Nigeria, the enabling environment was largely adequate to support development of a private sector-driven market and did not impede the development of private quality verification initiatives where public quality verification did not exist. In Kenya, the Kenya Bureau of Standards had a transparent process to approve the technologies before allowing them into the market, and competitors were able to brand and market their products without extensive problems of counterfeiting. When, at project inception, there was no way to test whether on-farm storage devices were effective against the large-grain borer, AgResults itself contributed with the development and deployment of a testing protocol, thereby avoiding substantial delays due to lack of a means of verifying product quality. In Nigeria, regulations already existed for aflatoxin standards, although they were not broadly enforced, and AgResults verification of Aflasafe-content served as de facto quality verification for buyers that sought aflatoxin-complaint maize. These AgResults projects show the importance of the enabling environment not only in defining and enforcing policies and product standards, but also in not impeding private sector- or donor-led initiatives in cases where public initiatives were lacking.

Access to credit is another aspect of the enabling environment that is particularly critical for PfR mechanisms, given that they do not pay competitors up front. In all AgResults projects, constrained access to operating or investment capital consistently limited the scale of investment, although the degree to which it impeded competitors differed across projects and competitors.

Tradeoffs can exist between market impact and development impact objectives

Since the incentive structure influences how competitors interact with their suppliers and buyers who are usually the ultimate intended beneficiaries of PfR mechanisms, potential tradeoffs between market impact and development impact should be identified and planned for at the outset. In Nigeria, a robust supply base of smallholder farmers was developed by project competitors, yet these smallholder farmers evidenced little awareness of aflatoxins as a health problem or Aflasafe-treated maize as a means to protect them against aflatoxin exposure, despite the fact that reducing smallholder farmers’ aflatoxin exposure was one of the most important intended impacts of the project. Consistent with their business incentive, competitors did not tend to emphasize the health implications of aflatoxins when they interacted with farmers, because they had an incentive to procure as much of the harvested, treated maize as possible. Farmers reacted to the prize incentives in the markets by selling almost all of their treated maize. Our evaluation found that the project had minimal impact on smallholder farmers’ consumption of Aflasafe-treated maize, increasing it by only 0.02 kg per day, or 13% of their daily consumption, in large part because smallholder farmers typically did not apply Aflasafe to all of their maize. As a result, the project generated limited health impacts for poor, rural farmers. In situations where competitor incentives may conflict with the project’s intended development impact, and where adjusting the outcome on which prizes are based cannot feasibly eliminate this conflict, there is arguably room for complementary activities on the part of the project or some other public actor to provide information to project beneficiaries that will help offset this unintended impact.

Prize designs

The evaluation metrics from the four AgResults projects give rise to several lessons learned about appropriate prize designs.

Prize structures with more frequent payouts enable competitors to increase scale by re-investing returns

AgResults projects are distinct from other PfR mechanisms, because they award prizes at multiple stages, rather than relying entirely on a single end-of-project prize payout. Our evaluations showed that more frequent payouts enabled competitors to increase the scale of their activity by re-investing their proceeds; this was particularly important given the capital constraints that these competitors faced. With the exception of the Uganda project, which never paid out a prize, competitors in all AgResults projects consistently reinvested a large proportion of their prizes to increase their scale of operations. For example, in Nigeria, competitors commonly used their incentive payments to finance input packages for their farmers. In Kenya, where only three competitors earned the first milestone prize, the recipients of these payouts gained a competitive advantage, because they could reinvest their proceeds in their operations and make further gains in relative scale.

Proportional and per-unit prizes attract multiple private sector actors

All four projects demonstrated that PfR mechanisms can attract multiple private sector actors. While no AgResults project offered a winner-take-all prize, precluding empirical observations on their effects in the AgResults context, economic theory holds that per-unit or proportional prizes attract more competitors than winner-take-all prizes [23,24,25].Footnote 2 Competitors’ reflections on how the prizes motivated their participation in AgResults and investment in the targeted market support this idea. Furthermore, across projects, competitors reflected that per-unit prizes increased expectations of positive returns from investment in the market relative to proportional prizes in which payment partially depends on other competitors’ performance. In Nigeria, which offered a per-unit prize, a wide range of private sector actors participated including some that were quite small. In Kenya, in contrast, relying on a proportional prize to motivate competitors (after the initial threshold was reached) induced a strategic competitive behavior that was unique to that project and market. In general, having more competitors can lead to a larger pool of winners who can remain viable market actors at the close of the project, which is conducive to creating a competitive market structure.

The prize should link directly to the desired development goals

Donors invest in developing a market for the technology, because they believe that the technology will improve smallholder farmers’ welfare. The prize design can define qualifying parameters to increase that effect on welfare. For example, in Kenya, the PfR mechanism introduced qualifying criteria that ensured that competitors would be rewarded for sales of technologies that were best suited to smallholder farmers: only improved on-farm storage devices of 540 kg or less (the average amount of maize consumed by an average family in a year) qualified for prizes.

In contrast, in Zambia, the prize did not promote outcomes that would directly benefit smallholder farmers. Competitor millers earned prizes based on their sales of milled PVA maize, which is typically purchased by better-off urban consumers. Poor farmers who have higher prevalence of vitamin-A deficiency rarely buy milled maize. Similarly, schools and health clinics serving nutritionally vulnerable populations usually buy un-milled maize due to its lower cost. Thus, by rewarding sales of milled maize specifically, the miller competition in Zambia did not directly promote procurement or sales of PVA maize to the most nutritionally vulnerable farmers or consumers, who were the project’s ultimate intended beneficiaries. Furthermore, when the Zambia project introduced prizes to seed companies, it did not specifically reward seed sales to rural farmers through commercial channels; instead, it rewarded sales to all domestic entities including government and non-profit actors that would subsidize their distribution of the seed to farmers, undermining the development of a commercial market for the seed.

Right sizing the prize amounts and results benchmarks requires ex-ante cost-effectiveness analysis

Estimating the social returns to a project can be difficult ex-ante, but should be conducted using scenario analysis to assess if the ranges that yield positive results are achievable by the PfR. Deeper investigation of the theory of change with careful fieldwork can support this analysis. In the Kenya project, the cost incurred to achieve that smallholder farmer adoption was high with no measurable benefits to farmers, because the farmers already had another method of reducing post-harvest losses (pesticides).Footnote 3 Evaluations of initial qualitative research had already unearthed this likelihood, which underscores the importance of this work. Such analysis was done for Nigeria where clear assumptions were made about the area that farmers may treat with Aflasafe, the yield results they may achieve, and the prize premium that will increase their incomes.

Theory of change

The evaluation metrics also give rise to an important lesson regarding the theory of change behind PfR projects.

The theory of change should be based on careful market and behavioral research and its assumptions should be clearly articulated and tested in early phases

An initial market systems analysis is essential for all aspects of the PfR mechanism design process—including identifying the development problem, its technological solution, and the competitors, and designing the incentive structure. The overriding objective is to identify the key constraints to developing a market for the target technology, to assess the business case for potential competitors, and identify the potential financial and economic returns for smallholder farmers’ engagement with the technology.

The market systems analysis allows project designers to ground the theory of change in the project’s market context. This grounding offers early opportunities to vet and strengthen the project design, especially when it is coupled with a detailed theory of change that describes how the competition will lead to the development of a sustained market and development impact among its target beneficiaries. In Uganda, for example, an underlying assumption that providing incentives to increase sales of improved legume seed would increase the availability of quality seed to farmers was challenged prior to the project’s initiation, following a market systems analysis by the evaluator. This led to major changes to the PfR mechanism’s design to account for this issue, although ultimately the project was canceled when the alternative quality verification system failed to gain public sector approval.

The theory of change rests on a set of assumptions about how the PfR mechanism would work; these assumptions should be clearly articulated and, to the extent possible, substantiated. Pilot or early phases of PfR mechanisms offer opportunities to evaluate assumptions about how the market actors will achieve the development goal. While it is difficult or even impossible for project sponsors to understand all aspects of a market system, stating critical assumptions at the beginning of the project helps to articulate expectations, realization of which can be monitored throughout the project and used as a basis for project modifications. The theory of change also guides monitoring and evaluation throughout the prize competition’s implementation and provides information to project sponsors on the need for modifications to the competition, while it is in process. In all four projects described in this article, at least one critical assumption underlying the project’s theory of change did not work out. For example, in Kenya, the assumption that on-farm storage products would reduce post-harvest loss did not work out, because farmers were already controlling losses using pesticide dust; this significantly reduced the financial benefits realized by farmers despite impressive uptake of on-farm storage products as a result of the project. In Nigeria, the assumption that the project would increase farmers’ consumption of Aflasafe-treated maize did not prove true; instead, they sold nearly all of the Aflasafe-treated maize that they produced and retained untreated maize for household consumption. Early identification and testing of these underlying assumptions might have offered opportunities to adjust the projects’ designs to increase their impact given emerging results.

Discussion and conclusion

PfR mechanisms, as presented by the AgResults initiative, offer an innovative means to leverage private sector ingenuity for the realization of food security goals. In this paper, we have drawn on an evaluation framework that includes market analysis, impact analysis, and cost-effectiveness analysis to highlight lessons from the first four PfR mechanisms used to advance development goals related to food security and agriculture. We have also identified lessons learned from those evaluations. The experience of AgResults thus far suggests that PfR mechanisms can be an effective tool to address challenging development problems by promoting the development of markets that promote and enable widespread adoption of promising agricultural technologies—under certain conditions. We conclude by offering insight into several critical questions about the suitability and design of PfR mechanisms.

What development contexts are best suited for PfR mechanisms?

Market stimulation prizes work when the development problem is situated in a context where the enabling environment for the private sector can support emergence of a market for the promoted technology. The PfR approach rewards private sector participants for their success in creating a market for a given technology, and in order for this approach to be effective, addressing the constraints inhibiting the market’s growth must be within the “manageable interest” of the project’s PfR participants. In the AgResults projects we have discussed, for example, competitors were in a good position to overcome smallholder farmers’ limited awareness of the technologies and to create effective procurement and distribution systems for their products; none of the projects were hampered by a weak enabling environment. In Uganda, however, the lack of an effective system to certify seed quality was an enabling environment constraint that undermined the development of supply and demand for quality legume seeds, and that was not in the manageable interest of the private sector participants in the project.

What types of private sector actors should a PfR mechanism incentivize?

The first four AgResults projects demonstrate the importance of targeting the prize competition to competitors that are both well positioned to address the key constraints limiting the development of a market and have a fairly strong business case for staying in the new market after it is established.

How should the prize be structured?

Per-unit and proportional prizes can attract a large number of market actors, and relatively, frequent prize payouts enable and increase private sector investment. While the prize should be large enough to motivate private sector investment, it should not be so large as to offset tepid interest among potential competitors that reflects the lack of a viable business case for those competitors. Furthermore, the prize needs to be modest enough for its development impact to be cost-effective and yield a net positive return on donor investment. An ex-ante cost-effectiveness analysis can help define minimum thresholds before the prize is paid out to ensure a positive return on donor investment.

The duration of the competition—the number of years over which prizes are paid—should reflect the amount of time it takes to make the needed investments to overcome key constraints. However, the timeline should be aggressive enough that results are achieved at least as quickly as they would be under alternative donor mechanisms. Graduated reductions in prize incentives in later stages of the competition can help orient competitors to overarching market conditions without completely nullifying the incentive offered by the prize to invest in the nascent markets.

To conclude, we acknowledge that policy makers and researchers alike still have much to learn about using PfR strategies to stimulate the growth of markets that provide welfare-enhancing technologies. We hope that the use and adaptation of our evaluation framework will help the policy and research community further enrich its learnings about future PfR programs and their optimal design.

Availability of data and materials

Analysis datafiles from the smallholder studies for Nigeria, Kenya, and Uganda, as well as an urban consumer survey that we used in the Zambia evaluation, are currently under review at the USAID’s Digital Data Library and are expected to be publicly available by end of 2021.

Notes

  1. Our evaluation framework does not apply cost–benefit analysis because full costs and benefits are often difficult to capture, and it is unclear whether donors should consider all aspects, for example, competitors’ opportunity costs and any resulting financial windfall for business owners.

  2. However, when competitors have the same abilities, winner-takes-all prizes maximize effort [25].

  3. One way to account for the long-run benefit of market creation is to estimate the expected farmer benefits further out in the future, assuming that the market is sustained. Our sustainability assessment for Kenya showed that the market for improved on-farm storage devices grew substantially [growth amount not available in this draft but in subsequent] in the two-and-a-half years following the conclusion of the project, implying significant improvements to its cost-effectiveness given that no more project costs were incurred over the same period [26].

References

  1. Davies R. AgResults: Innovation in research and delivery. Technical report. Australian National University; 2012. Uploaded September 2018, available on ResearchGate.com.

  2. Meuth Alldredge J, De Roy E, Mokgano E, Mwandri P, Narayan T, Prowse M, Puri J, Rafferty W, Rangarajan A, Usmani F. Evidence review on results-based payments: Evidence Gap map and intervention heat lap. IEU learning paper. Songdo, South Korea: Independent Evaluation Unit, Green Climate Fund; 2020 Dec.

  3. Mendelson A, et al. The effects of pay-for-performance programs on health, health care use, and processes of care. Ann Intern Med. 2017;166(5):341. https://doi.org/10.7326/m16-1881.

    Article  PubMed  Google Scholar 

  4. Suthar AB, et al. Performance-based financing for improving HIV/AIDS service delivery: a systematic review. BMC Health Serv Res. 2017;17(1):6.

    Article  Google Scholar 

  5. Arriagada RA, et al. Do payments for environmental services affect forest cover? A farm-level evaluation from Costa Rica. Land Econ. 2012;88(2):382–99. https://doi.org/10.3368/le.88.2.382.

    Article  Google Scholar 

  6. Jayachandran S, et al. Cash for carbon: a randomized trial of payments for ecosystem services to reduce deforestation. Science. 2017;357(6348):267–73. https://doi.org/10.1126/science.aan0568.

    Article  CAS  PubMed  Google Scholar 

  7. Usmani F, Steele J, Jeuland M. Can economic incentives enhance adoption and use of a household energy technology? Evidence from a pilot study in Cambodia. Environ Res Lett. 2017;12(3): 035009. https://doi.org/10.1088/1748-9326/aa6008.

    Article  Google Scholar 

  8. McKinsey & Company. ‘And the winner is…’: Capturing the promise of philanthropic prizes. 2009. http://mckinseyonsociety.com/capturing-the-promise-of-philanthropic-prizes. Accessed 5 Jan 2021.

  9. Conrad A, Narayan T, Geyer J, Bell S, Kay L. A Framework for Evaluating Innovation Challenges. Bethesda: Abt Associates; 2017.

    Google Scholar 

  10. Murray F, Stern S, Campbell G, MacCormack A. Grand innovation prizes: A theoretical, normative, and empirical evaluation. Res Policy. 2012;41:1779–92. https://doi.org/10.1016/j.respol.2012.06.013.

    Article  Google Scholar 

  11. Nicholls A, Edmiston D. Policy paper. How to facilitate the growth of different kinds of social innovation: The politics of social innovation. 2017.

  12. Jackson ET. Evaluating social impact bonds: questions, challenges, innovations, and possibilities in measuring outcomes in impact investing. Commun Dev. 2013;44(5):608–16.

    Article  Google Scholar 

  13. Gilchrist, David and Peter Wilkins. Book chapter. Social Impact Bonds: The Role of Evaluation, In: Pablo R. Guerrero O., Peter Wilkins, Ray C. Rist (eds), Doing Public Good? Private actors, evaluation and public value. 2017. https://doi.org/10.4324/9780203793008-11

  14. Dalberg. Agricultural pull mechanism: AflasafeTM pilot. Washington: World Bank; 2012.

    Google Scholar 

  15. Dalberg. Agricultural pull mechanism initiative: biofortification pilot. Washington: World Bank; 2012.

    Google Scholar 

  16. AgResults. Seed production in Uganda: increasing smallholder farmer access to improved legume seed varieties: Pilot business plan. Washington: Deloitte; 2014.

  17. Dalberg. Agricultural prize competition initiative: on-farm storage. Washington: World Bank; 2012b.

    Google Scholar 

  18. Caves R. American industry: structure, conduct, performance. London: Pearson; 1987.

    Google Scholar 

  19. Mainville D. Incorporating a gender lens into challenge project design. Evaluator’s lessons learned series, issue 5. Rockville: Abt Associates and Denise Mainville Consulting; 2020.

    Google Scholar 

  20. Narayan T, Mainville D, Geyer J, Hausdorff K, Cooley D. AgResults impact evaluation report: Nigeria Aflasafe™ Challenge Project. Rockville: Abt Associates; 2020.

    Google Scholar 

  21. Ness-Edelstein B, Mainville D, Narayan T, Geyer J, Cooley D. AgResults impact evaluation report: Kenya On-Farm Storage Challenge Project. Rockville: Abt Associates; 2019.

    Google Scholar 

  22. Mainville D, Narayan T. AgResults Zambia biofortified maize challenge project–close-out assessment. Rockville: Abt Associates; 2020.

    Google Scholar 

  23. Cason TN, Masters WA, Sheremeta RM. Entry into winner-take-all and proportional-prize contests: an experimental study. J Public Econ. 2010;94:604–11.

    Article  Google Scholar 

  24. Shupp R, Sheremeta RM, Schmidt D, Walker J. Resource allocation contests: experimental evidence. J Econ Psychol. 2013;39:257–67.

    Article  Google Scholar 

  25. Cason TN, Masters WA, Sheremeta RM. Winner-take-all and proportional-prize contests: theory and experimental results. J Econ Behav Organ. 2020;175:314–27. https://doi.org/10.1016/j.jebo.2018.01.023.

    Article  Google Scholar 

  26. Mainville D, Ness-Edelstein B. AgResults evaluation: Kenya on-farm storage challenge project. Sustainability assessment. Prepared for the U.K. Foreign, Commonwealth, and Development Office and the AgResults Steering Committee. Rockville: Abt Associates; 2021.

Download references

Acknowledgements

This paper is based on the authors’ experience evaluating the multi-donor AgResults initiative under an evaluation contract held by Abt Associates Inc. It has benefited from insights and experience of the AgResults in-country project managers in Nigeria, Kenya, Uganda, and Zambia; and the AgResults Secretariat. It has also benefitted from the thought leadership of the AgResults Steering Committee representing the UK Foreign, Commonwealth and Development Office (FCDO); Department of Foreign Affairs and Trade, Australian Government; U.S. Agency for International Development; Bill and Melinda Gates Foundation; Global Affairs Canada; and World Bank. We thank colleagues who assisted with data collection and general support. Diane Ferguson provided editing services.

Funding

Funding for this work came from the United Kingdom’s Foreign, Commonwealth and Development Office.

Author information

Authors and Affiliations

Authors

Contributions

TN drafted the first version of the paper, led the development of the evaluation framework and the overall research initiative, led the evaluation of the Nigeria AgResults project, and co-developed the lessons learned topics with DM. DM was the qualitative lead for all evaluations, led the market development and gender evaluations, and co-developed the lessons learned topics with TN. JG was the quantitative analysis lead for all evaluations, and contributed to the lessons learned. BNE led the Kenya and Uganda evaluations and contributed to the lessons learned. All authors read and approved the final manuscript.

Authors' information

Dr. Tulika Narayan is an agriculture and development economist with more than 15 years of experience leading international evaluation and research projects. Dr. Narayan conducts economic analysis to support agricultural development, climate change, low-emission development and environmental policy, as well as evaluation of international development projects. Until February 2020, she was the research director for the AgResults initiative at Abt Associates Inc. Currently, she leads the Food Security and Agriculture research practice at Mathematica. She has a Ph.D. in agriculture and resource economics from the University of Maryland, College Park.

Dr. Judy Geyer is an economist with over 12 years of professional expertise in conducting program evaluation. As the quantitative lead and then co-research director for the AgResults Evaluations, she creates evaluation designs and sampling plans, and leads data analysis and reporting tasks. Her broader work portfolio includes evaluations of household and organizational behavior, as affected by housing assistance, disability insurance, income subsidies, and rewards-focused programs. She holds a Ph.D. in economics from Carnegie Mellon University.

Dr. Denise Mainville is an agricultural economist with 20 years of experience in international agricultural research, development, and outreach. Her work focuses on enhancing the complementarities between private sector activity and development outcomes relating to poverty alleviation and social inclusion with a focus on agri-food market systems development, gender, and results-based development approaches. She has a Ph.D. in agricultural economics from Michigan State University.

Betsy Ness-Edelstein is an evaluation and research specialist focusing on the agriculture sector. She has more than 12 years of experience designing and implementing mixed-methods evaluations in developing countries. As the co-research director for the AgResults evaluations, she oversees qualitative and quantitative data collection and leads several of the country-level evaluations. She holds an M.A. in development economics and international business from Tufts University’s Fletcher School of Law and Diplomacy.

Corresponding author

Correspondence to Tulika A. Narayan.

Ethics declarations

Ethics approval and consent to participate

Abt Associates, Inc., IRB #1 (HHS registration IRB000001281), reviewed and approved all study protocols, recruitment and informed consent procedures, and data collection instruments.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Narayan, T.A., Geyer, J., Mainville, D. et al. An evaluation framework and empirical evidence on the effect of pay-for-results programs on the development of markets for welfare-enhancing agricultural technologies. Agric & Food Secur 11, 25 (2022). https://doi.org/10.1186/s40066-021-00346-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s40066-021-00346-1

Keywords

JEL classification