An evaluation framework and empirical evidence on the effect of pay-for-results programs on the development of markets for welfare-enhancing agricultural technologies

Donors and international development organizations increasingly recognize that private sector investment and creativity are needed to enhance global food security. Pay-for-results schemes are receiving greater attention as a means to catalyze private sector investment in sustainable, inclusive markets for goods and technologies that achieve food security and agriculture development goals. In pay-for-results schemes, the development organization promises prizes to private sector actors for achieving pre-specified goals. We describe an evaluation framework to help development organizations learn from both successful and failed pay-for-results projects to achieve agriculture and food security outcomes. Applying the evaluation framework, we describe the findings from four pay-for-results projects sponsored by AgResults, a multilateral initiative funded by development organizations from four countries (Australia, Canada, the UK, and the US) and the Bill & Melinda Gates Foundation. The lessons highlighted from these examples illustrate the importance of structuring the prize to encourage the creation of competitive agricultural markets; aligning the prize structure with the development goal of improving smallholder farmers’ food security; and constructing a theory of change that reflects a thorough understanding of the baseline market, enabling environment, and underlying assumptions about competitors’ response to the prize. Our work has several policy implications: Under certain conditions, pay-for-results mechanisms can help develop competitive, smallholder-inclusive agricultural markets and reduce food insecurity. Prize competitions offering multiyear, proportional prizes are more conducive than grand prizes to fostering the development of competitive agricultural markets. The enabling environment plays a significant role in pay-for-results mechanisms’ success or failure. Private sector-led actions alone may not be sufficient to adequately address the targeted development challenge. Under certain conditions, pay-for-results mechanisms can help develop competitive, smallholder-inclusive agricultural markets and reduce food insecurity. Prize competitions offering multiyear, proportional prizes are more conducive than grand prizes to fostering the development of competitive agricultural markets. The enabling environment plays a significant role in pay-for-results mechanisms’ success or failure. Private sector-led actions alone may not be sufficient to adequately address the targeted development challenge.


Background
In their 2010 meeting, G20 leaders called for the development of pay-for-results (PfR) mechanisms to achieve development goals in the food security and agriculture sector harnessing "the creativity and resources of the private sector" [1]. Two years later, the governments of Australia, Canada, the United Kingdom, and the United States, in partnership with the Bill & Melinda Gates Foundation, pledged funds to establish AgResults. AgResults is an initiative to design, implement, and evaluate PfR "challenge projects" to develop markets for welfare-enhancing agricultural technologies. Compared to traditional approaches which try to "prime the pump" for market development by raising awareness of and demand for welfare-enhancing agricultural technologies through subsidized distribution of those technologies, AgResults seeks to stimulate private sector investment in the development of smallholder farmer-inclusive commercial markets for these technologies. Insofar as these markets imply increased adoption of these technologies by smallholder farmers and poor households, they are expected to enhance their food security outcomes. This approach is also expected to have a more sustainable impact and cost less, because it develops markets.
Although AgResults is the first to test PfR schemes in the food security and agricultural sector, there is emerging evidence that the use of PfR schemes has broken down implementation barriers and driven progress on intractable social challenges in diverse sectors (see Meuth Alldredge et al. [2] for an evidence review). Mendelson et al. [3] and Suthar et al. [4], for example, document the success of PfR efforts in the health sector. Payment for environmental services has been used to pursue climate goals with evidence of efficacy in increasing forest cover [5,6]. In the energy sector, PfR schemes are also common ways to incentivize the adoption, sale, and use of climatefriendly energy technologies, and to promote innovation across the energy supply chain [7].
PfR schemes for agriculture market development are relatively new, and there is still much to learn from both successful and unsuccessful attempts to use them. To investigate the promise of PfR schemes for agriculture development, AgResults provides a rich portfolio of independent challenge projects in a variety of countries. As of 2021, AgResults is a US$152 million initiative with four concluded challenge projects, four additional challenge projects underway, and two projects in the design phase. Each AgResults challenge project has a common goal: to incentivize the private sector to develop and disseminate welfare-enhancing agricultural technologies in environments where technology uptake was limited, for example, due to uncertain or risky operating environments, credit constraints, lack of information, an unsupportive enabling environment, or other factors. To incentivize the private sector to overcome these market constraints in each of these contexts, each AgResults challenge project introduced cash prizes to targeted market players (or "competitors") that achieve pre-specified outcomes. The theory is that, if a prize is well designed and attractive to private sector competitors, they will creatively find solutions to constraints that otherwise inhibit the development of a market for the technology. Like other PfR schemes, AgResults provides payments to competitors only if and after they achieve the pre-specified outcomes; these outcomes are often expressed in terms of volume of output or sales of the technology or the output that it allows the farmer to produce. The expectation is that, if the private sector is successful, the resulting market for the technology will be self-sustaining after the competition ends. These prize competitions fall in the class of what McKinsey and Company [8] call "market stimulation prizes, " which includes grand prize competitions, advance market commitments, and social impact bonds (SIBs).
To the best of our knowledge, there are several papers that have evaluated PfR approaches but there is no evaluation framework for PfR schemes other than one proposed for evaluating grand challenges [9]. The PfR literature includes evaluations of grand prize challenges, advance market commitments, and social impact bonds. Grand prize challenges typically incentivize technology development, such as vaccine R&D [10], and there are few very evaluations of these types of challenges, perhaps because the prize is paid out only when the challenge is met which is considered as adequate evidence. SIBs mostly exercised in high-income countries, reward delivery of welfare services [11]. The literature on SIB makes an argument about the importance of evaluating social impact bonds, but does not put forth a formal framework to evaluate them [12,13].
Similar to SIBs and grand challenges, the AgResults projects are focused on achieving a development goal such as improved health through reduced aflatoxin contamination (Nigeria Aflasafe) and enhanced nutrition (Zambia biofortified maize and Uganda quality legume seeds), reduced post-harvest losses through improved on-farm storage (Kenya on-farm storage), and improved farmer incomes (Nigeria Aflasafe, Uganda quality legume seeds, Kenya on-farm storage) [14][15][16][17]. However, as noted above, these projects also aim to develop an inclusive market for the technologies.
We propose an evaluation framework for such market development PfR mechanisms and applying it to AgResults. We demonstrate how the evaluation framework provides insights about both completed and unsuccessful projects regarding markets for aflatoxin-reduced maize in Nigeria, improved legume seeds in Uganda, improved on-farm storage devices in Kenya, and biofortified maize in Zambia. All four projects were ultimately intended to impact smallholder farmers' food security outcomes by increasing their farm incomes and/or by increasing consumption of nutritious (vitamin-A fortified maize and legumes) and safe (aflatoxin-reduced maize) foods. Although the Uganda and Zambia projects were terminated due to lack of competitors' success in achieving pre-specified award criteria, all four AgResults projects discussed in the paper provide important insights into the unique challenges and lessons for designing PfR for market development. We synthesize and discuss lessons learned from our evaluations of four AgResults technology adoption projects. These lessons will help inform future PfR mechanisms to foster development of markets for welfare-enhancing technologies by answering questions about the development goals that PfR is best suited for, private sector actors that the PfR should incentivize and how the PfR's prize should be structured.

Method: evaluation framework
The overarching goal of our evaluation framework is to learn about the three key components of PfR mechanisms that aim to increase adoption of innovations by market creation: the development goal, the prize design, and the theory of change. The development goal of such PfR mechanisms is typically to enhance household wellbeing by the creation of a self-sustaining market. A PfR's prize structure rewards competitors that achieve pre-defined outcomes toward the development goal to develop markets for smallholder welfare-enhancing technologies. The prize design defines the specific outcomes that will trigger payout, the criteria against which those outcomes are judged, and the timing and value of the prize payments. The design also indicates the third-party verification process that confirms whether the competitors achieved the outcomes before the prize is awarded. Underpinning the prize design is a clear theory of change that articulates the expected causal linkages between (1) the prize structure, (2) the competitors' expected investments and activities in response to the prize structure, (3) the outcomes-both firm-and market-level-of those investments and activities, and (4) impact on the development problem.
We offer an evaluation framework designed to assess whether the development goal was achieved, whether the prize structure was appropriate, and whether the theory of change was correct. The evaluation framework proposes a market analysis, a causal impact analysis, and a cost-effectiveness analysis to answer several evaluation questions. Table 1 presents these evaluation questions, and the associated evaluation metrics that capture all elements of the PfR. Several of the metrics, marked "FS" for "fail-safe" are useful even in the case where no prizes are paid out due to the failure of any competitor to progress. In the next section of this paper, we apply this framework to share the evidence from four AgResults PfR projects. After, we illustrate how the framework lends itself to synthesizing lessons learned by looking at the patterns of results across projects.

Market systems analysis
An in-depth qualitative market systems analysis is critical to assess whether or not the PfR initiative catalyzed development of a sustainable market for the targeted technology and/or its product. Market development, by nature, generally produces changes across many levels or entities within a value chain. With many systemic changes-some exogeneous and others directly affected by the development project-it can be difficult to attribute any specific change to the development project itself. Therefore, the market systems analysis should encompass a qualitative industrial organization framework that systematically assesses relationships between the market's underlying conditions, firms' strategic behavior, and the market's structure and performance, before the PfR and after it to determine if PfR altered these elements. To capture the impact of the project on these elements, we propose metrics that describe market structure, performance, and sustainability drawing on structure-conduct and performance paradigm to characterize markets [18]. To gather data for these metrics, we suggest ex-ante, periodic, and ex-post interviews with private sector competitors, their suppliers and customers, and relevant market and regulatory experts; along with prize award verification data and results of the farm-or consumer-level impact analysis. A comparison of these metrics before and after the PfR helps assess if the PfR engaged private sector actors, if it incentivized them to address the underlying the constraints limiting market development, and if it created a market that is inclusive and sustainable. In the case of AgResults, the comparison of these metrics before and after the projects helped us to assess whether the PfR helped to establish markets for smallholder produced aflatoxin-free maize in Nigeria, for improved legume seeds in Uganda, for pro-vitamin-A maize grown by smallholders in Zambia, and for improved on-farm storage for smallholder predominant areas in Kenya.

Impact assessment
To the extent that PfR's ultimate aim is to develop markets that improve livelihoods and well-being of rural households, the evaluation should focus on the ultimate intended beneficiary: such as the consumers (including farmers) whose maize meal is contaminated with aflatoxins or who are Vitamin-A deficient, the farmers whose legume harvests are poor due to a lack of access to quality legume seed, or the poor farmers whose food security is undermined by excessive post-harvest losses of the grains they grew. To evaluate PfR's impact on the intended beneficiary, we propose metrics about the PfR's effect on farmers' knowledge about the innovation, adoption of an innovation, and farmers' well-being and the differential impact on women and other marginalized groups, since by their very nature, market systems development initiatives have the potential to exclude women given that women typically have lower levels of market integration than men [19].

Cost-effectiveness analysis
Cost-effectiveness analysis is useful to compare the efficiency of different program strategies, though it is important to consider the differences in implementation contexts, challenges, and complexity. If traditional development projects intend to achieve the same results in the same geographic regions, the associated projects' expected costs per targeted result could be compared with those of AgResults. Direct comparisons of costeffectiveness findings are best focused on alternative approaches to promoting uptake or market development of similar technologies in similar environments, rather than across implementation contexts and technologies. For the AgResults evaluations, we used the cost-effectiveness analysis to estimate projects costs per pre-specified outcome directly attributable to the project, within the timespan of the project. For example, as the goal of the Kenya project was to promote adoption of improved onfarm storage, we divided project cost by the number of smallholder farmers whose adoption of on-farm storage solutions was attributable to the project, based on findings from the impact analysis. To the extent that technology adoption continues to increase after the program, and that increase is due to the program, the cost-effectiveness measure calculated at the close of the project is an underestimate of its true cost-effectiveness. Therefore, we also recommend conducting forward-looking analysis to assess cost-effectiveness analysis under different scenarios of market development and sustainability. 1

Results: empirical application and evidence
The first four projects of the AgResults initiative targeted key food security crops in Africa. Two of the projects were completed (Nigeria and Kenya), and two were terminated early (Uganda and Zambia). Table 2 summarizes the development goal, prize design, and theory of change for each project. We apply the proposed evaluation framework to provide evidence of the effectiveness of AgResults' PfR approach. The balance of this section summarizes our findings on the evaluation metrics in the market analyses, impact analyses, and cost-benefit analyses to answer the evaluation questions there-in. The Uganda and Zambia projects have fewer data, owing to the fact that no competitor made sufficient, timely progress toward the award criteria.
The findings from our market analyses of the first four AgResults projects are summarized in Table 3 [20][21][22]. The positive findings are that the prizes can motivate a significant number of private sector actors to engage in a new market, and that the new market can be inclusive of women and smallholder farmers. The market is more likely to be sustained after the project ends if the private sector actors have a business interest in continuing to engage in the market. While initial supply-and demand-side constraints can be daunting and not all are addressed by competitors, the lack of a supportive environment is the biggest threat to the market success. Requirements for an enabling environment include a general trust between buyers and sellers, access to credit, and the absence of government investment in markets for substitutes.
The "R" in PfR, i.e., the results triggering award payment, are rarely the full measure of progress toward a development goal. A causal impact analysis is an opportunity for prize sponsors to examine a broader set of outcome measures describing progress toward a development goal. In addition, causal impact analysis compares the results achieved to the results that would have been achieved in the absence of the PfR project (rather than assuming that no results would have been achieved absent the PfR project).
Despite increased attention to random assignment evaluations to demonstrate the effectiveness of new development strategies, causal impact analysis of market-level interventions remains rarely practiced, because it is especially challenging. Consumers, farmers, or intended beneficiaries are only one component of a market systems approach: a systems change, by nature, generally produces changes across many levels or entities within the market system. The impact of market systems approaches on households, consumers, or other vulnerable groups is not easily evaluated using field experiments, because it is difficult or impossible to define and protect a comparison group that is not affected by some level or entity within the market system. In Nigeria, we compared farmers involved in AgResults to a comparison group of farmers selected by us after the intervention was under way and used propensity-score weights to improve comparability of the treatment and comparison groups on stable characteristics not affected by the intervention. In Kenya, we used a difference-in-differences impact evaluation design of farmers who adopted the technology and matched farmers who did not adopt the technology. We did not conduct a causal impact analysis for the Zambia or Uganda projects that ended early. Table 4 summarizes the findings from our causal impact analysis for the projects in Nigeria and Kenya. The causal impact analysis showed that adoption of the technology was considerable, though less than otherwise suggested using the project monitoring data. In Nigeria, technology adoption improved farmer income, while in Kenya, it did not. Although, in Nigeria, PfR did not raise farmers' awareness about the full health benefits of the technology.
The cost-effectiveness analysis from the four projects revealed that, on average, prize award costs roughly equaled management costs. This suggests that, if a sponsor hosts a portfolio of projects, the management and verification costs alone will be a significant proportion of the budget. The two successful projects, Nigeria and Kenya, spent vastly different proportions of their budgets on the prize award: this is due to high verification costs in Nigeria and also raises the question of whether the prizes were too generous in Kenya. The cost per farmer reached in Kenya ($39) is high compared to the cost of an average storage solution (near $2.50); the high cost per farmer reached in Nigeria ($134) may be warranted owing to the difficulty in changing farmer cultivation methods if verification costs are very high; a failed PfR could cost more than a traditional technical assistance project if no competitor realizes the prize criteria or if the newly created market is not sustained beyond the life of the project. Table 5 summarizes our cost-effectiveness results for the two completed projects. Guaranteeing a minimum return on procurement of Aflasafetreated maize will reduce traders' risk of investing in Aflasafetreated maize market, helping to create a supply base that will serve the market as demand develops

Kenya on-farm storage project
Create market for smallholder farmer-appropriate improved on-farm storage technologies to reduce post-harvest loss and improve food security Proportional and milestone prizes based on competitors' market share in sales of total storage capacity as long as they met minimum threshold of sales Prize incentives will de-risk investment needed to overcome major barriers to development of improved on-farm storage market, leading to establishment of a competitive, smallholder farmerinclusive and sustainable market

Zambia biofortified maize project
Reduce vitamin-A deficiency by creating a market for biofortified pro-Vitamin-A maize Per-unit prizes to participating maize millers and seed companies based on sales of Pro-vitamin-A maize milled flour and maize seeds Guaranteed returns will increase seed companies and millers' appetites for risk of investing in pro-vitamin-A maize market, leading to development of both demand for and supply of provitamin-A maize Uganda legume seed project Improve yield, consumption, and income and yields from smallholder farmers' legume production by strengthening the market for quality legume seed Participating seed companies receive annual prizes for increasing sales of quality legume seed above their own expected baseline levels of sales Guaranteed returns on sales of legume seed will increase profitability and reduce risk to seed companies of investing to production and sale of legume seed, sustainably increasing levels of legume seed market sales Table 3 Findings from the market analyses Did the PfR motivate the private sector to overcome constraints to developing an inclusive market?

Demand-side and supply-side constraints to market development [FS]
Nigeria: Farmer's knowledge constraints remained somewhat unaddressed by PfR. While farmers learned about the Aflasafe as a way to gain access to premium markets, limited knowledge of the adverse health impact of aflatoxins and Aflasafe as a solution remained a constraint Kenya: Key constraints to market development were addressed by competitors. Smallholders' access to improved on-farm storage increased, they learnt how to use it and found value in using it because it mitigated the need to use pesticide dust to prevent storage loss Uganda: Demand-side and supply-side constraints remained unchanged since the PfR was not successful. PfR was primed to address supply-side constraint by providing seed companies incentive to invest in legume seed production. However, without quality certification they could not distinguish their product and the PfR could have paid out for increase in sales of even poor-quality seeds Zambia: Demand-side and supply-side constraints remained unchanged since the PfR was not successful

Role of enabling environment in promoting or constraining development of market [FS]
Nigeria: Enabling environment did not limit market development.
Limited enforcement of regulatory limitations on aflatoxins in maize did not constraint PfR outcome because markets existed that were willing to pay a premium price for aflatoxin-free maize Kenya: The enabling environment was neutral, even supportive Uganda: Lack of seed quality certification remained a key constraint to market development. A weak national seed certification system meant that buyers had no effective means of differentiating between high-and low-quality legume seed on the market, which depressed the price that they were willing to pay for legume seed in general Zambia: Low demand and government subsidies for mainstream white maize depressed incentives for PVA maize. Ongoing government involvement in the standard/white maize market may have depressed investment in the PVA maize market. Maize millers had little experience in developing supply or demand for a differentiated product

Type of market actors that compete and type that win [FS]
Nigeria: Diverse types of market actors participated and won prizes. PfR prize structure motivated participation by a large number of competitors all of whom won prizes. These competitors sold the maize to largescale buyers such as feed and flour mills and multinationals such as Nestle and small and medium-scale buyers including the poultry producer market and feed market Kenya: Larger companies participated. Of the nine companies that participated six made significant investments to tailor and develop their supply distribution networks and won prizes. The participating companies were generally large and were either manufacturers of the technology or distributors Uganda: None of the competitors won prizes. Seven seed companies participated but none of them won prizes Zambia: Very few competitors won prizes. Three seed companies and eight millers participated. Of these, two seed companies achieved the outcomes and won prizes. None of the millers won prizes: the millers together procured and marketed less than one metric ton of PVA maize over three project years, whereas the minimum threshold that would have qualified an individual miller for an AgResults incentive payment was itself one metric ton of PVA maize

Private sector actors' business cases for investment in market [FS]
Nigeria: Some final buyers pay price premiums for aflatoxin-free maize making it profitable for maize aggregators to acquire it, although not necessarily from smallholder farmers. Poultry feed market and export market have strong demand for aflatoxin-compliant maize, offering a highmarket value to maize aggregators. Other than the relationships developed through PfR there is economic incentive for the maize aggregators from smallholder farmers. That said, some of these relationships have sustained Kenya: Smallholders farmers are buying improved on-farm storage in large numbers and at prices that are profitable. On-farm storage suppliers maintained their business model of selling storage solutions to farmers. Farmers gain by not having to apply pesticides that they perceive to be unhealthy and unwieldy to use Uganda: Profit margins were slim in producing quality legume seeds. Potential seed distributors had little case for investing in improved legume seed markets, owing to very slim profit margins on account of a weak national seed certification system Zambia: Maize millers did not see a role in developing the demand for Vitamin-A enhanced maize. Demand for PVA maize was growing among urban consumers but the maize millers had little experience in developing the supply for the differentiated product and could not capitalize on this demand

Percentage of target product produced or transacted in the relevant market area
Nigeria: Niche market for aflatoxin-free maize emerged as a result of the PfR. From 1 private sector market actor aggregating Aflasafe-treated maize to 24; from 1700 MT of Aflasafe-treated maize at baseline to 82,355 MT five years later (0.4% of maize marketed nationally); 100% of sales through commercial market channels Kenya: PfR led to significant sales of diverse types of improved onfarm storage. Number of commercial suppliers of improved on-farm storage increased by 6 during the project, with an additional 3 new suppliers in the two years following project; 334,000 MT of improved on-farm storage sold to 220,000 smallholder farmers; emergence of commercial distribution channels through which most sales are made Uganda: N/A Zambia: Very small volume of milled maize was transacted but with waning interest. Eight maize millers processed 993 MT of PVA maize, of which 82% was marketed through commercial channels to middle-class, nutrition-focused consumers

Lessons learned
The evaluation metrics summarized in the previous section directly answered the research questions posed for the market, impact, and cost-effectiveness analyses. Synthesizing findings across these analyses, the evaluator has an opportunity to identify some lessons learned for the research community regarding the design of PfR. For example, integrating findings with respect to the key constraints for market development (market analysis) and impacts on farmers (causal impact analysis), researchers are able to reflect on whether the pathways in the theory of change proved correct, allowing the full set of development goals to be met. In this section, we discuss our synthesis of the evaluation findings from AgResults in the form of lessons learned. These lessons are organized by PfR component: development goal, prize design, and theory of change.

Development goals
Based on the evidence from these four projects, we summarize the circumstances in which prize competitions may be more likely to achieve their development goals.

The enabling environment must be adequate to support development of a market
Two AgResults PfR mechanisms failed at least in part as a result of enabling environment constraints that existed before the PfR mechanism was put in place. Uganda lacked an effective seed quality certification system. Without the ability to distinguish quality of seeds or to control counterfeit labels, seed companies lacked incentive to invest in producing or marketing high-quality legume seeds. In Zambia, extensive government involvement in the market for white maize depressed investment in the market for PVA maize, which was a close substitute for white maize. On the other hand, in Kenya and Nigeria, the enabling environment was largely adequate to support development of a private sector-driven market and did not impede the development of private quality verification initiatives where public quality verification did not exist. In Kenya, the Kenya Bureau of Standards had a transparent process to approve the technologies before allowing them into the market, and competitors were able to brand and market their products without extensive problems of counterfeiting. When, at project inception, there was no way to test whether on-farm storage devices were effective against the large-grain borer, AgResults itself contributed with the development and deployment of a testing protocol, thereby avoiding substantial delays due to lack of a means of verifying product quality. In Nigeria, regulations already existed for aflatoxin standards, although they were not broadly enforced, and AgResults verification of Aflasafe-content served as de facto quality verification for buyers that sought aflatoxin-complaint maize. These AgResults projects show the importance of the enabling environment not only in defining and enforcing policies and product standards, but also in not impeding private sector-or donor-led initiatives in cases where public initiatives were lacking.

Participation of smallholder farmers, women and other marginalized groups relative to their presence in the relevant market area
Nigeria: Smallholder farmers engaged extensively; women's participation was low. Smallholder farmers were responsible for nearly 100% of Aflasafe-treated maize produced because the PfR management was requiring it as part of informal prize requirement; women's participation varies widely but averaged 19% in final year of project (compared to 50% participation of women farmers nationally). Outside of the PfR there are no real incentives for the aggregators to work with smallholder farmers and a greater incentive to work with larger farmers who can fill their quantity and quality needs more easily Kenya: Women had a higher level of adoption. Women-headed households also adopted the improved on-farm storage 3.7 percent greater than other households Access to credit is another aspect of the enabling environment that is particularly critical for PfR mechanisms, given that they do not pay competitors up front. In all AgResults projects, constrained access to operating or investment capital consistently limited the scale of investment, although the degree to which it impeded competitors differed across projects and competitors.  Farmer well-being Nigeria: Smallholder farmers increased their incomes and to some extent the intake of aflatoxin-free maize. Smallholders had 16% higher incomes (increase of $318 annually) from maize among smallholder farmers adopting Aflasafe; only 13% percentage point increase in daily maize intake of Aflasafe-treated maize (from zero) when expectation was complete shift to Aflasafe-treated maize;If the farmers had greater awareness of the health benefits of Aflasafe and/or the adverse health impact of aflatoxins, the uptake and impact could have been greater with potentially some tradeoff in increasing consumption and reducing the extent of sale Kenya: Farmers did not benefit financially from adopting improved on-farm storage. Evaluation found no impact on household food security or income from maize. Qualitatively, farmers reported using pesticide dust to mitigate against post-harvest losses.

Tradeoffs can exist between market impact and development impact objectives
Since the incentive structure influences how competitors interact with their suppliers and buyers who are usually the ultimate intended beneficiaries of PfR mechanisms, potential tradeoffs between market impact and development impact should be identified and planned for at the outset. In Nigeria, a robust supply base of smallholder farmers was developed by project competitors, yet these smallholder farmers evidenced little awareness of aflatoxins as a health problem or Aflasafe-treated maize as a means to protect them against aflatoxin exposure, despite the fact that reducing smallholder farmers' aflatoxin exposure was one of the most important intended impacts of the project. Consistent with their business incentive, competitors did not tend to emphasize the health implications of aflatoxins when they interacted with farmers, because they had an incentive to procure as much of the harvested, treated maize as possible. Farmers reacted to the prize incentives in the markets by selling almost all of their treated maize. Our evaluation found that the project had minimal impact on smallholder farmers' consumption of Aflasafe-treated maize, increasing it by only 0.02 kg per day, or 13% of their daily consumption, in large part because smallholder farmers typically did not apply Aflasafe to all of their maize. As a result, the project generated limited health impacts for poor, rural farmers. In situations where competitor incentives may conflict with the project's intended development impact, and where adjusting the outcome on which prizes are based cannot feasibly eliminate this conflict, there is arguably room for complementary activities on the part of the project or some other public actor to provide information to project beneficiaries that will help offset this unintended impact.

Prize designs
The evaluation metrics from the four AgResults projects give rise to several lessons learned about appropriate prize designs.

Prize structures with more frequent payouts enable competitors to increase scale by re-investing returns
AgResults projects are distinct from other PfR mechanisms, because they award prizes at multiple stages, rather than relying entirely on a single end-of-project prize payout. Our evaluations showed that more frequent payouts enabled competitors to increase the scale of their activity by re-investing their proceeds; this was particularly important given the capital constraints that these competitors faced. With the exception of the Uganda project, which never paid out a prize, competitors in all AgResults projects consistently reinvested a large proportion of their prizes to increase their scale of operations. For example, in Nigeria, competitors commonly used their incentive payments to finance input packages for their farmers. In Kenya, where only three competitors earned the first milestone prize, the recipients of these payouts gained a competitive advantage, because they could reinvest their proceeds in their operations and make further gains in relative scale.

Table 5 Findings from the cost-effectiveness analysis
Cost-effectiveness analysis: Was the PfR cost-effective?

Proportional and per-unit prizes attract multiple private sector actors
All four projects demonstrated that PfR mechanisms can attract multiple private sector actors. While no AgResults project offered a winner-take-all prize, precluding empirical observations on their effects in the AgResults context, economic theory holds that per-unit or proportional prizes attract more competitors than winner-take-all prizes [23][24][25]. 2 Competitors' reflections on how the prizes motivated their participation in AgResults and investment in the targeted market support this idea. Furthermore, across projects, competitors reflected that per-unit prizes increased expectations of positive returns from investment in the market relative to proportional prizes in which payment partially depends on other competitors' performance. In Nigeria, which offered a per-unit prize, a wide range of private sector actors participated including some that were quite small. In Kenya, in contrast, relying on a proportional prize to motivate competitors (after the initial threshold was reached) induced a strategic competitive behavior that was unique to that project and market. In general, having more competitors can lead to a larger pool of winners who can remain viable market actors at the close of the project, which is conducive to creating a competitive market structure.

The prize should link directly to the desired development goals
Donors invest in developing a market for the technology, because they believe that the technology will improve smallholder farmers' welfare. The prize design can define qualifying parameters to increase that effect on welfare. For example, in Kenya, the PfR mechanism introduced qualifying criteria that ensured that competitors would be rewarded for sales of technologies that were best suited to smallholder farmers: only improved on-farm storage devices of 540 kg or less (the average amount of maize consumed by an average family in a year) qualified for prizes.
In contrast, in Zambia, the prize did not promote outcomes that would directly benefit smallholder farmers. Competitor millers earned prizes based on their sales of milled PVA maize, which is typically purchased by better-off urban consumers. Poor farmers who have higher prevalence of vitamin-A deficiency rarely buy milled maize. Similarly, schools and health clinics serving nutritionally vulnerable populations usually buy un-milled maize due to its lower cost. Thus, by rewarding sales of milled maize specifically, the miller competition in Zambia did not directly promote procurement or sales of PVA maize to the most nutritionally vulnerable farmers or consumers, who were the project's ultimate intended beneficiaries. Furthermore, when the Zambia project introduced prizes to seed companies, it did not specifically reward seed sales to rural farmers through commercial channels; instead, it rewarded sales to all domestic entities including government and non-profit actors that would subsidize their distribution of the seed to farmers, undermining the development of a commercial market for the seed.

Right sizing the prize amounts and results benchmarks requires ex-ante cost-effectiveness analysis
Estimating the social returns to a project can be difficult ex-ante, but should be conducted using scenario analysis to assess if the ranges that yield positive results are achievable by the PfR. Deeper investigation of the theory of change with careful fieldwork can support this analysis. In the Kenya project, the cost incurred to achieve that smallholder farmer adoption was high with no measurable benefits to farmers, because the farmers already had another method of reducing post-harvest losses (pesticides). 3 Evaluations of initial qualitative research had already unearthed this likelihood, which underscores the importance of this work. Such analysis was done for Nigeria where clear assumptions were made about the area that farmers may treat with Aflasafe, the yield results they may achieve, and the prize premium that will increase their incomes.

Theory of change
The evaluation metrics also give rise to an important lesson regarding the theory of change behind PfR projects.

The theory of change should be based on careful market and behavioral research and its assumptions should be clearly articulated and tested in early phases
An initial market systems analysis is essential for all aspects of the PfR mechanism design process-including identifying the development problem, its technological solution, and the competitors, and designing the incentive structure. The overriding objective is to identify the key constraints to developing a market for the target technology, to assess the business case for potential competitors, and identify the potential financial and economic returns for smallholder farmers' engagement with the technology.
The market systems analysis allows project designers to ground the theory of change in the project's market context. This grounding offers early opportunities to vet and strengthen the project design, especially when it is coupled with a detailed theory of change that describes how the competition will lead to the development of a sustained market and development impact among its target beneficiaries. In Uganda, for example, an underlying assumption that providing incentives to increase sales of improved legume seed would increase the availability of quality seed to farmers was challenged prior to the project's initiation, following a market systems analysis by the evaluator. This led to major changes to the PfR mechanism's design to account for this issue, although ultimately the project was canceled when the alternative quality verification system failed to gain public sector approval.
The theory of change rests on a set of assumptions about how the PfR mechanism would work; these assumptions should be clearly articulated and, to the extent possible, substantiated. Pilot or early phases of PfR mechanisms offer opportunities to evaluate assumptions about how the market actors will achieve the development goal. While it is difficult or even impossible for project sponsors to understand all aspects of a market system, stating critical assumptions at the beginning of the project helps to articulate expectations, realization of which can be monitored throughout the project and used as a basis for project modifications. The theory of change also guides monitoring and evaluation throughout the prize competition's implementation and provides information to project sponsors on the need for modifications to the competition, while it is in process. In all four projects described in this article, at least one critical assumption underlying the project's theory of change did not work out. For example, in Kenya, the assumption that onfarm storage products would reduce post-harvest loss did not work out, because farmers were already controlling losses using pesticide dust; this significantly reduced the financial benefits realized by farmers despite impressive uptake of on-farm storage products as a result of the project. In Nigeria, the assumption that the project would increase farmers' consumption of Aflasafe-treated maize did not prove true; instead, they sold nearly all of the Aflasafe-treated maize that they produced and retained untreated maize for household consumption. Early identification and testing of these underlying assumptions might have offered opportunities to adjust the projects' designs to increase their impact given emerging results.

Discussion and conclusion
PfR mechanisms, as presented by the AgResults initiative, offer an innovative means to leverage private sector ingenuity for the realization of food security goals. In this paper, we have drawn on an evaluation framework that includes market analysis, impact analysis, and cost-effectiveness analysis to highlight lessons from the first four PfR mechanisms used to advance development goals related to food security and agriculture. We have also identified lessons learned from those evaluations. The experience of AgResults thus far suggests that PfR mechanisms can be an effective tool to address challenging development problems by promoting the development of markets that promote and enable widespread adoption of promising agricultural technologies-under certain conditions. We conclude by offering insight into several critical questions about the suitability and design of PfR mechanisms.

What development contexts are best suited for PfR mechanisms?
Market stimulation prizes work when the development problem is situated in a context where the enabling environment for the private sector can support emergence of a market for the promoted technology. The PfR approach rewards private sector participants for their success in creating a market for a given technology, and in order for this approach to be effective, addressing the constraints inhibiting the market's growth must be within the "manageable interest" of the project's PfR participants. In the AgResults projects we have discussed, for example, competitors were in a good position to overcome smallholder farmers' limited awareness of the technologies and to create effective procurement and distribution systems for their products; none of the projects were hampered by a weak enabling environment. In Uganda, however, the lack of an effective system to certify seed quality was an enabling environment constraint that undermined the development of supply and demand for quality legume seeds, and that was not in the manageable interest of the private sector participants in the project.

What types of private sector actors should a PfR mechanism incentivize?
The first four AgResults projects demonstrate the importance of targeting the prize competition to competitors that are both well positioned to address the key constraints limiting the development of a market and have a fairly strong business case for staying in the new market after it is established.

How should the prize be structured?
Per-unit and proportional prizes can attract a large number of market actors, and relatively, frequent prize payouts enable and increase private sector investment. While the prize should be large enough to motivate private sector investment, it should not be so large as to offset tepid interest among potential competitors that reflects the lack of a viable business case for those competitors. Furthermore, the prize needs to be modest enough for its development impact to be cost-effective and yield a net positive return on donor investment. An ex-ante cost-effectiveness analysis can help define minimum thresholds before the prize is paid out to ensure a positive return on donor investment.
The duration of the competition-the number of years over which prizes are paid-should reflect the amount of time it takes to make the needed investments to overcome key constraints. However, the timeline should be aggressive enough that results are achieved at least as quickly as they would be under alternative donor mechanisms. Graduated reductions in prize incentives in later stages of the competition can help orient competitors to overarching market conditions without completely nullifying the incentive offered by the prize to invest in the nascent markets.
To conclude, we acknowledge that policy makers and researchers alike still have much to learn about using PfR strategies to stimulate the growth of markets that provide welfare-enhancing technologies. We hope that the use and adaptation of our evaluation framework will help the policy and research community further enrich its learnings about future PfR programs and their optimal design.