Can results-based prizes to private sector incentivize technology adoption by farmers? Evidence from the AgResults Nigeria project that uses prizes to incentivize adoption of AflasafeTM

Abstract

Background

The AgResults initiative tests the efficacy of results-based prizes to scale-up smallholder technology adoption. In Nigeria the project awarded a $18.75/ton prize for private sector actors who aggregated maize from smallholders that was treated by Aflasafe—a biocontrol that addresses liver cancer-causing aflatoxin contamination in maize. This paper examines the impact of AgResults initiative on smallholder farmers.

Methods

This evaluation estimates the causal effect of the AgResults program on farmer outcomes by comparing survey data from AgResults farmers to survey data from a matched comparison group of farmers. To improve balance, we use propensity score weights. In considering inestimable selection bias, we describe several key considerations, including the inclusion of comparison areas for treatment post-evaluation.

Results

The project increased Aflasafe adoption by 56% points, farmers earned 16% more net maize income on average. However, the majority of farmers in villages engaged by the project did not know about Aflasafe’s health benefits. This suggests that complimentary donor-directed efforts may still be needed to generate general awareness about the technologies whose benefit is not immediately visible to the smallholders. With the prize focused on aggregation, private sector actors may have reduced incentive to raise awareness about Aflasafe’s health benefits in case farmers held back Aflasafe-treated maize for consumption.

Conclusions

This paper highlights the potential of results-based prizes to engage the private sector in solving development problems. However, it also equally highlights the gaps that such an approach may have, arguing for the need for having complimentary efforts to address those gaps. This is particularly the case when the technology’s benefits are not perceived by the consumer (aflatoxins are not visible to the eye, and the health benefits are not immediate), or when the technology results in positive externalities (final consumers of Aflasafe-treated maize also benefit). Broader consumer awareness needs to be raised to promote continued development of the market for Aflasafe-treated maize, aflatoxin standards need to be enforced, and aflatoxin testing needs to be more easily available.

Introduction

After the food crises of 2007–2008, there was a growing realization that donor resources were not adequate to meet agricultural development challenges. The AgResults initiative was launched at the June 2012 G20 Summit in Los Cabos, Mexico, as an innovation to boost private sector engagement in meeting these challenges. With funding from several governments—Australia, Canada, United Kingdom, United States, and the Bill & Melinda Gates Foundation—the AgResults initiative uses results-based prizes, also referred to as “pull mechanisms”, with the aim to draw on the resources and creativity of multiple private sector actors to drive agricultural innovation and delivery for smallholder farmers in developing countries. Pull mechanisms are a type of results-based payment mechanisms that bring multiple actors into competition to win prizes, typically proportional to the results achieved. Masters and Dalbeq [20] argue that proportional prizes are better suited to achieve agriculture development outcomes. Compared to grand prize challenges that typically incentivize technology development, such as vaccine Research and Development (R&D), and social impact bonds, mostly exercised in high-income countries, that reward delivery of welfare services [9, 10, 13], pull mechanisms mimic a marketplace. The winners receive payments relative to their ability to achieve the outcomes, which can be defined to meet the buyers’ need. Pull mechanisms can mitigate other drawbacks of winner-take-all prizes, which limits both entry into the competition and the effort there in [10]. In comparison, when competitors have different abilities, proportional prizes encourage greater entry and effort [5, 21, 26].¹ Freeman and Gelber [14], also find that multiple-prize structure results in higher aggregate performance than the winner-take-all payment. By awarding prizes to only one winner, despite the effort put in by other competitors, winner-take-all prizes can also lead to more unequal revenues and possible monopolistic pricing if other competitors drop out from delivering services after the competition [12].

Recognizing an opportunity to test pull mechanisms in agriculture, the AgResults initiative sponsors prize competitions to incentivize the private sector to develop and deliver innovative products to smallholder farmers in settings where markets for agricultural inputs, services, and outputs are underdeveloped or nonexistent.² At the time of this writing, it is a $152 million initiative comprising four completed and five ongoing projects. Each project pits multiple private sector companies into competition to win prizes, often proportional to their effort, if and only if they achieve predefined agriculture development results. The ultimate objective of the pull mechanisms is to encourage private sector investments in addressing the constraints that have limited the development of a market for beneficial agricultural technologies, and to create sustainable markets for these technologies.

The AgResults Nigeria project is the first ever pull mechanism applied in the agriculture sector. It focused on addressing contamination of maize by a naturally occurring toxin called aflatoxin, which is produced by fungi commonly found in African soils—Aspergillus flavus and Aspergillus parasiticus [17, 25]. Aflatoxins cause liver cancer with chronic exposure, and liver edema and death with acute exposure [8, 18, 22, 28]. Chronic exposure to aflatoxins is also associated with stunting [15]. Researchers at the International Institute for Tropical Agriculture and the U.S. Department of Agriculture developed Aflasafe, a biocontrol which can effectively address aflatoxins [3, 4]. However, most smallholder farmers in Nigeria are not aware of aflatoxins, which are invisible to the naked eye, or Aflasafe as a solution. Even if smallholders were made aware of aflatoxins and Aflasafe, Aflasafe is not economically viable for most smallholders to adopt without a price premium on aflatoxin-free maize and/or an increase in maize yield. Our qualitative research for this evaluation found that premium markets for Aflasafe-treated maize (AT maize) are limited but do exist: export markets, grocery chains and the poultry-feed markets pay higher prices for aflatoxin-free maize. However, it is hard for smallholder farmers to access these markets because of information and infrastructural constraints, including constraints to access Aflasafe. The AgResults Nigeria project aimed to address these underlying constraints, which had led to a missing market for Aflasafe and AT maize.

This paper presents the first evidence on the use of pull mechanisms to increase smallholder technology adoption and improve smallholder wellbeing. The evidence is based a study of smallholder farmers cultivating maize on at least one hectare of maize in the Kano, Kaduno, and Katsina states of Nigeria in 2017. The evaluation compares villages targeted by AgResults competitors in the first 3 years of the project, and comparison villages not yet targeted by AgResults competitors in the first three years (some of which were involved in AgResults in later years). The paper assesses if the pull mechanism in Nigeria increased farmer use of Aflasafe, improved smallholder income from maize, and created smallholder awareness about Aflasafe and aflatoxins. A separate qualitative evaluation describes how private sector Nigerian maize value chain actors reacted to the AgResults project in Nigeria and documents the development of the market for aflatoxin compliant and AT maize over the course of the AgResults project [23].

Background: the project’s theory of change

The AgResults Nigeria project sought to create a market for AT maize by temporarily increasing the expected returns for the private sector’s investment in the supply of AT maize by smallholders, with the hope that the investments sustain the market even after the incentives end. Before the project, Aflasafe was not easily available even though limited buyers were willing to pay a premium Aflasafe-treated maize. Some maize sellers were transacting with these buyers, but they were not sourcing the maize from smallholder farmers. The potential for winning cash prizes was expected to increase the profitability of successfully taking on the supply side risks of identifying a viable smallholder supply base, making Aflasafe available to these farmers, encouraging them to adopt it and aggregating and selling more AT maize. It was also expected to increase the number of actors aggregating and supplying AT maize. Specifically, the project rewarded participating maize intermediaries, or maize aggregators that procure AT maize from smallholder farmers by paying the aggregators a premium of US$18.75 per metric ton (MT) of AT maize aggregated. This payment corresponded to a premium of about 4.7% of the average price of maize, which we found to be approximately US$400 in Spring 2017. Relative to the cost of Aflasafe, which costs about $1.5 per kilogram, implying a cost of $7.50 per metric ton of maize, the premium more than covers the cost of Aflasafe per hectare.³

To qualify for the incentive payment, the maize needed to have more than 70% Aflasafe concentration based on tests by a third-party verifier. In addition, the project required, but did not verify, that the private sector actors (hereafter referred to as competitors) must aggregate the maize from smallholders.After aggregating the AT maize, it was expected that the competitors would sell to downstream buyers of maize—preferably buyers who are willing to pay a premium for maize with reduced aflatoxin levels.

The pull mechanism expected that these competitors would use innovative approaches to encourage adoption of Aflasafe by smallholders such as by providing access to credit and other inputs with a promise for better quality maize and better maize yields, access to premium markets or output sale guarantees.⁴ Competitors could have also motivated smallholders to adopt Aflasafe by raising their awareness about the health benefits to their families, causing them to grow more AT maize for both consumption and sale. However, there is also a countervailing incentive for competitors to not raise health awareness among smallholders so that farmers sell the AT maize to the competitor rather than saving it for consumption. Given the preponderance of smallholder maize farmers in the Nigerian population, and their risk of aflatoxin consumption, it is an important empirical question whether smallholder awareness of aflatoxins was raised because of the project or if the competitors instead encouraged them to adopt Aflasafe so that they could access premium markets or if they learned about both benefits [19, 24].

Regardless of the motivations for smallholders to adopt Aflasafe, the project expected to benefit smallholders in two ways. First, it expected to increase smallholder incomes by increasing maize yields (through improved access to inputs from competitors) and/or through higher prices at which smallholders sell AT maize. Second, it expected that farmers would have health benefits from consuming AT maize even if the farmers were not made aware of the benefits of consuming AT maize because the farmers consume what they grow.

The project did not focus on generating awareness more broadly among Nigerian consumers or increasing consumption of AT maize among Nigerians. This was because there were pragmatic concerns with generating awareness about the health impacts of consuming aflatoxin-contaminated maize without ensuring the availability of AT maize to meet the demand. It was expected that once the AgResults project ensured a steady supply of AT maize, government-driven efforts would be well-positioned to generate broader aflatoxin awareness and enforce existing aflatoxin-regulations especially given that Aflasafe production is ready to scale.

Motivated by the prize, 24 diverse competitors—seed producers, poultry-feed producers, maize aggregators, and social enterprises—participated in the competition, and worked with smallholders to produce AT maize over the 4 years of the project from 2014 to 2018. In this paper, we assess if these competitors created smallholder awareness about Aflasafe and encouraged them to adopt Aflasafe with a consequent increase in smallholder incomes and smallholder consumption of AT maize. We also assess if AgResults created awareness about aflatoxin health impacts among farmers and those responsible for cooking (typically the wives if the farmer is male), and whether it impacted smallholder farmer’s decisions to sell or consume aflatoxin-safe maize.

Methods

Evaluation design

We assessed the impact of the AgResults Nigeria project on the smallholder outcomes by comparing smallholders in villages targeted by AgResults to smallholders in other villages.⁵ The challenge of a design relying on ex post identification of the treatment and comparison group is selection bias. There may be underlying reasons why some villages were targeted by AgResults, and those reasons might explain observed differences between the treatment and comparison group even in the absence of AgResults.

There are several reasons that the comparison remains valid. First, our evaluation took place 2 years before the end of the Nigeria AgResults project and the project expanded widely in the fourth and fifth years to include many more villages, including some from the 2017 comparison group, showing that villages in the comparison group can be favorable places for competitors to collaborate with smallholders to grow AT maize. Second, all of the comparison villages were either: (a) identified ex ante as villages in the AgResults competitors planned to work; or (b) specified as a non-AgResults area prior to 2017 but integrated into the AgResults area after the survey took place.⁶ Third, we use analysis weights for the comparison farmers so that they are better aligned to treatment farmers with respect to demographic characteristics, relatively time-invariant wealth and farm characteristics, and geographic/environmental characteristics.

Data collection

For both the treatment and comparison groups, we recruited farmers to respond to questions in a detailed agricultural survey. To select respondents, we used two-stage sampling, first by village and then smallholder. All farmers who gave consent to be interviewed, and understood that participation was completely voluntary, responded to questions about household demographics, household wealth, agricultural income, and detailed plot-level information about maize cultivation, harvest, and sales. In all, 1823 farmers (944 treatment, 879 comparison) participated in the survey effort, representing 219 villages (112 treatment, 107 comparison) in the northern Nigerian states of Kano, Kaduna, and Katsina.⁷

Within treatment and comparison villages, the data collection effort recruited farmers to participate in a long, intensive interview. Farmers were randomly recruited if they met at least one of these criteria: (a) they were identified ex ante by competitors as farmers the competitor would like to recruit to grow AT maize;² or (b) they were part of a farmer group and cultivated at least one hectare but less than 10 hectares for maize.

Complementing the impact evaluation, the evaluation also obtained qualitative data from competitor interviews and a small number of farmer interviews.

Balance between treatment and comparison villages

As the data are clustered in villages, it difficult to achieve a perfect balance of farmer characteristics that may explain why their villages were chosen as AgResults villages or not and are not simultaneously influenced by AgResults. We found that propensity-score based weighting achieved balance on most characteristics.⁸ Out of 20 hypothesis tests for non-equivalence, only three involve treatment-comparison differences in background characteristics with effect sizes greater than 0.15. Comparison groups farmers are more likely to be Islamic, are exposed to slightly less temperature variability on average, and are less likely to have known an AgResults competitor as long as three years (additional AgResults competitors joined the program in later years).⁹ Of these three characteristics, only the latter is statistically significant (p < 0.01 level). Table 2 in the Appendix 1 displays the results of these equivalence tests. We include all 20 characteristics as covariates in all impact regressions.

Impact estimation

We use a linear regression model to estimate the average impact of AgResults on farmers’ outcomes (see Eq. 1). The model compares average outcomes of farmers in treatment villages to farmers in villages where AgResults did not take place. In Eq. 1, y_ij is the outcome for respondent i in village j; D_j, the treatment indicator, is equal to one if the individual’s village j is a treated village, and 0 if it is a comparison village. The average differences between farmers in treatment villages and farmers in comparison villages is equal to δ, which we interpret as the average treatment effect:

$${y}_{ij}=\alpha + {D}_{j}\delta +{X}_{i}\beta + {Z}_{j}\gamma +{\varepsilon }_{ij}.$$

(1)

To control for household characteristics and village characters that do not vary over time, we include variables X_i (household-level covariates for household i) and Z_j (village-level covariates for village j). We selected covariates that control for factors that could influence the primary outcomes of interest—maize yield and returns. First, we include household characteristics (age and education of the household head, religion, household size, whether the main economic activity is maize farming, type of house, fuel, toilet and water access), and then farm characteristics (size of land owned, average maize harvest in a typical year, ownership of farm equipment and animals).¹⁰ Third, we control for environmental and contextual factors such as distance to an urban center (population > 100 K), soil carbon content, temperature variability, and rainfall for which we had baseline values from secondary data sources.

Household survey responses are clustered in villages, and we do not view these responses as independent across farmers because AgResults was implemented at the village level instead of the household level. Thus, the primary sampling unit is a village and statistical precision is more a function of the number of villages in the treatment and comparison groups than of the number of households in the treatment and comparison groups. This design precludes selection of comparison group farmers who live in the same village as one or more treatment group farmers—a circumstance that might result in agricultural practices induced by the AgResults intervention spilling over into the comparison group.

Results

The evaluation assessed the effectiveness of results-based prizes to increase smallholder awareness and uptake of Aflasafe, and its impact on smallholder income and consumption. These are discussed below.

Impact on uptake and awareness

The project incentivized competitors to increase smallholder uptake of Aflasafe. The project also expected that the competitors would convey how to use Aflasafe to increase the chances of their maize passing verification. Therefore, the evaluation assessed if the project increased farmers’ uptake of Aflasafe and their application practice. In addition, the evaluation also assessed the extent to which the smallholders adopted Aflasafe by evaluating the proportion of their maize cultivated area that the farmers treated with Aflasafe. Further, there was an expectation that the competitors, motivated by the incentive to procure maize from smallholders, may have focused less on generating awareness about the health benefits of Aflasafe and more on the price premiums that smallholders may receive. Therefore, the evaluation assessed smallholders’ levels of awareness of aflatoxins, and their awareness about the adverse health impacts of aflatoxins and the benefits of using Aflasafe. Table 1 presents all results cited in this section’s text.

Table 1 AgResults Nigeria’s impact on smallholder outcomes

Uptake

The evaluation found that the project did have an impact on uptake of Aflasafe: 57% of smallholders in the villages targeted by AgResults applied Aflasafe on at least one plot, compared to only 1% in the comparison group, implying an increase in uptake by 56% points as a result of the competitors’ efforts to encourage farmers to apply Aflasafe.

However, smallholders did not apply Aflasafe to all their maize plots; smallholders in the treatment group applied Aflasafe to only 44% of their maize area. Still, this is an increase of 43% points when compared to 1% of the maize area treated in the comparison group. Interestingly, we find that the farmers either did not learn how to apply Aflasafe or did not follow the directions: in the treatment group farmers applied Aflasafe correctly on only 7% of the maize area (a difference of 6% points compared to the comparison group).¹¹ Smallholders in the treatment group had a lower application rate than that prescribed, even though the project’s verification data indicate that 93% of the maize did pass the verification test.¹² Table 1 displays these impact estimates and regression-adjusted means.

Awareness

The project’s theory of change did not have an explicit expectation that competitors would raise awareness about the health benefits of Aflasafe, nor did it state increasing smallholder awareness as an objective. An implicit assumption was that the smallholders would apply Aflasafe to all their maize plots and that they will consume AT/AC maize with consequent health benefits. However, this evaluation found that smallholders did not apply AT/AC maize to all their maize plots, implying that the smallholders would have two types of maize and will be faced with a decision about which one to sell and which one to consume. We argue that in an environment where AT/AC maize sells for a premium, without awareness about the health benefits of AT/AC maize the farmer would choose to sell this maize. This would undermine the expected human health impacts of the project especially if the final buyer is the poultry market (implying that humans do not consume it). This emphasizes the importance of generating awareness about the health risks of aflatoxins.

We found that the project had an impact on knowledge about Aflasafe (67% point increase in knowledge about Aflasafe among smallholders), but a much smaller impact on the knowledge of aflatoxins and their health risks (22% point increase) and smallholders’ knowledge about how it works (22% point increase). Those who cooked meals, typically the women, learnt even less about aflatoxins and their health risks (6% point increase) or how Alfasafe works (9% point increase). While this shows a large impact, it reflects that nearly two-thirds of the smallholders who used Aflasafe did not know the health benefits it was providing.

For the household to realize the health benefits of aflatoxin-free maize especially when all the maize they grow is not Aflasafe treated, it is important that knowledge about aflatoxins is shared within the household with the women and those that cook for the household, so they use aflatoxin-free maize to cook their meal. The evaluation found that the project did have an impact on cooks’ knowledge about Aflasafe and aflatoxins, but the magnitude of impact was small. Only 29% of cooks in the treatment group had heard of Aflasafe compared to less than 1% of cooks in the comparison group. Only 10% of cooks in the treatment group knew how Aflasafe works, compared to 0.3% of cooks in the comparison group. Far fewer cooks in the treatment group (8%) knew what aflatoxins are, and only 6% understood the health risks of aflatoxins, while no cooks in the comparison group knew about the health risks of aflatoxins. Given that farmers themselves tended to have relatively low awareness of aflatoxins and their health implications, as well as the specific role that Aflasafe plays in reducing aflatoxins, it is unsurprising that the intra-household knowledge transfer to cooks of the household was low.

Our qualitative research indicated that the competitors’ focus and indeed the farmers’ main reasons for engagement were focused more on economic gains from the relationship by way of access to improved agricultural practices and inputs, promise of better yields, and access to guaranteed or premium markets for maize. This suggests that the information flow from the competitor to the smallholder was not perfect regarding the health benefits of the technology or how the technology works, and how it should be applied.

Impact on smallholder returns

We assessed the project’s impact on net maize income, which was calculated as the total value of maize produced in one year, i.e., total sales revenue plus the value of harvested maize not sold, and deducting the cost incurred for maize production.¹³ The evaluation found that the project increased smallholders’ net revenue by US$318 or by 16%. Treatment farmers earned, on average, US$2305 per farmer, while comparison farmers earned an average of US$1987 per crop.

To understand these effects on smallholder income, we explored the pathways through which the project expected to affect smallholder incomes from maize. The project’s theory of change expected to affect smallholder incomes from maize in three ways. First, competitors were expected to provide information about inputs and agricultural practices to farmers that would help farmers realize an increase in maize yields, with yields ultimately leading to higher sales revenue. Second, the project hypothesized that competitors may pass on price premiums for AT maize to the farmer, or even higher premiums if the AgResults competitor discovered access to premium markets for AT maize. Third, the project hypothesized that competitors would encourage smallholders’ uptake of Aflasafe in maize production by offering interest-free credit on inputs, free inputs, or output buy-back guarantee.

Yield

Our results indicate that average maize yield was 2.8 MT/ha for smallholders in the treatment group and 2.7 MT/ha for smallholders in the comparison group, reflecting a 4% increase, but this difference was not statistically significant.¹⁴ This increase translates to a 0.2 MT/ha increase for the group that adopted Aflasafe (an impact of 0.1 divided by the percent who adopted Aflasafe). The fact that yield in the comparison group was higher compared to average maize yield in Nigeria (typically 1.6 MT/ha [11]) could reflect competitors’ efforts to either identify areas where maize yields were higher or to engage with smallholders who were serious about maize farming and who thus obtained higher than average yield with or without the competitors’ influence. Most of the comparison group farmers were those that competitors hoped to engage with, like treatment group farmers, and were a part of a farmer’s cooperative. In addition, qualitative interviews indicated that competitors were seeking the “more serious maize farmers”. It is also important to note that the smallholders’ average maize yield was greater than the 2 MT, which was identified ex ante by the project planners as the breakeven point that the project’s business plan indicated was necessary for Aflasafe adoption to be economical [7].

Maize price premiums

We found that as a result of the project, smallholders received a higher price on their maize sales. Smallholders in the treatment group sold maize at an average price of US$428/MT, while smallholders in the comparison group sold maize at an average price of US$407, implying that farmers in the treatment group received a 5% price premium. This effect was statistically significant.

Other outcomes

In addition to price, AgResults competitors may have offered other incentives to smallholders, such as output buy-back guarantees or improved access to credit and inputs. Our evaluation found that the project increased farmers’ annual maize sales revenue by 31% (see Table 1).

While this increase reflects the higher prices the treatment group received, it also reflects the larger amount of maize they transacted through purchase guarantees (this amount also includes maize given as payment for past credit taken from the buyer).¹⁵ Farmers in the treatment group transacted 3.7 metric tons of maize, while the comparison group transacted only 2.7 MT, a difference of roughly 1 MT. This implies that farmers in the treatment group were able to complete more sales transactions. It also implies that smallholders in the treatment group kept less of the maize harvested aside for other uses. Farmers in the treatment group set aside 3.5 MT of maize for other uses in contrast to 3.7 MT of maize set aside by farmers in the comparison group, which amounted to a 6% decrease.

Other incentives that the competitors reported offering to farmers were better access to inputs and access to improved inputs. The total costs of inputs could have decreased if competitors gave some inputs for free or at subsidized costs. However, these input costs could have also increased if the competitors provided better inputs that were more costly, or if they encouraged farmers to apply additional productivity-enhancing inputs. We found no statistically significant difference in total input costs, though the negative point estimate suggests that the comparison group did incur slightly higher input costs than the treatment group.

Further, it is also possible that the project’s focus on influencing returns to maize production could have resulted in smallholders increasing the area under maize cultivation. In our exploratory assessment we found that the average maize area under cultivation was higher in the treatment group than in the comparison group, but the difference was not statistically significant. Note that dedicating more land for maize could reduce income from other crops if the farmers did not increase the overall cultivated area. The evaluation did not assess net household agricultural revenue, thus the impact on net maize revenue from maize is an upper bound estimate of the project’s impact on total agricultural income.

Discussion and conclusion

The AgResults Nigeria project aimed to use results-based prizes for the private sector to create a smallholder-inclusive market for AT maize wherein smallholders both produce and consume AT maize. The project provided temporary incentives to maize aggregators—the competitors—that mimicked the premium that some end user markets provide for AT maize. This approach engaged multiple private sector actors in aggregating and supplying AT maize, all of whom invested to organize smallholders, supply Aflasafe, procure AT maize and sell it to premium markets.

To some degree, the project’s theory of change was realized—private sector actors incentivized technology adoption driven by their private motive. Smallholders targeted by the private sector actors experienced an increase in annual maize income of $318, or 16%, through access to premium markets and being able to sell more maize. Smallholders did learn about the health impact of aflatoxins, or Aflasafe as a solution, but there were significant gaps in their understanding. This limited the project’s impact on human health because smallholders did not apply Aflasafe to all their maize. The results-based approach possibly had a drawback in that the competitors, motivated by the incentives, focused more on the messages about the profitability of adopting Aflasafe and did not focus as much on sharing information about the health benefits of Aflasafe. Overall, this finding highlights the potential for private sector led approach, but it also equally highlights the gaps that such an approach may have, arguing for the need for having complimentary efforts to address those gaps. This is particularly the case when the technology’s benefits are not perceived by the consumer (aflatoxins are not visible to the eye, and the health benefits are not immediate), or when the technology results in positive externalities (final consumers of Aflasafe-treated maize also benefit).

The results-based approach was successful in creating a niche smallholder-inclusive market for AT maize, which is a key feature of this approach: attracting multiple private actors to engage in the market stimulates the pre-conditions for a sustainable market to function, particularly compared to other more traditional approaches that engage with one entity. However, by the end of the project there were barriers to the market becoming mainstream. Broader consumer awareness needs to be raised to promote continued development of the market for AT maize, aflatoxin standards need to be enforced, and aflatoxin testing needs to be more easily available. In the fall of 2016, Johnson and colleagues found that a majority of farmers in Kaduna had heard of alfatoxins and that a sizeable proportion used Alfasafe (exact amount unclear) [16]. Ayedun and colleagues [2] confirm that awareness increases willingness to pay for Alfasafe.

As we conclude, it is important to list the study’s limitations. First, the quantitative evaluation is quasi-experimental rather than experimental, so the impact estimates cannot be interpreted as causal. Specifically, there may be underlying, unobservable differences between villages and smallholders selected for AgResults participation and those that were not. This is a standard limitation in any quasi-experimental evaluation. Second, the impact evaluation is based on follow-up data only, with baseline values on observables based on recall data. The initial evaluation design was an RCT, which failed because the competitors shifted their treatment strategy such that they had a low treatment rate in the villages assigned to the treatment group, and a similar treatment rate in the villages assigned to the comparison group. This implied that we needed to add treatment households to the sample and collect baseline information for them. We mitigated the concerns with recall data by focusing on data that do not change (e.g., gender, education of household head), that can be calculated (e.g., age), or are sticky so have good recall (e.g., type of wall, number of toilets) and by gathering baseline values using secondary data on rainfall, temperature, and distance to market (see Table 2 in Appendix 1). Finally, the impact evaluation focused on the zone of influence of six competitors (out of a total of 24 competitors) in Northern Nigeria that joined the program early because of the project’s intended focus on the North. Insofar as the six competitors of the program engaged 75% of the smallholders in the program until project year 3, this is not a significant constraint.

Appendix 1

See Table 2.

Table 2 Baseline equivalence on key outcomes