Young cocoa farmers in Indonesia. Photo credit: Mondelez International

Mondelēz International’s Cocoa Life is an outcome-driven sustainability program for cocoa-producing communities, for which Ipsos has been conducting impact studies since 2015. In this article, Meghann Jones, Senior Vice President at Ipsos, describes some lessons learned from impact measurement in this challenging environment.

Mondelēz International’s Cocoa Life program is a major platform for cocoa sustainability activities in six cocoa origins: Cote d’Ivoire, Ghana, Indonesia, Dominican Republic, Brazil, and India. The program works with thousands of smallholder farmers, their communities, and their families, by providing access to training for on-farm and off-farm skills, building platforms within communities to steer their own development, assisting with funding for community projects, and partnering to protect natural resources.

Ipsos has been tracking the impact of Cocoa Life in yearly longitudinal panel studies since 2015. The studies comprise three main elements: household surveys with cocoa farmers and their spouse to understand the situation of their household; cocoa farm visits together to observe the state of the cocoa farm; and studies with the village and cooperative leaders, and special interest groups such as young people and women.

As Cocoa Life enters a new decade, with the target of meeting 100% of Mondelēz International’s cocoa requirement by 2025, and resulting new measurement challenges associated with program scale, here we look back on some of the key lessons we have learned:

1. KPIs are useful for business leaders but not for learning: make a robust theory of change the #1 priority

The need to report concisely to a broad audience such as corporate leaders and investors can lead to focusing resources on high-level indicators of success. High-level indicators, however, are not especially helpful for driving learning and enabling the delivery of impact. At the outset of the impact studies, Ipsos worked with the Cocoa Life team to develop a comprehensive theory of change (TOC) as a robust hypothesis for program impact, rallying the program and evaluation team jointly around the program hypothesis, and helping the team to understand how to use different types of data as evidence that outcomes did or did not occur. The program TOC, which “rolled-up” to the original 10 program KPIs, offered a basis for a much deeper understanding of both program implementation and the potential for impact, and has become a fundamental of program iteration.

2. Be realistic about challenges and constraints; don’t let perfect get in the way of good

The problem of attribution – whether an observed outcome is the result of the program - is an enduring challenge in impact evaluation. The way attribution is understood in the “hard sciences” is with randomized control trials (RCTs). Early in the Cocoa Life impact studies, Ipsos used RCTs and quasi-experimental designs to understand program effects in “treatment” communities. As we described in a paper published in early 2019, while these studies yielded some useful data points, they were challenging to implement at ground-level, which impacted the reliability of the data, and were prohibitively expensive. With a rigorous, theory-based approach to non-experimental impact measurement for subsequent studies, the Cocoa Life impact studies can deliver findings with high levels of confidence within reasonable timeframes and budgets.

Cocoa harvest in Indonesia. Photo credit: Mondelez International

3. Don’t reinvent the wheel: align and collaborate

If surveys and data management are aligned to open standards that are recognized by other actors, this can reduce the burden of planning and documenting while enhancing the acceptance and usability of the information. There are many types of standards, but they are not usually built to enable the tracking of individual objectives of a program. For Cocoa Life, which is outcome-based, it was necessary to design new impact measurement tools and survey instruments to capture data optimally for tracking specific outcomes over time, while factoring in an appropriate measurement standards and best practices, such as CocoaAction protocol, approaches to assessing living income and, and Ipsos’ framework for measuring impact in women’s economic empowerment.

4. Be patient

When significant resources are being invested in programs such as Cocoa Life, investors want to know what the return on investment (ROI) is. However, impacts are often achieved over a long timeframe and are diffuse. For example, one year after an intervention takes place, it may be possible to see an immediate impact in farm management and cocoa productivity, due to implementing good farming practices. However, for other interventions, such as community development and environmental initiatives, the impact is only revealed over a greater length of time. This tension between the desire to know what’s working and the evaluator’s ability to deliver results in the short to medium term forces us to look for interim outcomes, proxies for long term outcomes, and to use short term findings as a basis for targeting the intervention towards the areas that need most improvement in an ongoing way.

Women at a community water point in Cote d’Ivoire. Photo credit: Mondelez International

5. Talk about it

Learning requires people to reflect and act upon data. For Cocoa Life, that means continuously re-evaluating and adapting how reports are created for internal teams, partners, and external stakeholders. Often this involves an element of innovation, trial, and taking time to talk with recipients about the best ways for them to experience data. The Cocoa Life team – with Ipsos as its partner – has committed to sharing impact results and discussing the challenges inherent in delivering and measuring programs in this complex space, including with the SEEP network!

is Senior Vice President at Ipsos. She brings expertise from across Ipsos services to measure the impact of corporate sustainability investments. She is passionate about promoting equality and social progress through Ipsos' work, and helping businesses to understand their role in society, how to address their sustainability challenges, and measure their impact. Meghann has been the lead evaluator for the Mondelēz International Cocoa Life program since 2016.