SEEP Conference participants discuss ways to improve data and use of evidence in organizations during the session, Using Data and Evidence to Drive Policy and Program Improvement. Photo: © mari matsuri

For those of us working in development, the need for data and evidence-informed decision-making is no longer controversial. Of course, there are ongoing related debates, such as whether randomized trials are leading development research in the right direction, and what types of data to use when such methods are not feasible. But by now, most groups have internalized the value of evidence, and a wide array of data collection methods are available.

So increasingly, we have data, but how are we using them to learn? And perhaps more crucially, how can we leverage the important multiplier effect our organizations have (as donors, researchers, and catalysts) to make the best use of the growing evidence base? At Evidence for Policy Design (EPoD) at Harvard Kennedy School, this question is central to our mission of uniting research and practice for smart policy. During our lunch dialogue at the 2018 SEEP Annual Conference, we explored these questions with participants in an interactive session focused on using data to inform decisions. The surrounding discussion revealed ongoing challenges and frustrations – but also concrete ideas for improvement.

Why is using data so challenging?

When we polled SEEP conference-goers about the barriers to data and use of evidence in their organizations, individual capacity gaps (lack of capacity for data analysis) topped the list, but organizational barriers were also important (lack of organizational resources/incentives and problems with the format of existing datasets). These results resonate for us based on our surveys of – and experience working with – civil servants in South Asia, who cite similar barriers to evidence use in their organizations.

Responses from SEEP conference-goers on top barriers to use of evidence in their organizations.

Data quality also remains a major challenge. Though there are a host of technological solutions that can help maintain high standards, such as digital data collection with built-in audit functionality, they require organizational investments. And even what seem like relatively simple fixes – like using common unique identifiers to facilitate linking across multiple datasets – are often not adopted due to lack of coordination across departments or agencies.

Addressing these barriers can seem daunting, especially when most of our organizations work on complex problems where achieving precise measurement and setting metrics for success is difficult. For example, in our own work building policy organizations’ capacity to use data and evidence, we face challenges both in defining objectives (what does ‘capacity to use evidence’ look like?) and tracking participants to understand how they integrate new knowledge into practice.

What can we do to make better use of all that data?

SEEP conference-goers had no shortage of good ideas when it came to small steps that they or their organizations could take to improve data use. Some of these ideas tied back to aligning organizational incentives – such as setting explicit targets and incentives for staff related to data collection and use, better communicating about available data and evidence, and building in more collaboration at the outset to define the questions different stakeholders within the organization need data to answer.

Participants noted that long reports often go unread, but short summaries, dashboards, and central repositories combined with visualization tools allow users at all levels to extract relevant information. We know that human beings are visual learners, and in our work we’ve seen that converting data into simple visualizations has great potential for expanding data use.

Through our work with government agencies, we’ve seen examples of how small changes in organizational systems can lead to greater data use, with significant impact on outcomes. Our pilot projects, where we collaborate directly with policymakers to give them hands-on experience with data and evidence, provide several examples. Through this work, we’ve recognized the value of integrating systems for data use (not just data collection and reporting) into the organizational processes of government ministries and development groups alike. This might mean setting up standards for data collection in order to ensure data systems and datasets are synchronized, building data use into our standard practices (like using a data dashboard as part of a standing agenda for a regular staff meeting), and making investments in good knowledge management (perhaps hiring someone who can serve as a ‘data champion’ to support data use across the organization). To get the most out of the data we collect, aligning incentives across the organization is key.

Some participants at #SEEP2018 noted that their organizations would benefit if the staff as a whole – and not just a few key analysts – had a better understanding of how to use data. Here, we can offer some concrete guidance: EPoD has recently made two online modules on using Descriptive Evidence and Impact Evaluation available for open access use, and we invite readers to enroll and share with colleagues.

Charity Troyer Moore is India Research Director for Evidence for Policy Design at the Harvard Kennedy School. She leads research-policy engagements with a variety of entities in India to ensure that research is attuned to the problems facing policymakers and integrated into policy design and program implementation. Charity's research examines how to use technology to improve public service delivery and governance; the drivers and potential solutions to India’s low female labor force participation; land rights; and social protection programs. She holds an M.A. in Economics and Ph.D. in Agricultural, Environmental, and Development Economics from The Ohio State University.

Emily Myers is the Associate Director of Training and Policy at EPoD, where she leads EPoD’s training portfolio and develops new policy partnerships. Her work focuses on strategies for building government capacity to use evidence to strengthen policy design and service delivery. In 2013, she helped to design and launch the Harvard Building Capacity to Use Research Evidence (BCURE) Program. She has a background in international development and health, and has previously worked on health systems reform projects in Malaysia and Mexico. She holds a Master’s in Public Administration in International Development (MPA/ID) from the Harvard Kennedy School.

Charlotte Tuminelli is the Senior Training Manager at Evidence for Policy Design (EPoD). In her role, Charlotte plans and delivers EPoD’s capacity building activities, which aim to build trainees’ capacity to use evidence in decision making. Her work focuses on ensuring that EPoD’s trainings are pedagogically sound, practically relevant, and smoothly delivered. Prior to joining EPoD, she supported course delivery and research projects for faculty at Harvard Business School. She has also served as a teacher in rural India, worked for a number of Boston-area nonprofits, and consulted on local and international education projects. Charlotte holds a Master’s in Education (Ed.M.) in International Education Policy from the Harvard Graduate School of Education and a B.A. in International Studies and Foreign Languages from Stonehill College.