The opportunity to carry out a meta-evaluation of USAID’s Grand Challenges (GCs) for Development was one not to be missed for Triple Line’s evaluation and fund management teams. We had recently completed an evaluation of Sida’s Global Challenge Funds, and had analysed similar open innovation approaches in our evaluations of the Global Innovation Fund and the Amplify Fund. Additionally, over many years of fund management practice, evaluation, and research, we have established ourselves as thought leaders in the evaluation of challenge funds. Added to this was the chance to work with a new client, USAID, a pioneer in the use of grand challenges for development cooperation.

Since 2011, USAID’s Grand Challenges for Development programme has mobilised governments, companies and foundations around pressing international development problems to source new solutions, test new ideas and scale what works. All GCs offer challenge grants, but many use additional tools depending on the problem they intend to solve. These can include prizes, hackathons, and capacity building services. The model also leverages donors’ ability to de-risk investments and crowd in follow-on funding from both the public and private sectors once a development innovation has been proven to be effective and commercially viable. The portfolio of nine USAID GCs studied for the meta-evaluation represented a total investment of US$463.4 million from USAID and donor partners.  767 awards had been made to innovators working in 83 countries in every region of the globe.

After more than eight years of implementation, USAID wanted to take a systematic look at the achievements and challenges of the GCs and their open innovation approaches in the areas of health, education, agriculture, energy, food security and conflict and violence. The evaluation generated an actionable evidence base to inform recommendations at both strategic and operational levels that USAID and its partners could apply to guide investment decisions and advance the design, management, and measurement of GCs. Our team of innovation, challenge fund, cost effectiveness and multi-donor evaluation experts brought an established track record in complex evaluation. We drew on deep experience of the design, management and evaluation of challenge and grant fund schemes and combined participatory, utilisation-focused approaches to make effective use of co-creation processes for successful take-up of recommendations.

Our generic intervention model provided a framework against which each GC can be overlaid, enabling similarities and differences to be analysed comparatively and against OECD DAC criteria.

Between March and September 2020, the evaluation team worked on the design and planning phase of the evaluation. Following preliminary data collection and consultation activities, we drew on our deep knowledge of challenge fund approaches, theories, key dilemmas, and implementation constraints to develop a generic Grand Challenge intervention model. The model set out how each challenge is intended to go from initial design through delivery to the achievement of objectives. The model was developed iteratively together with USAID and GC managers and provides a framework against which each GC can be overlaid, enabling similarities and differences to be analysed comparatively and against OECD DAC criteria.
We built in broad flexibility to allow for different online platforms, timing and preparation, connectivity, and bandwidth issues, as well as well-developed protocols for facilitation of online consultations.

During inception, our team had to adapt to the restrictions posed by the onset of Covid-19. We restructured the evaluation methodology to accommodate the need to work remotely. We facilitated a web-based interactive prioritisation workshop with GC managers in the USA and Canada, and USAID personnel in Washington DC to review our evaluability assessment and agree priority questions for the evaluation. The restructuring involved replacing planned country visits for case studies with remote consultation. We also built in broad flexibility to allow for stakeholder preferences regarding online platforms (for conferencing, interactive presentation, visual collaborative working – with backup options in place), timing and preparation (including prior testing of technologies), and issues with connectivity or bandwidth. Additionally, we established robust protocols for the facilitation of online interviews and workshops that we developed and refined over the course of the pandemic. Drawing on our track record of flexible, adaptive evaluation approaches, we were able to design, implement, appraise and redesign tools in response to changing circumstances in an iterative fashion to meet present needs and priorities.

Flexibility from all partners, together with the rapid adoption of digital technologies, enabled us to design a robust, participatory evaluation that focused clearly on the priority information needs and concerns of USAID and its partners.  The evaluation provided well evidenced findings of GC achievements (results, scaling, acceleration, ecosystem investment and catalytic effects), GC implementation (procurement and reducing barriers, and engagement with USAID Missions), and GC management (results measurement, cost-effectiveness analysis and governance and management) as well as a suite of practical recommendations for future programming.  The evaluation also included a comparator analysis of four other funds, a learning memo about the integration of gender equality and social inclusion across GCs, case studies of successful examples of scaling from GCs and a framework for cost-effectiveness analysis for USAID to apply going forward. To ensure the dissemination of USAID's evaluation findings and recommendations, we ran two active learning workshops with policymakers and GC managers, as well as a publishing a set of knowledge products that covered issues such as scaling and monitoring, evaluation, and learning.

Please see here for a link to the report. 
 

Credit: title photos by The Futurepump and Morgana Wingard for USAID

The Futurepump: low-cost solar irrigation pumps for the world's one-acre farmers. Supported by USAID – Powering Agriculture: An Energy Grand Challenge for Development. www.futurepump.com

Fighting Ebola: using design thinking to help fight Ebola. A worldwide open innovation challenge to generate new solutions for the healthcare workers, care-givers and communities on the frontlines of the Ebola crisis.