Our evaluation of the FCDO-funded Strategic Partnerships for Higher Education Reform (SPHEIR) programme is one of our longest-running evaluation assignments, being implemented over six years from 2016-22. This provides opportunities not often encountered in evaluation – to build strong and constructive relationships with implementers, to gather a large volume of data for greater quality and robustness, and to find evidence of wider, systemic change that is difficult to discern over more usual project durations of up to three years.

Run by the British Council, the £45 million SPHEIR programme works by catalysing and funding collaborative partnerships to transform the quality, relevance, scale, accessibility and affordability of higher education – helping to address challenges and generate the job-ready, skilled graduates that business and societies need. Each of the nine partnerships seeks to improve higher education outcomes in developing countries by facilitating north-south and south-south knowledge sharing between higher education institutions on innovative teaching techniques, curriculum development best practices, and public policy engagement. With improved educational outcomes leading to widespread knock-on effects on individual employability, women’s empowerment and poverty alleviation, the programme contributes to achieving more than half of the 17 Sustainable Development Goals (SDGs).

Triple Line leads a highly-qualified consortium combining innovative, mixed-methods MEL approaches, expertise in institutional capacity building and policy analysis with specialist knowledge in higher education, science, technology and innovation

SPHEIR’s ambition is to transform the higher education systems of the countries it operates in. In order to monitor progress towards that goal and to capture emerging insights, the programme requires a similarly high-level monitoring, evaluation and learning (MEL) partner to conduct robust baseline, mid-term, and endline evaluations over the course of the programme, and surface learning to strengthen programming. Triple Line leads a highly-qualified consortium with Technopolis Group and the University of Bedfordshire, combining our experience of using innovative, mixed-methods MEL approaches and our expertise in fund management and institutional capacity building, with the specialist knowledge of our partners in higher education, science, technology and innovation policy and practice.

Since 2016 the evaluation team has completed the baseline evaluation and a formative process evaluation, as well as undertaking a number of rapid evidence reviews and research projects, such as the design and testing of an innovative tool for measuring 21st century skills and competencies (which has been deployed in the evaluation) and research into higher education practices for 21st century learning. The mid-term evaluation was finished at the end of 2020.

Guided by the SPHEIR theory of change and a set of agreed evaluation questions, the evaluation is designed to measure the impact, outcomes and effectiveness of the programme, and provide evidence of what works and why. It is structured across five levels:   

• Higher education system level. SPHEIR aims to catalyse reform and innovation in entire higher education systems at national level. The final impact goal is that such reforms contribute to economic and social development.
• Sector/employer (world of work) level. A significant number of SPHEIR partnerships aim to contribute to a better match of supply and demand in specific sectors of the economy. The partnerships seek to build a relationship between higher education institutions and employers to acquire the relevant knowledge, skills and attitudes for the world of work.
• Higher education institution (partnership) level. This is the typical entry point of intervention of the partnerships. SPHEIR aims to contribute to institutional capacity building, innovation capabilities and department-level reform. This is also the entry point to evaluate partnership dimensions and process aspects
• Educator level. A large share of SPHEIR projects include a component of developing the capacity of lecturers (or educators), for instance for new curricula elements, or the use of digital tools or new pedagogies.
• Student level. SPHEIR partnerships aim to increase access to higher education and to increase quality and relevance of teaching so that graduates enter the labour market with the skills and competencies they need. The evaluation also looks at student experience of higher education more broadly and is exploring gender equality and social inclusion.

Primary and secondary data, both quantitative and qualitative, has been collected through desk research, interviews, focus groups, institutional assessments and large-scale online surveys. With visits to countries including Somaliland, Myanmar, Kenya and Sierra Leone for the baseline exercise, the evaluation team gathered evidence about the experience of higher education directly from beneficiaries and from those implementing it, including students, lecturers, university leaders and administrators and a wide range of external stakeholders such as employers and higher education regulatory bodies. The evaluation team also heard from partnerships about their plans and aspirations for SPHEIR projects, early progress and challenges.

In order to take advantage of the long duration of the evaluation, our methodology includes a detailed tracer study to follow a sample of graduates after their courses end. The study will focus on measuring the quality of graduate employment on a number of key dimensions (study relevance, career opportunity, salary) for students exposed to SPHEIR and for a control group, to gauge the impact that SPHEIR has had.

We have replaced field visits with a data gathering tool that encourages students and lecturers in SPHEIR institutions to tell rich, qualitative stories about their experiences around key themes

Our initial design drew on remote working approaches to conduct student and lecturer surveys and stakeholder interviews. This approach has been deployed to great effect during the Covid-19 crisis, enabling us to continue to work to a high level with minimal disruption. In the absence of field visits for the mid-term evaluation, all interviews were conducted remotely and we replaced focus group discussions with a data gathering tool that encouraged students and lecturers in SPHEIR institutions to tell rich, qualitative stories about their experiences around key themes. We sought to make these tools accessible and attractive to encourage and incentivise the students and lecturers to respond with stories (particularly in the context of Covid-19) by limiting the number of questions, allowing responses through a range of channels (SurveyGizmo, Facebook, emails, WhatsApp), allowing submissions in the form of documents, video, voice notes, images and directly into SurveyGizmo, and with a prize entry.

SPHEIR is a strong, unfolding example of the potential advantages of taking a long-term approach to programming as well as MEL. Through it we are demonstrating our capacity to monitor, evaluate and learn from complex programmes aimed at systemic change through innovative methodologies and effective multi-stakeholder collaboration.

Other long-running MEL assignments:

• Independent Review Team for DFID Forest Governance, Markets and Climate Programme, 2013-17
• Monitoring and results measurement partner for the Africa Enterprise Challenge Fund, 2008-17
• Monitoring & Evaluation Services to the MasterCard Foundation’s Fund for Rural Prosperity, 2008-21