Fostering Collaboration through Evaluation: The Stevens Initiative, an International Public-Private Partnership to Support Virtual Exchange
This post is part of the NORRAG Debates stream on Philanthropy in Education, and follows on from a presentation by the authors at the Inaugural Symposium held in Geneva, 22-24 November 2017, titled “An international philanthropic coalition to support international exchange and online learning”. It discusses a specific case study of evaluating philanthropic involvement in education and the importance of using an evolving evaluation process for organizational learning.
The Stevens Initiative was established as a partnership between the U.S. Department of State, the Bezos Family Foundation, and the family of Ambassador Chris Stevens, to honor his legacy by expanding the field of online exchange. The Initiative is housed at the Aspen Institute, and has benefited from the support of the governments of the United Arab Emirates (UAE) and Morocco, and from in-kind donations by technology partners. RTI International is a nonprofit research organization which is collaborating with the Initiative to evaluate its impact and effectiveness.
Jennifer Laird is a Program Director at RTI International’s Center for Evaluation and Study of Educational Equity. Maysam Ali is Assistant Director of the Stevens Initiative.
Evaluating an initiative requires agreement about what success looks like and how to measure it. A good evaluator will push an initiative’s designers and other stakeholders to clearly articulate their definition of success, and then propose assessment methods and metrics. If an initiative is new and likely to evolve, the evaluator can support continuous improvement by sharing emerging findings and help stakeholders respond and adjust. The evaluation collaboration between the Stevens Initiative and RTI, described below, demonstrates one such evaluation approach to foster collaboration and learning among the diverse stakeholders – developmental evaluation. This partnership is an attempt to measure impact for a complex program that includes multiple governments, foundations, and education organizations in more than 17 countries for feedback and evolution in real time.
The Stevens Initiative
The Stevens Initiative is an international public-private partnership that gives young people the knowledge, skills and experiences they need to prosper in an interconnected world. The mission of the Stevens Initiative is to build global competence for young people in the United States and the Middle East and North Africa (MENA) by growing and enhancing the field of virtual exchange: online, international and collaborative learning. The Initiative aims to improve young people’s global competence skills though virtual exchange, including collaboration, communication, problem-solving, awareness of diverse perspectives and civic and global engagement.
The Initiative accomplishes its mission by providing financial support to virtual exchange programs both in the US and the MENA region. The Initiative has so far provided two rounds of grants to organizations in both regions. The first cohort consisted of 10 US-based organizations, including universities, and educational exchange organizations. The second cohort, announced in November 2017, included 12 grants in the US and MENA. The new programs will expand the Initiative’s reach to approximately 30,000 students in 18 countries in the MENA region and more than 31 US states.
The Initiative is supported by a diverse group of stakeholders, including public and private donors, governments, educators, civil society organizations, technology corporations, and virtual exchange practitioners, among others. Public and private partners, including governments, private foundations, and tech corporations, offer financial and in-kind contributions as well as advice and connections to expand the work of the Initiative to new communities. The Initiative also engages with education leaders in the US and the MENA region, seeking input from state and local officials, including on local priorities for education and workforce development. Regular meetings with policymakers ensure that the program aligns with the needs of local communities. In addition to governments and education leaders, the Initiative is fostering collaboration among its grantees – mostly educational organizations in the K12, higher education, and non-profit sectors – and evaluation has proven to be an important vehicle driving this collaboration.
How is RTI approaching the evaluation?
RTI is taking a developmental evaluation approach for the Stevens Initiative evaluation. Whereas traditional evaluations often provide summative results to external funders to validate program and practices, developmental evaluation allows for real-time feedback, supporting ongoing adaption and improvement, and allows researchers the flexibility to respond to the real-time needs of program stakeholders. This is critical, as the Initiative has multiple stakeholders representing various sectors and interests.
Prior to the existence of the Stevens Initiative, these organizations had different approaches, participant demographics, regional focuses, and other variations in their evaluation efforts. The Initiative brought these organizations together through the Evaluation Working Group (EWG) as a way of maintaining transparency and ensuring feedback from grantee organizations on the evaluation plan being created by RTI. With the overwhelming amount of information and tools available, convening this group under the guidance of professional researchers was a critical way to organize, codify, and make usable the vast amount of information and tools available for evaluation.
The EWG is comprised of representatives from key stakeholders including the U.S. Department of State’s Education and Cultural Affairs Office of Policy and Evaluation, the Bezos Family Foundation, and evaluation managers from each grantee organizations, in addition to the Stevens Initiative team. The group, which meets quarterly, is not a decision-making body, but has enabled RTI to incorporate diverse perspective and priorities of various stakeholders in the following areas:
- Refining the monitoring and evaluation plan
- Identifying the key global competencies that are of interest to the Stevens Initiative
- Reviewing proposed survey items for those global competencies
- Agreeing to a set of common survey items to be administered across all implementing organizations
- Sharing strategies for achieving high survey responses rates
The EWG has also been a venue for RTI to share evaluation results through reflection sessions. Grantees often share successful tools, practices or program features, challenges, and risks. Discussion items included questions or issues related to program practice that could be addressed by research, external sources of information that have informed program design and implementation, and potential directions for the Stevens Initiative research agenda.
The Group also offers a forum for thinking about measurement tools and a research agenda for the future. It has made great progress in the first two years in defining common terms, and agreeing on common measures and methods, which can prove challenging in a cross-cultural context. Working together, the Initiative’s grantees have made strides towards refining broad common constructs; building a community of practice; increasing organizational capacity; and strengthening connection between research and practice. Perhaps most of all, the evaluation process has led to an improved understanding of virtual exchange practices, strengths, and challenges among a diverse set of organizations.
Fostering Collaboration through Evaluation
Evaluation activities at the Initiative have been increasing over the past few months. These activities include hosting meetings for evaluators working in the field of virtual exchange or international exchange and/or experts in the measurement of global competencies to discuss promising measurements and evaluation design; conducting webinars for practitioners; presenting at conferences; and producing a public report including the findings.
Going forward, RTI will improve the rigor, consistency, and quality of the next phase of evaluation through (1) more targeted survey measures that are standardized across programs; (2) technical assistance for implementing organizations’ quantitative data collection, in order to improve data quality and compatibility; (3) technical assistance for implementing organizations’ qualitative data collection methods; and (4) support for more rigorous evaluation designs to yield stronger levels of evidence. This rigorous evaluation and feedback exchange will be vital to ensuring that the Stevens Initiative achieves its goals as a hub of knowledge transfer in the field of virtual exchange and continues to serve as a platform for collaboration and learning in the field.
Citations:
Bureau of Labor Statistics, U.S. Department of Labor, The Economics Daily, Youth unemployment rate, 11.5 percent; employment–population ratio, 53.2 percent in July 2016 on the Internet at https://www.bls.gov/opub/ted/2016/youth-unemployment-rate-11-point-5-percent-employment-population-ratio-53-point-2-percent-in-july-2016.htm (visited September 08, 2017).
International Labor Organization, World Employment and Social Outlook 2016: Trends for Youth, August 24, 2016, on the Internet at http://www.ilo.org/global/research/global-reports/weso/2016/WCMS_513739/lang–en/index.htm (visited September 8, 2017)
Disclaimer: NORRAG’s blog offers a space for dialogue about issues, research and opinion on education and development. The views and factual claims made in NORRAG posts are the responsibility of their authors and are not necessarily representative of NORRAG’s opinion, policy or activities.
Contribute: The NORRAG Blog provides a platform for debate and ideas exchange for education stakeholders. Therefore if you would like to contribute to the discussion by writing your own blog post please visit our dedicated contribute page for detailed instructions on how to submit.