SOCIAL INNOVATION IN HEALTH

MONITORING & EVALUATION FRAMEWORK

This monitoring and evaluation framework is a new research tool to guide researchers, innovators, community members, decision-makers and other social innovation actors on how to carefully monitor all steps of the social innovation development process and evaluate effectiveness, sustainability and scalability. The guide includes a step-by-step process, a conceptual framework, considerations for assessing hackathons, social innovation contacts, and open access resources. The accompanying social innovation in health research checklist can be found here.

SOCIAL INNOVATION IN HEALTH

MONITORING & EVALUATION FRAMEWORK

The purpose of this interactive guide is to provide a monitoring and evaluation framework for social innovations in health. The guide includes a step-by-step process for monitoring and evaluation, a conceptual framework, considerations for assessing hackathons, social innovation contacts, and open access resources. The accompanying social innovation in health research checklist can be found here.

Key Issues	Specific techniques/research methods	Open access resources
Create marketing/dissemination strategy Document and share the process to create the social innovation in health Document and broadly share the impact of the social innovation in health Develop and strengthen partnerships with supporting market actors who can sustain the model Create systems and/or integrate into existing systems and structures to institutionalize the social innovation Identify unmet needs Adaptation and replication in other settings	Documenting best practices: methods proven reliable to lead desirable results Impact analysis: assess the results of the project Delphi techniques: a structured way to aggregate knowledge from a group of people, often used in health guideline development Mathematical modelling: using mathematical formulations to guide problems	WHO Best Practices Guide: documents criteria and format to analyse best practices Impact Assessment: guide to conduct an impact analysis Implementation Research in Health – A Practical Guide: a guide on implementation research for multiple stakeholders Reporting guidelines for modelling studies: evidence-based tools to guide research reporting Nesta’s Funding Innovation – A Practice Guide: a guide to different financial tools to support innovations Nesta’s tools and insights for governments: a guide to improve governance for innovations Anticipatory regulation: an emerging approach that is proactive, iterative and responds to evolving markets Strengthening evaluation capacity: a guide on strengthening the M&E capacity of individuals, organizations, communities and networks

Key Issues	Specific techniques/research methods	Open access resources
Define longer-term metrics of impact, including fidelity and related implementation metrics Assess potential to scale down to a specific local context or be adapted for another context Assess potential for dissemination of innovation to different contexts and cultures Develop impact and sustainability metrics with stakeholders and end users	Social impact assessment: investigates social effects (that is, non-medical) on end-user population from the innovation guideline development – dissemination of ideas and best practices Implementation research: implementation across subunits/network identi-fication/confirmatory inferences designs/effectiveness-implementation hybrids Qualitative research: ethnographic study, in-depth interviews, other methods Discrete choice experiments: quantitative measure to elicit individual preferences on innovations	Effectiveness-Implementation Hybrid Designs: assess both clinical outcomes and implementation Social Network Analysis: guide to map relationships and networks in communities Adaptive trial designs: guide on designing adaptive clinical trials Discrete Choice Experiment: WHO guide on conducting discrete choice experiments Scaling-up guidance & tools: ExpandNet resources to guide sustainable scale up of innovations

Key Issues	Specific techniques/research methods	Open access resources
Continue to test different combinations of interventions while collecting metrics on performance and impact Identify aspects of innovation that can be expanded and possible contexts for application Develop communications strategy and risk management plans Integrate community voices and stakeholder engagement (including public officials, local leaders) Communicate findings to all participants and stakeholders Include cost-benefit analysis and an investment plan	Process evaluation: process to determine the development and implementation strategy as planned Economic evaluation: more rigorous economic analyses to examine costs and outcomes (for example, cost-effectiveness) Qualitative research: ethnographic study, in-depth interviews, other methods Participatory action research: qualitative research methods to learn about and engage with participants Time series analysis: serial and comparative interrupted time series analysis Cluster randomized trials: groups are randomized into control and intervention arms Randomized controlled trials: participants are randomly allocated to receive the intervention Observational studies: observing the effect of an intervention without interfering with the exposure or outcome Implementation research: scientific inquiry into questions concerning implementation	Community Based Participatory Action Research: research framework to be conducted in communities Interrupted time series analysis tutorial: stepwise study design to evaluate population-level study at a given point in time Serial and comparative interrupted time series analysis: quantitative measure guide where outcome is evaluated at multiple time points Cluster-randomized trials: Cochrane information guide Observational studies: brief guide to different types of observational studies Nesta – Running RCTs: introductory guide on RCTs in innovation Public and social innovation labs: teams, units and funds dedicated to structuring and embedding innovation methods and practice in government to tackle social and public problems Nesta – People Powered Results: a 100-day challenge methodology that is structured and combined with coaching support

Key Issues	Specific techniques/research methods	Open access resources
Assess performance of social innovation based on pre-established criteria, especially: Community Acceptability Usability Social and cultural impact Effectiveness Accessibility Feasibility Resources Potential sustainability Potential scalability Adaptability Reliance on other systems	Observational studies: non-randomized studies of an innovation (for example, quasi-experimental studies, ecological studies) Implementation research: scientific inquiry into questions concerning implementation Social impact assessment: investigates social effects (that is, non-medical) on end-user population from the innovation Qualitative research: ethnographic study, case study, phenomenology, discourse analysis, grounded theory, in-depth interviews, mixed methods, participant observations Preliminary economic evaluation: costing studies in the local context Randomized controlled trials: participants are randomly allocated to receive the intervention	Implementation Research – A Practical Guide: a guide on implementation research for multiple stakeholders B Impact Assessment: guide to measure impact of a social business Evaluating a Social Innovation: a developmental evaluation approach with an adaptive process Social Innovation Evaluation Toolbox: a tool to assess impact of investment and management of social innovations Cochrane Risk of Bias Tool: information of bias in study designs TDR MOOC on implementation research: an open online training to design and demonstrate robust IR projects for better health outcomes RCT Consort: consolidated standards for reporting randomized controlled trials Nesta – Running RCTs: introductory guide on RCTs in innovation

Key Issues	Specific techniques/research methods	Open access resources
Test different ideas by building tangible products or creating experiences to gather feedback from end users, project delivery staff and key stakeholders Create clear and actionable metrics to test the social innovation Satisfy regulatory and ethical requirements, consider risks and socio-cultural/environmental/ economic/political impact Create a database to capture data about feasibility (given the local context), effectiveness and other key variables	Process evaluation: process to determine the development and implementation strategy as planned Usability: tests the fitness and ease of access of the social innovation Prototype research: studying a sample/design to test the concept or process User-design research: user-focused design studies to meet end users’ needs Intercept interviews: a qualitative market research to collect feedback from consumers	Qualitative Evaluation Checklist: guidance to use qualitative techniques for evaluation purposes Nesta Prototype Framework: structured prototyping toolkit The Most Significant Change Technique: participatory M&E method using systematic participatory interpretation of stories Ethical issues: describes ethical issues associated with open calls TDR MOOC on implementation research: an open online training to design and demonstrate robust IR projects for better health outcomes TDR Implementation Research Toolkit: seven non-linear learning modules on implementation research

Key Issues	Specific techniques/research methods	Open access resources
Build your team with end users, leaders and public and private sector partners Specify the end user population as much as possible and strengthen collaborations with end users Identify overall objectives and aims in partnership with community members Develop your research objective, research design, project plan, roadmap and change theory Build on existing structures, available resources and stakeholder analysis (if applicable) Identify social innovation metrics (key progress indicators) that are specific, measurable, achievable, relevant and culturally appropriate Develop an iterative planning, monitoring, evaluation and feedback plan	Open challenge contests: public open calls to solicit creative solutions Hackathons (also called designathons): a time-bound event to bring together collective ideas to solve a problem User led design-thinking studies: an iterative design-based process/tool that identifies relevant and beneficial ways to meet end users’ needs Participatory learning and action: qualitative research methods to learn about and engage with participants Results chain: develop a visual model that illustrates how from input, through process, outputs and expected outcomes the innovation will contribute to the impact	Design for Health: a design process to develop effective solutions Crowdsourcing Clinic: practical tools on open calls Crowdsourcing in Health – A Practical Guide: guidance on open calls Development Impact & You Innovative Toolkit: practical tools to trigger & support social innovations Participatory Approaches: guide on using participatory methods for impact evaluation Community Based Participatory Action Research: research approach to enhance community-based research Implementation Research in Health – A Practical Guide: a guide on implementation research for multiple stakeholders TDR Massive Open Online Course (MOOC) on Implementation Research (IR): an open online training to design and demonstrate robust IR projects for better health outcomes TDR Implementation Research Toolkit: seven non-linear learning modules on implementation research Results Chain: a guide for enterprises on creating a theory of change and results chain

Key Issues	Specific techniques/research methods	Open access resources
Identify an unmet need according to end users, community and partners (co-creation items highlighted in bold throughout) Identify a group of innovators, end users, partners and key individuals through a stakeholder analysis Consider the opportunities in terms of settings, key stakeholders, beneficiaries (and potential for co-creation), power structures, technology and sustainability Consider problems in terms of necessary staffing, funding, time, populations and partnerships for sustainability Identify existing struggling innovators who can benefit from seed financing, project structuring and design support or linkage with partners that they have limited access to	Scoping reviews: structured literature review synthesizing pre-specified eligibility criteria and existing knowledge Systematic reviews: more extensive literature review with pre-specified eligibility criteria and potential for pooled results Qualitative research: ethnographic study, case study, phenomenology, discourse analysis, grounded theory, in-depth interviews, mixed methods Landscape analysis: policy analysis about needs and opportunities Observational studies: observing the effect of an intervention without interfering with the exposure or outcome	Cochrane Library: systematic reviews PRISMA Statement on Systematic Reviews and Meta Analyses: research checklist Conducting Concerns Surveys: structured way to get community feedback Development Impact & You: Nesta’s toolkit on practical tools to support and trigger ideas for better results Problem Analysis: checklist to discuss and solve problems Qualitative Research Guidelines Project: a comprehensive guide for designing, writing, reviewing and reporting qualitative research Community Based Participatory Research Approach: research approach to enhance community-based research Crowdsourcing Clinic: practical tools on crowdsourcing Crowdsourcing in Health – A Practical Guide: guidance on crowdsourcing Design for Health: a design process to develop effective solutions Implementation Research in Health – A Practical Guide: a guide on implementation research for multiple stakeholders

Key Issues

Specific techniques/research methods

Open access resources

Identify an unmet need according to end users, community and partners (co-creation items highlighted in bold throughout)

Identify a group of innovators, end users, partners and key individuals through a stakeholder analysis

Consider the opportunities in terms of settings, key stakeholders, beneficiaries (and potential for co-creation), power structures, technology and sustainability

Consider problems in terms of necessary staffing, funding, time, populations and partnerships for sustainability

Identify existing struggling innovators who can benefit from seed financing, project structuring and design support or linkage with partners that they have limited access to

Scoping reviews: structured literature review synthesizing pre-specified eligibility criteria and existing knowledge

Systematic reviews: more extensive literature review with pre-specified eligibility criteria and potential for pooled results

Qualitative research: ethnographic study, case study, phenomenology, discourse analysis, grounded theory, in-depth interviews, mixed methods

Landscape analysis: policy analysis about needs and opportunities

Observational studies: observing the effect of an intervention without interfering with the exposure or outcome

Cochrane Library: systematic reviews

PRISMA Statement on Systematic Reviews and Meta Analyses: research checklist

Conducting Concerns Surveys: structured way to get community feedback

Development Impact & You: Nesta’s toolkit on practical tools to support and trigger ideas for better results

Problem Analysis: checklist to discuss and solve problems

Qualitative Research Guidelines Project: a comprehensive guide for designing, writing, reviewing and reporting qualitative research

Community Based Participatory Research Approach: research approach to enhance community-based research

Crowdsourcing Clinic: practical tools on crowdsourcing

Crowdsourcing in Health – A Practical Guide: guidance on crowdsourcing

Design for Health: a design process to develop effective solutions

Implementation Research in Health – A Practical Guide: a guide on implementation research for multiple stakeholders