Judging Social Innovation in Hackathons

Given the importance of social innovation monitoring and evaluation, these tools should also be considered in the context of specific events such as hackathons. Hackathons (also called design challenges, innovation pitches, innovation challenges and related events) to identify social innovations have expanded in the past five years. A hackathon is a multistage process that brings together diverse individuals to solve a problem.^9,10 Hackathons have been used in many medical and public health settings, including health systems, health policy, communications, community engagement and clinical medicine. Early hackathons focused on bringing together diverse groups to develop software.¹¹ Since then hackathons have expanded to enhance educational programmes^12,13 and develop new social care interventions for vulnerable populations.^14-16 In the COVID-19 era, hackathons have been further adapted to develop new technology for COVID-19 responses (for example, mobile phone applications for contact tracing) and deal with the post-crisis era.¹⁷

Several types of monitoring and evaluation information may help judge social innovations as part of hackathons. Box 2 below outlines several considerations related to monitoring and evaluation of hackathons, organized by stage.

Box 2. Key questions related to assessing hackathons (organized by stage)

Steering committee. How was the overall topic for the hackathon established? Did the steering committee include individuals with knowledge of monitoring and evaluation? Has the steering committee developed a hackathon plan that engages potential participants and encourages them to bring evidence to support their social innovation?

Community engagement. Was the hackathon disseminated in diverse forums to ensure broad participation (especially related to location, gender, ethnicity and disability)?

Judging innovations. Was there a prior plan for judging criteria that was used? Do judges include end users and other key stakeholders? Do judging criteria include an assessment of health and non-health impact? Did the social innovation team describe a plan for formal evaluation of their idea?

Recognizing finalists. Did the prize structure provide mentorship, monetary support or other implementation support for monitoring and evaluation of the social innovation? Do the hackathon organizers track finalists for use in the community?

Key Issues	Specific techniques/research methods	Open access resources
Create marketing/dissemination strategy Document and share the process to create the social innovation in health Document and broadly share the impact of the social innovation in health Develop and strengthen partnerships with supporting market actors who can sustain the model Create systems and/or integrate into existing systems and structures to institutionalize the social innovation Identify unmet needs Adaptation and replication in other settings	Documenting best practices: methods proven reliable to lead desirable results Impact analysis: assess the results of the project Delphi techniques: a structured way to aggregate knowledge from a group of people, often used in health guideline development Mathematical modelling: using mathematical formulations to guide problems	WHO Best Practices Guide: documents criteria and format to analyse best practices Impact Assessment: guide to conduct an impact analysis Implementation Research in Health – A Practical Guide: a guide on implementation research for multiple stakeholders Reporting guidelines for modelling studies: evidence-based tools to guide research reporting Nesta’s Funding Innovation – A Practice Guide: a guide to different financial tools to support innovations Nesta’s tools and insights for governments: a guide to improve governance for innovations Anticipatory regulation: an emerging approach that is proactive, iterative and responds to evolving markets Strengthening evaluation capacity: a guide on strengthening the M&E capacity of individuals, organizations, communities and networks

Key Issues	Specific techniques/research methods	Open access resources
Define longer-term metrics of impact, including fidelity and related implementation metrics Assess potential to scale down to a specific local context or be adapted for another context Assess potential for dissemination of innovation to different contexts and cultures Develop impact and sustainability metrics with stakeholders and end users	Social impact assessment: investigates social effects (that is, non-medical) on end-user population from the innovation guideline development – dissemination of ideas and best practices Implementation research: implementation across subunits/network identi-fication/confirmatory inferences designs/effectiveness-implementation hybrids Qualitative research: ethnographic study, in-depth interviews, other methods Discrete choice experiments: quantitative measure to elicit individual preferences on innovations	Effectiveness-Implementation Hybrid Designs: assess both clinical outcomes and implementation Social Network Analysis: guide to map relationships and networks in communities Adaptive trial designs: guide on designing adaptive clinical trials Discrete Choice Experiment: WHO guide on conducting discrete choice experiments Scaling-up guidance & tools: ExpandNet resources to guide sustainable scale up of innovations

Key Issues	Specific techniques/research methods	Open access resources
Continue to test different combinations of interventions while collecting metrics on performance and impact Identify aspects of innovation that can be expanded and possible contexts for application Develop communications strategy and risk management plans Integrate community voices and stakeholder engagement (including public officials, local leaders) Communicate findings to all participants and stakeholders Include cost-benefit analysis and an investment plan	Process evaluation: process to determine the development and implementation strategy as planned Economic evaluation: more rigorous economic analyses to examine costs and outcomes (for example, cost-effectiveness) Qualitative research: ethnographic study, in-depth interviews, other methods Participatory action research: qualitative research methods to learn about and engage with participants Time series analysis: serial and comparative interrupted time series analysis Cluster randomized trials: groups are randomized into control and intervention arms Randomized controlled trials: participants are randomly allocated to receive the intervention Observational studies: observing the effect of an intervention without interfering with the exposure or outcome Implementation research: scientific inquiry into questions concerning implementation	Community Based Participatory Action Research: research framework to be conducted in communities Interrupted time series analysis tutorial: stepwise study design to evaluate population-level study at a given point in time Serial and comparative interrupted time series analysis: quantitative measure guide where outcome is evaluated at multiple time points Cluster-randomized trials: Cochrane information guide Observational studies: brief guide to different types of observational studies Nesta – Running RCTs: introductory guide on RCTs in innovation Public and social innovation labs: teams, units and funds dedicated to structuring and embedding innovation methods and practice in government to tackle social and public problems Nesta – People Powered Results: a 100-day challenge methodology that is structured and combined with coaching support

Key Issues	Specific techniques/research methods	Open access resources
Assess performance of social innovation based on pre-established criteria, especially: Community Acceptability Usability Social and cultural impact Effectiveness Accessibility Feasibility Resources Potential sustainability Potential scalability Adaptability Reliance on other systems	Observational studies: non-randomized studies of an innovation (for example, quasi-experimental studies, ecological studies) Implementation research: scientific inquiry into questions concerning implementation Social impact assessment: investigates social effects (that is, non-medical) on end-user population from the innovation Qualitative research: ethnographic study, case study, phenomenology, discourse analysis, grounded theory, in-depth interviews, mixed methods, participant observations Preliminary economic evaluation: costing studies in the local context Randomized controlled trials: participants are randomly allocated to receive the intervention	Implementation Research – A Practical Guide: a guide on implementation research for multiple stakeholders B Impact Assessment: guide to measure impact of a social business Evaluating a Social Innovation: a developmental evaluation approach with an adaptive process Social Innovation Evaluation Toolbox: a tool to assess impact of investment and management of social innovations Cochrane Risk of Bias Tool: information of bias in study designs TDR MOOC on implementation research: an open online training to design and demonstrate robust IR projects for better health outcomes RCT Consort: consolidated standards for reporting randomized controlled trials Nesta – Running RCTs: introductory guide on RCTs in innovation

Key Issues	Specific techniques/research methods	Open access resources
Test different ideas by building tangible products or creating experiences to gather feedback from end users, project delivery staff and key stakeholders Create clear and actionable metrics to test the social innovation Satisfy regulatory and ethical requirements, consider risks and socio-cultural/environmental/ economic/political impact Create a database to capture data about feasibility (given the local context), effectiveness and other key variables	Process evaluation: process to determine the development and implementation strategy as planned Usability: tests the fitness and ease of access of the social innovation Prototype research: studying a sample/design to test the concept or process User-design research: user-focused design studies to meet end users’ needs Intercept interviews: a qualitative market research to collect feedback from consumers	Qualitative Evaluation Checklist: guidance to use qualitative techniques for evaluation purposes Nesta Prototype Framework: structured prototyping toolkit The Most Significant Change Technique: participatory M&E method using systematic participatory interpretation of stories Ethical issues: describes ethical issues associated with open calls TDR MOOC on implementation research: an open online training to design and demonstrate robust IR projects for better health outcomes TDR Implementation Research Toolkit: seven non-linear learning modules on implementation research

Key Issues	Specific techniques/research methods	Open access resources
Build your team with end users, leaders and public and private sector partners Specify the end user population as much as possible and strengthen collaborations with end users Identify overall objectives and aims in partnership with community members Develop your research objective, research design, project plan, roadmap and change theory Build on existing structures, available resources and stakeholder analysis (if applicable) Identify social innovation metrics (key progress indicators) that are specific, measurable, achievable, relevant and culturally appropriate Develop an iterative planning, monitoring, evaluation and feedback plan	Open challenge contests: public open calls to solicit creative solutions Hackathons (also called designathons): a time-bound event to bring together collective ideas to solve a problem User led design-thinking studies: an iterative design-based process/tool that identifies relevant and beneficial ways to meet end users’ needs Participatory learning and action: qualitative research methods to learn about and engage with participants Results chain: develop a visual model that illustrates how from input, through process, outputs and expected outcomes the innovation will contribute to the impact	Design for Health: a design process to develop effective solutions Crowdsourcing Clinic: practical tools on open calls Crowdsourcing in Health – A Practical Guide: guidance on open calls Development Impact & You Innovative Toolkit: practical tools to trigger & support social innovations Participatory Approaches: guide on using participatory methods for impact evaluation Community Based Participatory Action Research: research approach to enhance community-based research Implementation Research in Health – A Practical Guide: a guide on implementation research for multiple stakeholders TDR Massive Open Online Course (MOOC) on Implementation Research (IR): an open online training to design and demonstrate robust IR projects for better health outcomes TDR Implementation Research Toolkit: seven non-linear learning modules on implementation research Results Chain: a guide for enterprises on creating a theory of change and results chain

Key Issues	Specific techniques/research methods	Open access resources
Identify an unmet need according to end users, community and partners (co-creation items highlighted in bold throughout) Identify a group of innovators, end users, partners and key individuals through a stakeholder analysis Consider the opportunities in terms of settings, key stakeholders, beneficiaries (and potential for co-creation), power structures, technology and sustainability Consider problems in terms of necessary staffing, funding, time, populations and partnerships for sustainability Identify existing struggling innovators who can benefit from seed financing, project structuring and design support or linkage with partners that they have limited access to	Scoping reviews: structured literature review synthesizing pre-specified eligibility criteria and existing knowledge Systematic reviews: more extensive literature review with pre-specified eligibility criteria and potential for pooled results Qualitative research: ethnographic study, case study, phenomenology, discourse analysis, grounded theory, in-depth interviews, mixed methods Landscape analysis: policy analysis about needs and opportunities Observational studies: observing the effect of an intervention without interfering with the exposure or outcome	Cochrane Library: systematic reviews PRISMA Statement on Systematic Reviews and Meta Analyses: research checklist Conducting Concerns Surveys: structured way to get community feedback Development Impact & You: Nesta’s toolkit on practical tools to support and trigger ideas for better results Problem Analysis: checklist to discuss and solve problems Qualitative Research Guidelines Project: a comprehensive guide for designing, writing, reviewing and reporting qualitative research Community Based Participatory Research Approach: research approach to enhance community-based research Crowdsourcing Clinic: practical tools on crowdsourcing Crowdsourcing in Health – A Practical Guide: guidance on crowdsourcing Design for Health: a design process to develop effective solutions Implementation Research in Health – A Practical Guide: a guide on implementation research for multiple stakeholders

Key Issues

Specific techniques/research methods

Open access resources

Identify an unmet need according to end users, community and partners (co-creation items highlighted in bold throughout)

Identify a group of innovators, end users, partners and key individuals through a stakeholder analysis

Consider the opportunities in terms of settings, key stakeholders, beneficiaries (and potential for co-creation), power structures, technology and sustainability

Consider problems in terms of necessary staffing, funding, time, populations and partnerships for sustainability

Identify existing struggling innovators who can benefit from seed financing, project structuring and design support or linkage with partners that they have limited access to

Scoping reviews: structured literature review synthesizing pre-specified eligibility criteria and existing knowledge

Systematic reviews: more extensive literature review with pre-specified eligibility criteria and potential for pooled results

Qualitative research: ethnographic study, case study, phenomenology, discourse analysis, grounded theory, in-depth interviews, mixed methods

Landscape analysis: policy analysis about needs and opportunities

Observational studies: observing the effect of an intervention without interfering with the exposure or outcome

Cochrane Library: systematic reviews

PRISMA Statement on Systematic Reviews and Meta Analyses: research checklist

Conducting Concerns Surveys: structured way to get community feedback

Development Impact & You: Nesta’s toolkit on practical tools to support and trigger ideas for better results

Problem Analysis: checklist to discuss and solve problems

Qualitative Research Guidelines Project: a comprehensive guide for designing, writing, reviewing and reporting qualitative research

Community Based Participatory Research Approach: research approach to enhance community-based research

Crowdsourcing Clinic: practical tools on crowdsourcing

Crowdsourcing in Health – A Practical Guide: guidance on crowdsourcing

Design for Health: a design process to develop effective solutions

Implementation Research in Health – A Practical Guide: a guide on implementation research for multiple stakeholders

Social Innovation in Health
Monitoring &
Evaluation Framework

Social Innovation in Health
Monitoring &
Evaluation Framework

Judging Social Innovation in Hackathons

Box 2. Key questions related to assessing hackathons (organized by stage)

Social
Innovation
in Health
Initiative

Social Innovation in Health

Monitoring &
Evaluation Framework

Phase 7
Changing systems: integrating the social innovation into routine practice

Key Issues

Specific techniques/research methods

Open access resources

Phase 6
Growing and scaling: scaling up impact or scaling down for local relevance

Key Issues

Specific techniques/research methods

Open access resources

Phase 5
Delivering and implementing: expanding a social innovation project and/or research study to show impact

Key Issues

Specific techniques/research methods

Open access resources

Phase 4
Making the case: evaluating a social innovation project and/or research study

Key Issues

Specific techniques/research methods

Open access resources

Phase 3
Developing and testing: piloting a social innovation project and/or research study

Key Issues

Specific techniques/research methods

Open access resources

Phase 2
Generating ideas: developing a social innovation project and/or research study

Key Issues

Specific techniques/research methods

Open access resources

Phase 1
Opportunities and challenges: identifying a need and opportunity for social innovation in health

Key Issues

Specific techniques/research methods

Open access resources

Social Innovation in Health Monitoring & Evaluation Framework

Social Innovation in Health Monitoring & Evaluation Framework

Judging Social Innovation in Hackathons

Box 2. Key questions related to assessing hackathons (organized by stage)

SocialInnovationin HealthInitiative

Social Innovation in Health

Monitoring & Evaluation Framework

Phase 7 Changing systems: integrating the social innovation into routine practice

Key Issues

Specific techniques/research methods

Open access resources

Phase 6 Growing and scaling: scaling up impact or scaling down for local relevance

Key Issues

Specific techniques/research methods

Open access resources

Phase 5 Delivering and implementing: expanding a social innovation project and/or research study to show impact

Key Issues

Specific techniques/research methods

Open access resources

Phase 4 Making the case: evaluating a social innovation project and/or research study

Key Issues

Specific techniques/research methods

Open access resources

Phase 3 Developing and testing: piloting a social innovation project and/or research study

Key Issues

Specific techniques/research methods

Open access resources

Phase 2 Generating ideas: developing a social innovation project and/or research study

Key Issues

Specific techniques/research methods

Open access resources

Phase 1 Opportunities and challenges: identifying a need and opportunity for social innovation in health

Key Issues

Specific techniques/research methods

Open access resources

Social Innovation in Health
Monitoring &
Evaluation Framework

Social Innovation in Health
Monitoring &
Evaluation Framework

Social
Innovation
in Health
Initiative

Monitoring &
Evaluation Framework

Phase 7
Changing systems: integrating the social innovation into routine practice

Phase 6
Growing and scaling: scaling up impact or scaling down for local relevance

Phase 5
Delivering and implementing: expanding a social innovation project and/or research study to show impact

Phase 4
Making the case: evaluating a social innovation project and/or research study

Phase 3
Developing and testing: piloting a social innovation project and/or research study

Phase 2
Generating ideas: developing a social innovation project and/or research study

Phase 1
Opportunities and challenges: identifying a need and opportunity for social innovation in health