Understanding social change from outcomes to impact.
In recent years there has been a shift in the field of program evaluation from activities-based to outcomes-based evaluation practice. While an activities-based evaluation focuses on measuring the outputs of a program (i.e. the number of participants, sessions or trees planted etc.), an outcomes-based evaluation focuses on measuring the change that occurred as a result.
For example, while an activities-based evaluation of a health promotion program might look at metrics such as the number of participants, the number of sessions, and the promotional materials distributed, an outcomes based evaluation would examine how much people learned and how their learning improved their ability to manage their health.
Social return on investment: The methodology
The move toward outcomes-based evaluation has moved the focus from how activities intend to achieve change to how much change actually occurs. The next step in this journey is to examine whether the change that actually occurs is happening entirely due to the activities, i.e. were there any other contributing factors involved in the creation of change? Considering outcomes in relation to other contributing factors is known as assessing impact in terms of the Social Return on Investment (SROI) methodology.
The SROI methodology is an outcomes-based evaluation framework that maps program outcomes (the logic model) and then assigns value to these outcomes for the purpose of relating the value of the social change that is achieved to the upfront investment in the activity. Once value has been assigned to outcomes, SROI examines the impact of the program by considering four things: deadweight, displacement, attribution, and drop off.
Impact of social investment: Four things to consider
Deadweight is the change that would have happened anyway. It accounts for any portion of change included in an outcome that would have occurred either due to environmental factors or an internal motivation to change. For example, a program may aim at helping people to get sustainable jobs, but a portion of those people employed would likely have found a job regardless, either due to their own efforts or due to a change in the labour market. The value of those likely to have found a sustainable job anyway would not be counted as part of the overall impact of the program.
Displacement looks at whether the social change due to the intervention displaced something else or had some sort of unintended consequence. For example, a crime prevention initiative may reduce crime in one neighbourhood by unintentionally displacing that crime to another neighbourhood nearby. The value of achieving the outcome of reduced crime in the first neighbourhood would be discounted in order to acknowledge the negative result of increased crime in the second neighbourhood.
Attribution acknowledges that a portion of the change represented by an outcome was due to the contribution of another organization, initiative or person. This acknowledges that social change often results from interconnected initiatives, therefore a portion of the value of the outcome achieved is attributed to other initiatives, organizations or people.
Finally, drop off, considers how a change resulting from a program will diminish over time. For example, if a program helps people to quit smoking, what portion may start smoking again in the future? When considering the value of associated outcomes, the rate of potential and likely drop off needs to be considered while valuing overall program impact.
Overall, by taking into account deadweight, displacement, attribution and drop off in association with any outcomes achieved, it becomes possible to present the value of social impact. As explained throughout A Guide to Social Return on Investment published by the SROI Network in January 2012, establishing the difference between the value of outcomes and the value of impact is very important. Taking into account these four factors significantly reduces the risk of over-claiming the value associated with a program, which means that the story of social change accomplished will be that much more credible.
Stephanie Robertson founded SiMPACT Strategy Group in 2004 and launched LBG Canada in 2005. As the first accredited SROI practitioner in North America, Stephanie is a leading professional in the area of social impact management and measurement.
Anne Miller is Team Lead of the Social Return On Investment (SROI) Initiative at the SiMPACT Strategy Group, a specialist agency in social impact management, measurement and valuation.