When we eavesdrop on conversations surrounding impact, it always seems that it’s about measurement. Yet to us, as Geneviève Leclerc, founder of the ##MEET4IMPACT project, writes, measurement should be perceived as a tool for understanding impact, not as the end goal. Metrics are not enough to fuel an organization’s wish to capture and amplify the overall value it creates. However, using data to assess their impact should be part of an overall impact management program that will help demonstrate their relevance.
In a previous article we mentioned that the practice of social impact measurement relies on the utilization of “impact” as a currency by which an organization is able to report on its organizational effectiveness in carrying out its mission; much like one would report on financial results. To measure the social impact of its activities, such as an event, the association will want to assess the effects on the local host communities, on the event participants, and on its community of interest as a whole. But how to measure the intangible?
Designing Your Data Strategy
The third phase in our impact management program is about learning to measure what matters. Developing a robust data strategy that will support your whole impact journey is key to demonstrate the impact that you have planned for.
Collecting and analyzing the right data informs you on how efficient your initiative is. Initially, you must think of the entities to which you wish to report to, and what their information needs are:
- How can you report on the positive change you claim you’re responsible for?
- What information is relevant and informative to that effect?
- How can you back up your claims?
You will then plan your strategy around the collection of data informing on the progression of impact over time and relevant to the above questions.
Your strategy will be dictated by three major questions: 1) What are you assessing? 2) What is the data collection process? 3) What purpose will it serve and how will it be reported?
All About Indicators
We already wrote about the Theory of Change (ToC) to guide your impact project, which is a tool used to model how an activity and its short-term results can lead to a longer-term impact. When developing your ToC, it is important to understand how the results on objectives can be evaluated.
Each of the objectives selected should be supported by information/data that can be evaluated in one of four ways:
- Qualitatively, by demonstrating a visible change;
- Quantitatively, with figures that may indicate a change;
- Strategically, by attributing results to previously stated objectives or fixed guidelines;
- In a transposed manner, by giving a monetary value to a social gain.
Measuring impact is not an exact science, but it suggests that one must adopt indicators that will make it possible to attribute a certain ownership (of the impact generated) to the organization. Indicators are generally defined as a specific, observable and measurable characteristic. It will show a change rather than just the performance of our activities, and that change measured should represent progress toward achieving a specific outcome. And indicators should give a relatively good idea of the data required and the population amongst whom they are measured.
There are dozens of existing sets of social impact indicators that you can choose from (such as those offered by GRI, IRIS, The Global Exchange, Guidestar, etc.). But some metrics may be missing as most catalogs contain too few indicators linked to knowledge transfer, professional development and innovation growth, which are typical outcomes linked to business events. Therefore, our sector has to develop or adapt its own set of indicators, an endeavor which #MEET4IMPACT has undertaken with the collaboration of industry experts and academics.
Collecting Data
Let’s now go over some notions about impact data.
First, in order to understand the range of outcomes being generated by your project, you’ll need different types of data. When assessing impact, the most common types of data collected will comprise: user data (who participated? Who is impacted?); participation and engagement data (linked to the activities and output); feedback data (user or participant evaluations); outcome data (perceptions of what they got out of it on the short-term); impact data (mid and longer term—the difference your activity has made); and industry & research data (external sources backing up your claims).
Secondly, data may come from different sources. Some are subjective such as the participant or the beneficiary of the impact being surveyed; others can be objective like the one collected by yourself or a third-party, such as measurable variations in practice or performance.
| Need help in developing your legacy project?
Meet4impact isa global not-for-profit aiming to build a community passionate about social impact in our sector, helping organisations increase their capacity to generate impact through their activities, deliver more value on their mission and implement positive change. Keep looking out for updates on social media following the #Meet4impact/ #Associations4impact/ #Cities4Impact keywords; check out our website www.meet4impact.global; or write to us at community@meet4impact.global to tell us your story.  |
Thirdly, data will be available at different times as impact is not a static occurrence. Impact theory tells us that outcomes and impact will evolve over time, over three major levels:
First level outcomes: observed immediately after an activity and are considered a “Change in Capacity”. It’s the acquisition by a participant of new information, new knowledge, new skills, or an increase in connectivity by meeting new people. Together these nearly immediate results increase the capacity of a person to convert awareness into action.
Second level outcomes: broader, they indicate that more important transformations have occurred as a result of an activity. When one acquires new information, and decides to consider it, research it, and incorporate its new learnings into its practice, there has been a “Change in Behavior”. This is where true impact starts to occur, as an observable change in practice can lead to a change in performance.
Third level outcomes: longer-term impact. They are a “Change in Condition or Status” and are very difficult to track and measure. Most often, a lot of time has passed before they are observed, we might no longer have access to the data readily, and various external factors have contributed to change occurring.
The reason why this is relevant is because each of the previous levels in our logic chain is a pre-condition for the next one to arise. In proper impact practice, the data collection will be done for all of these levels, including the output level (the activity itself). Each level of outcomes should be assessed via its own indicators, therefore giving a better portrait of the evolution in the outcomes and providing richer data.
Now an example. If a meeting aims to increase awareness of the public and decision makers on a particular medical issue, the indicators selected would inform: 1) how broadly the information is disseminated (indicators of exposure, such as media presence around the campaign & tracking of social media postings); 2) how it is received (monitor interactions with the content on social media, observable reactions, on-the-spot interviews with our audience being exposed to the campaign); 3) how it is being used and converted into change of behavior (both self-assessment surveys and third-party external sources should be deployed in this case); and if one has the resources and the time to commit, 4) whether this has improved health outcomes for the population that was targeted over a period of time (this data would be more difficult to compile, would require relationships and engagement of health officials and would demand that baseline data had been previously identified to be able to determine if a positive change occurred).
Finally, various data collection methods will be used, depending on what is being measured and how it will be used. Surveys, focus groups and interviews collecting anecdotal evidence are best to demonstrate that a change has occurred before it can be measured and are very effective at providing “emotionally convincing” data and storytelling, but there may be low return rates and the information can be difficult to evaluate and quantify. The study of documents and case studies can be fairly quick and easy to implement and yield relatively reliable findings, but they may not be useful in demonstrating a cause to effect relationship or identify what triggered the change. And standardized tests and measurements will provide accurate and reliable “hard” data, but they are the most time consuming and often require external expertise. While the quantitative data will tell you how much something has changed, the qualitative data is essential to understanding the nature of what has changed, and how this has affected the beneficiaries of that change.
Using Data
Impact measurement is a process that should always be guided by the need for reporting. In other words, why you need the data will dictate your data strategy. Reporting on impact typically serves three purpose: communicating to stakeholders, being held accountable, and learning from the process. Social impact data analysis should ultimately be used for decision-making purposes—the story told by the data should support a broader strategy and inform on the success of stakeholders in achieving their goals. At all times, a mix of types of data, sources, measurement methods and evaluation at different time periods should be used, and this constitutes a best practice in tracking and reporting the social impact of your activity.
This article was written by Boardroom partner Genevieve Leclerc. The right to use, part or all of it in subsequent works has to be granted by the Publisher.