Measuring Nonprofits’ Impacts: A Necessary Process for the 21st Century
By Eugene Fram
Nonprofit boards and CEOs in the United States are being overwhelmed with requests from foundations and governmental agencies to move from providing outcome data to providing impact data. One nonprofit with which I am well acquainted has been required to reform its IT program to meet the requirements of a local governmental IT program, so that impacts can be assessed. It will be interesting to see how this scenario plays out.
Unfortunately, outcomes and impact are often unrelated, which is why a program that seems to produce better outcomes may create no impact at all. Worse, sometimes they point in opposite directions, as can happen when a program works with harder-to- service populations resulting in seemingly worse conditions, but (has) higher value-added impact. … Rigorous evaluations can measure impact (to a level of statistical accuracy), but they are usually costly (a nonstarter for many nonprofit), difficult and slow. * But how do the medium and small size nonprofits measure actual results in the outside world such as enhanced quality of life, elevated artistic sensitivity and community commitment?
A Compromise Solution:
To close the gap, funders and recipients would need to agree to apply imperfect metrics over time. These are metrics that can be anecdotal, subjective or interpretative. Also they may rely on small samples, uncontrolled situational factors, or they cannot be precisely replicated. ** This would require agreement and trust between funders and recipients as to what level of imprecision can be accepted and perhaps be improved, to assess impacts. It is an experimental approach
How To Get to Impact Assessment:
1. Agree on relevant impacts: Metrics should be used to reflect organizational related impacts, not activities or efforts. Impacts should focus on a desired change in the nonprofit’s universe, rather than a set of process activities.
2. Agree on measurement approaches: These can range from personal interviews to comparisons of local results with national data.
3. Agree on specific indicators: Outside of available data, such as financial results, and membership numbers, nonprofits should designate behavioral impacts for clients should achieve. Do not add other indicators because they are easily developed or “would be interesting to examine.” Keep the focus on the agreed-upon behavioral outcomes.
4. Agree on judgment rules: Board and management need to agree at the outset upon the metric numbers for each specific indicator that contributes to the desired strategic objective. The rules can also specify values that are “too high” as well as “too low.”
5. Compare measurement outcomes with judgment rules to determine organizational impact: Determine how may specific program objectives have reached impact levels to assess whether or not the organization’s strategic impacts have been achieved.
The five-point process described above closely follows the philosophy of lean experimentation, *** now suggested for profit making and nonprofit organizations.
Lean allows nonprofits to use imperfect metrics to obtain impact data from constituents/ stakeholders over time. Under a lean approach, as long as the organizations garners some positive insights after each iteration, it continues to improve the measurement venues and becomes more comfortable with the advantages and limitations of using these metrics.
Organizationally the nonprofit can use this process to drive change over time by better understanding what is behind the imperfect metrics, especially when a small sample can yield substantial insights, and actually improve the use of the metrics.
I need help to fully understand this: “Unfortunately, outcomes and impact are often unrelated, which is why a program that seems to produce better outcomes may create no impact at all. Worse, sometimes they point in opposite directions, as can happen when a program works with harder-to- service populations resulting in seemingly worse conditions, but (has) higher value-added impact.” Can you please provide at least one example of each?
Beth: Note that this section is a direct quote. The original article has links to all the studies and examples on which the quote was based
In brief, many qualitative impacts are not measured. Example, a couple seeking marriage counseling works with a therapist who uses the latest treatment modalities, and the couple leaves satisfied after a year, but in two years they are having the same problems. It is evident that the modality worked temporarily but the couple did not get at the root of the problem. Accreditation of the agency may be on the basis of the modality used, but really needs to be based on the long term impact.
You also may be interested in these links: Blog site http://bit.ly/yfRZpz
BTW: This post is going viral. Where did you encounter it?
Thank you! I’ve worked for human services nonprofits for a long time and I understand. To answer your question, I received an email notification through LinkedIn.