Measuring Nonprofits’ Impacts: A Necessary Process for the 21st Century

By Eugene Fram

Nonprofit boards and CEOs in the United States are being overwhelmed with requests from foundations and governmental agencies to move from providing outcome data to providing impact data. One nonprofit with which I am well acquainted has been required to reform its IT program to meet the requirements of a local governmental IT program, so that impacts can be assessed. It will be interesting to see how this scenario plays out.

Unfortunately, outcomes and impact are often unrelated, which is why a program that seems to produce better outcomes may create no impact at all. Worse, sometimes they point in opposite directions, as can happen when a program works with harder-to- service populations resulting in seemingly worse conditions, but (has) higher value-added impact. … Rigorous evaluations can measure impact (to a level of statistical accuracy), but they are usually costly (a nonstarter for many nonprofit), difficult and slow. * But how do the medium and small size nonprofits measure actual results in the outside world such as enhanced quality of life, elevated artistic sensitivity and community commitment?

A Compromise Solution:

To close the gap, funders and recipients would need to agree to apply imperfect metrics over time. These are metrics that can be anecdotal, subjective or interpretative. Also they may rely on small samples, uncontrolled situational factors, or they cannot be precisely replicated. ** This would require agreement and trust between funders and recipients as to what level of imprecision can be accepted and perhaps be improved, to assess impacts. It is an experimental approach

How To Get to Impact Assessment:

1. Agree on relevant impacts: Metrics should be used to reflect organizational related impacts, not activities or efforts. Impacts should focus on a desired change in the nonprofit’s universe, rather than a set of process activities.
2. Agree on measurement approaches: These can range from personal interviews to comparisons of local results with national data.
3. Agree on specific indicators: Outside of available data, such as financial results, and membership numbers, nonprofits should designate behavioral impacts for clients should achieve. Do not add other indicators because they are easily developed or “would be interesting to examine.” Keep the focus on the agreed-upon behavioral outcomes.
4. Agree on judgment rules: Board and management need to agree at the outset upon the metric numbers for each specific indicator that contributes to the desired strategic objective. The rules can also specify values that are “too high” as well as “too low.”
5. Compare measurement outcomes with judgment rules to determine organizational impact: Determine how may specific program objectives have reached impact levels to assess whether or not the organization’s strategic impacts have been achieved.

Lean Experimentation

The five-point process described above closely follows the philosophy of lean experimentation, *** now suggested for profit making and nonprofit organizations.

Lean allows nonprofits to use imperfect metrics to obtain impact data from constituents/ stakeholders over time. Under a lean approach, as long as the organizations garners some positive insights after each iteration, it continues to improve the measurement venues and becomes more comfortable with the advantages and limitations of using these metrics.

Organizationally the nonprofit can use this process to drive change over time by better understanding what is behind the imperfect metrics, especially when a small sample can yield substantial insights, and actually improve the use of the metrics.

* http://ssir.org/articles/entry/the_promise_and_peril_of_an_outcomes_mindset
** https://nonprofitquarterly.org/2012/07/24/using-imperfect-metrics-well-tracking-progress-and-driving-change/
*** http://ssir.org/articles/entry/the_promise_of_lean_experimentation