How Listen4Good has made a difference.
We think it’s important to look at the impact we’ve had on organizations in multiple ways. Our overall focus, though, is simple: To actively review our results and use them to improve what we do. Our own feedback loop!
of Listen4Good organizations make programmatic or operational changes in response to feedback after one year.
of organizations continue to collect feedback one year after graduating from Listen4Good
of organizations would recommend Listen4Good to a peer organization
of funder sponsors would recommend Listen4Good to another funder
During our first five years, Listen4Good was routinely assessed by third-party evaluation firm, ORS Impact, along with a range of other research partners.
ORS Impact surveyed nonprofits one year after the end of their participation in Listen4Good to learn if they were still implementing high-quality feedback loops and what that work looked like after they were no longer receiving support through Listen4Good. To gauge the quality of feedback, organizations were asked to what extent they continued using practices associated with the five steps identified by Listen4Good as comprising a high-quality feedback loop: design, collection, analysis, response, reporting back to survey takers.
- A majority of organizations in the 2017 and 2018 cohorts continued to collect feedback after their participation in Listen4Good ended. The COVID-19 pandemic affected feedback work in different ways, but no organization stopped collecting feedback indefinitely, signaling that feedback work is sustainable, even in challenging times.
- Almost all organizations are engaging high- and medium-quality practices in steps 2 and 3 of the feedback loop, data collection and data analysis. Responding to data and closing the loop continue to be the most difficult steps for organizations.
- There are specific practices within each step of a high-quality feedback loop that can help advance equity, diversity or inclusion (such as by uncovering disparities in services or making the survey process more inclusive). Most organizations are engaging in these practices during data collection and data analysis, but half or less are engaging in these practices in other steps of the loop.
- At least half of the organizations from both cohorts reported that feedback work after Listen4Good was primarily driven by one person tasked with collecting and analyzing data and then sharing it with other staff.
For this report, ORS Impact interviewed co-funders that nominated nonprofits to participate in Listen4Good, examining the extent to which the feedback program may have led to changes in the funders’ thinking, organizational culture, and practices around feedback.
- More than three out of four funders overall reported making changes related to feedback.
- Returning funders, those that nominated nonprofits in more than one Listen4Good cohort, have different experiences than new funders, reporting more concrete changes in feedback practices and being more likely to say that Listen4Good contributed to changes.
- Both new and returning funders mentioned additional benefits of Listen4Good, including their organization’s increased buy-in into the power of feedback; the opportunity to participate in a new/better way of getting feedback; increased funder knowledge and understanding regarding feedback and the capacity needed to do it well: and more activity around closing the loop with clients.
- Most funders overall reported plans to continue their feedback work internally and with their grantees. Many also reported additional feedback work outside of Listen4Good.
- Most funders overall see a relationship between feedback practice and equity, diversity, and inclusion, with the largest share of those funders pointing to a link between feedback and consciously centering clients and valuing their experiences.
In this report, ORS impact takes its first look at Listen4Good Online Beta, a program to test Listen4Good’s new, interactive, feedback-loop building web app. In the summer of 2019, nearly 200 direct-service nonprofits and government agencies were selected to participate in a low-touch Listen4Good program where participants had full access to the web app, but extremely limited access to coaching or other ad-hoc support. This report looks at how the Online Beta participants, halfway through their 18-month engagement with the program, fared on reaching desired outcomes compared with organizations participating in Listen4Good’s co-funded grant program, a much more fully supported model.
Among the key findings
- Overall, we found that Online Beta organizations are progressing more slowly than prior cohorts, though COVID-19 also likely impacted feedback work.
- Online Beta organizations report significantly increased ability to conduct feedback work, which is comparable to past cohorts.
- Online Beta organizations that progress through the feedback cycle report higher organizational impact than prior cohorts and similar results in terms of learning from and responding to feedback.
- Despite these promising signs of outcomes by participating organizations, this cohort reports a significantly lower Net Promoter Score (NPS) than prior cohorts.
- This version of a more scalable model can clearly work for some organizations, even in a challenging external context. More needs to be learned about which organizations don’t progress or who is not served well by this version of L4G.
This report by ORS Impact, Fund for Shared Insight’s learning and evaluation partner, looks at the impact of Listen4Good on the funders and nonprofits that participated in the feedback program in its first four years.
- L4G has had strong impacts across three cohorts of nonprofits on their capacity to collect feedback, gain insights, take actions, and see organizational effects by the end of the L4G grant. For example, 81 percent of participating nonprofits made at least one change in response to feedback in programming, operations, client-staff interactions, or offering new services.
- L4G has had some impacts on co-funders, despite Shared Insight’s light touch approach to engaging with them. For example, 33 percent of interviewed co-funders reported changes to their foundation practices related to feedback from grantees or communities and mentioned that L4G contributed to that change. (Since Shared Insight began increasing its engagement activities with co-funders beginning with the 2019 cohort, ORS will assess — and include in future reports — whether these increased efforts lead to better outcomes.)
- L4G has made some contributions to meaningfully connecting nonprofits and foundations, but contributions are uneven, and there is opportunity for greater impact.
- There is some progress — and more opportunity — for using feedback to advance equity and inclusion. For example, Shared Insight might further assist more organizations in segmenting data by sub-groups to identify differences in client experience, interpreting findings and developing appropriate responses to those differences.
In early 2017, Shared Insight engaged Harder+Company Community Research to analyze the L4G surveys’ aggregate dataset, consisting of more than 29,000 responses; summarize the data and identify patterns and/or differences in respondent feedback; and assess the effectiveness and utility of the Net Promoter System overall and specifically in the nonprofit context.
- A majority of respondents provided very positive feedback to the organizations that served them.
- Youth under age 18 and people receiving education-related services gave significantly lower ratings on survey questions than did other respondents.
- The Listen4Good survey tool is understandable to the people participating nonprofits serve.
- This analysis provides some evidence that the Net Promoter System works in the nonprofit context. It also reveals the need for further exploration.
Over the past three years, ORS Impact has served as learning and evaluation partner to Shared Insight and L4G. This report analyzes data collected from 104 out of the 112 nonprofit partners from the 2016 and 2017 cohorts, providing a high-level view of how L4G has worked across the two grantmaking periods, and identifying key lessons and implications for new L4G partners under the current program model, for planned changes to the program, and for evaluation efforts moving forward.
L4G Impact on Nonprofits
- Organizations from both cohorts report improvements in their ability to implement feedback-related tasks, but there may be a ceiling effect to this growth.
- Organizations with higher ability to implement feedback loops are more likely to see impact on program effectiveness.
- L4G provides organizational insights through both new information and data to confirm prior beliefs.
- Organizations that gain more insights are more likely to see impact on program effectiveness.
- Organizations are translating these insights into changes to better serve clients.
- L4G advances organizations’ equity, diversity, and inclusion work by providing clients a seat at the table at various steps in the process and allowing them to shape the services that benefit their communities.
- L4G improves organizations’ ability to serve clients while changing organizational culture and values around listening to the people they seek to help.
- All organizations plan to continue their feedback practice after their L4G grant.
- L4G increases leadership and staff commitment to feedback practice.
L4G’s Support Model
- Organizations find the L4G model effective in supporting their feedback practice.
- The amount of TA provided has remained adequate despite changes in the model, and organizations highly value the SurveyMonkey platform.
Listen4Good 201602017 Cohort Findings by ORS Impact is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License