This blog post begins a periodic series of interviews with people working on various aspects of building networks for social change. Susan Foster is a consultant who specializes in evaluating networks, who I have worked with on the Massachusetts Global Warming Solutions Project. The field of evaluation has grown as foundations and governments seek to monitor and evaluate the effectiveness of their grants and initiatives.

1. How did you get involved in working on networks for social change?
I began to focus on networks and systems change when I switched careers from clinical social work to working for Abt Associates on how to improve outcomes across populations. During that time, it was recognized that people with complex medical and mental health issues were receiving fragmented services from many agencies. The federal agencies involved began looking at how to use networks to increase coordination of care, improve collaboration and create efficiencies. I had the opportunity to work on multiple projects that evaluated cross-sector efforts to improve outcomes for families and improve and integrate service systems.

2. What is different in evaluating network approaches when you compare it to how you would evaluate an organization or a program?
It is much harder to evaluate a network – one of the most challenging issues in the field is to measure impact. With an organization, it is relatively straightforward to look at a particular intervention and assess results, e.g., with a homelessness program you can track how many days were people in a shelter prior to and following a housing intervention. With networks, you need to look beyond individual outcomes to how the system changed, how the network influenced policy, or how the network created change at the local, county, or state level. When many actors and complex processes are in play, this creates challenges of attribution.

In a network, more dimensions need to be considered, including both the process and the outcomes. Some funders get frustrated with the emphasis on how networks grow and develop. They want a bottom line answer: “what did they achieve?” I feel strongly that how networks develop, the relationships they build, the collaborations that occur along the way, the conflicts that arise, who participates and how – these things make a difference in terms of understanding the impacts and results a network achieves.

For example, Healthy Start is a large multi-agency initiative across the US that was created to improve birth outcomes in low-income communities. Healthy Start provides services, but it also creates multi-agency networks to promote broader system changes. An early evaluation focused on quantitative, individual outcomes and found that infant mortality did not decrease significantly. Some saw this as a program failure, but subsequent evaluations examined both individual and system changes and found positive results in intermediate outcomes like utilization of prenatal care, behavior changes, and having a medical home.  All of these are associated with better birth outcomes, and the evaluation documented ways in which successful network collaboration and advocacy had resulted in changes that led to those outcomes.

3. How have you seen approaches to evaluate networks evolve in the fields you work in?
In the mid 1990’s, the evaluations tended to emphasize case studies, interviews, and site visits, e.g., we would visit a site for 2-3 days, interview 20-25 people, and write a 35 page prose report. As the field developed, the tools and methods got more sophisticated. Two key developments were:

• The Wilder Foundation in St. Paul, MN created the Collaboration Factors Inventory for networks based on years of experience in evaluating collaborative entities. This was a great self-assessment tool, e.g., to evaluate how well the network dealt with conflict, engaged the right mix of people.
• Social network analysis (SNA) became more integrated with traditional evaluation methods.

Another major shift is that early on, it was thought that had to be objective independent researchers with no relationship to the participants. Now the field has shifted and many evaluators have embraced the notion of participatory research and action research. Their research methods are embedded in community; the people being evaluated play an active role in the evaluation and in exploring the questions being asked.

4. Networks evolve and grow. What does that mean for how you evaluate them year to year?
In first year of a network, the work focuses on understanding the issue to be addressed and mapping the wider context of how various organizations and networks are already operating. Social network analysis maps can be quite helpful so people can see their place in the larger ecosystem. You want everyone participating to see the necessity of the network and where there are gaps/overlaps and then build toward a collective agenda.

Evaluation in the first year of a network focuses on telling the story of how the network came to focus on core strategies. The focus is on process and assessing how the network is creating the conditions to generate impact.

In subsequent years, as a network moves into action, its impacts can start to be evaluated. For example, with a policy advocacy network the evaluator can interview people in state agencies involved in legislation to assess how the network and/or its members affected policy and legislation. With the Global Warming Solutions Project, when we interviewed state agency staff, a big theme was that they had people from five sectors delivering common messages about policy solutions, which had more impact than just environmental groups advocating for policies.

5. What resources would you recommend for people interested in the latest research or conversations from the field of evaluation and networks?

Catalyzing Networks for Social Change: A Funder’s Guide. Scearce, Diana, Monitor Institute and Grantmakers for Effective Organizations

Innovations for Scaling Impact and Keystone Accountability (June, 2010). Next Generation Network Evaluation

Knowledge Networks: Guidelines for Assessment. Creech, H., and Ramji, A – International Institute for Sustainable Development (2004)

NET GAINS: A Handbook for Network Builders Seeking Social Change. Plastrik, P. and Taylor, M (2006)

Network Effectiveness Diagnostic and Development Tool – Monitor Institute (2010)

Wilder Collaboration Factors Inventory. Wilder Research, Mattessich, P., Murray-Close, M., and Monsey, B. (2001)

6. What are example(s) of networks you have seen that have worked really well?
In my experience in the early childhood field, the networks that worked best did several things well:

• They brought the latest research and data to bear on the issue they wanted to address, e.g., to promote benefits of addressing mental health in young children, they brought experts in and incorporated their research into the advocacy.
• They created connections at all levels of government and across unlikely players. For example, the federal funding agency, HRSA, set guidelines for state grantees on network development and programming. At the state and local level, they connected representatives from departments of public health, education, early childhood, special education, and mental health. This enabled agencies to facilitate smooth transitions from early childhood programs to public school and created connections across agencies working with same kids.
• The effective networks had the right people at the table, including decision makers and the people being served (e.g., families or organizations representing them, advocates for underserved populations.)
• The best networks had strong visionary leadership and someone who got the job done on the ground with details of keeping communication going, strong facilitation, and follow up between meetings.

The network structures that did not work brought “everyone and their brother” to the table in a situation where no one knew what the mission was. In these cases, lots of networking happened but they did not move to strategic action. Over time, people just stopped attending because it was not clear why they were there.

7. What were common missteps or lessons learned in networks you have worked with?
I have worked for over 10 years with a large federally funded collaborative initiative to address children’s mental health. It is based on the premise that kids with significant mental health issues and their families needed to be served through coordinated systems of care rather than individual agencies each working independently. The initial assumption that lasted for years was that by developing a more collaborative, coordinated system of care that individual child outcomes would improve.

While child outcomes and family satisfaction improved, stakeholders learned that they had focused so extensively on building the networks and coordinated systems, they had lost track of basic quality of care issues: on having highly skilled therapists trained in cutting edge practices that actually work. Now there is more of a thrust towards combining network approaches with a focus on high quality care informed by evidence.

Another key lesson that grantees learned over and over again was how important it is to get the right people at the table for the right reasons. In one network, for three of five years of a grant they did not have decision makers in the room. Those involved in service delivery did a lot of talking and collaborating but without the decision makers in the room they could not make systemic changes or sustain the program once federal funding ended.

Also, a common dynamic that inhibits networks gaining traction is that funders who convene and fund the network do not put enough thought into what it takes to create meaningful partnerships. A common problem is that one organization is given a sizable amount of money to run a network but they do not share the wealth so there are fewer incentives for other agencies or organizations to participate. It is important to look at how funding is allocated across a network and how people are supported to participate. It is also helpful to provide up front education about network theory, how to build governance structures, how to set expectations, and how to develop partnerships.