Measuring Health and Capacity

Why Measure Ecosystem Health?

Civic data ecosystems are complex and challenging to assess. But measuring the health and capacity of your ecosystem -- to establish a baseline when you begin your work and monitor through periodic check-ins -- can help you quantify and communicate benefits, identify focus areas as you continue to cultivate your ecosystem, and justify the time and effort of maintaining strong ecosystems.

What is a Healthy Civic Data Ecosystem?

What is a healthy civic data ecosystem? One way of thinking about this question is to consider the ways that a region with a healthy ecosystem might differ from a region without one - what value does having a robust ecosystem lend above and beyond having a single strong partner or several unconnected partners? Cultivating a strong and healthy civic data ecosystem results in a variety of benefits to your work and communities. For example, a community with a strong civic data ecosystem may be more resilient, more connected, more agile, and more inclusive - although all these potential benefits take intentional work to realize.

You can also begin thinking about the value-add of being a member of a connected ecosystem to your own organization’s work. When you have an innovative program idea, are you able to leverage resources (people, spaces, funding) from a diverse set of partners to make it happen? Are you able to quickly ramp up and respond to new challenges and opportunities within your community by sharing knowledge and connections within your ecosystem? Are you able to develop more effective solutions because you can easily obtain feedback from a wider range of user voices via your ecosystem partners? Are you able to address complex, systemic challenges by aligning work being done by different ecosystem partners?

Terminology: Ecosystems vs Networks

A civic data ecosystem is “network like” - it can be visualized using a network diagram (that’s essentially what you created if you followed our framework for mapping your ecosystem) and it shares many qualities of a network. However, it often intentionally lacks the formal structures (membership dues, member accountability structures, centralized communication tools, ongoing shared policy goals) that might be in place for a network, coalition, or collective impact initiative.

Many resources that Civic Switchboard recommends for thinking about ecosystem evaluation were designed to measure the health and capacity of networks. For this section of the guide, we will use the term “network” if the source material uses that term - but we believe these tools and concepts are just as relevant to ecosystems as they are to networks.

Just remember to think through the potential differences for an ecosystem before applying network-specific evaluation tools to your ecosystem. Every civic data community is different - adapt these tools to fit your own needs based on your own ecosystem dynamics and desired collaboration structures.

What to Measure

Even if you are familiar with outcome measurement and evaluation generally, deciding how to evaluate your civic data ecosystem - likely a loose mesh of partners that you may not have close communication with - can pose a challenge. What are you even supposed to measure?

A good starting point can be found in the report Evaluating Networks for Social Change: A Casebook, jointly published in 2014 by Network Impact and Center for Evaluation Innovation. The report offers three pillars of network evaluation: connectivity, health, and results.

• Network Connectivity: The people and organizations participating in the network, the structure, and what information or resources flow through which connections.

• Network Health: Whether the network is sufficiently resourced and whether the internal systems, processes, and infrastructures are sufficient to support the network.

• Network Results: Interim outcomes or intended impacts, such as adoption of best practices or policy changes. This pillar is potentially the most complex to evaluate, especially because what results you are looking for will depend on your particular community.

We encourage you to take a closer look at the case studies included in the Evaluating Networks for Social Change: A Casebook. Each one offers a specific example of what kinds of questions drove the measurements, what tools and methods were used, associated costs, and how the results were evaluated. You may find a case study that resonates with measures that are important to your local ecosystem.

How to Measure

There are a variety of methods to measuring ecosystem health and capacity. Some will be small-scale and free to use while others will require tools and additional staff.

Measuring the health and capacity of your ecosystem can begin with the process of interpreting your ecosystem map, which is most relevant to measuring network connectivity, but you can also think about measuring ecosystem capacity in ways that are not dependent on a visual map.

You might decide to create a custom survey to assess a specific quality of your ecosystem. You might decide to conduct interviews or focus groups with ecosystem members to collect qualitative data. You might research your community to identify trends and external strengths, weaknesses, opportunities and threats that may influence how your ecosystem functions.

Several organizations have created network capacity tools you can use as a starting point, such as the Evaluation Framework (beginning on page 47) from the Centre for Social Innovation pocket guide's Network Evaluation Cultivating Healthy Networks for Social Change or the Network Health Scorecard from Network Impact.

Who to Work With

Especially at first, you will likely want to think about small, bite-sized measurement techniques you can do with internal ecosystem partner staff. However, it is also worth considering external evaluation consultants if you can obtain funding or if you want to measure an aspect of your ecosystem that may be difficult for partners to talk about openly. A person or organization may not feel comfortable identifying ecosystem weaknesses if the evaluation is being done by a partner they rely on for funding or political support.

Regardless of whether you work with existing partners or an outside consultant, the Cultivating Healthy Networks for Social Change pocket guide referenced earlier identifies some of the skills a network evaluator - whether internal or external - should ideally possess or work to develop:

1. Being present in the network

2. Being an active listener

3. Recognizing patterns, synthesizing information

4. Seeing power dynamics and their impacts

5. Seeing simultaneously through multiple lenses

6. Identifying different forms of leadership

7. Giving people a voice to tell their stories

Consider these skills when you look within or outside your ecosystem for someone to assist with evaluation.

Projects Related to Ecosystem Mapping

It is worth taking a closer look at what kinds of results we might expect a strong civic data ecosystem to support or accelerate - either related to shared goals of the ecosystem or, more commonly, the ability of specific partner organizations within the ecosystem to leverage their connections to create stronger, more impactful programs. Below are two potential resources.

• The Knight Foundation report Assessing Civic Tech: Case Studies and Resources for Tracking Outcomes includes sample measurement techniques for five common civic tech objectives that are also applicable to many civic data ecosystems. These objectives are to (1) build place-based social capital, (2) increase civic engagement, (3) promote deliberative democracy, (4) support open governance, and (5) foster inclusion and diversity. These common objectives are good starting place if you are not yet sure what results you expect from your civic data ecosystem and want to think through how other communities have been thinking about similar work.

• The Civic Tech and Data Collaborative project looked directly at the structure and impact of civic tech and data ecosystems and identified the key ingredients of a civic tech and data ecosystem in their Civic Tech and Data Collaborative Toolkit. They emphasize the importance of, and offer resources related to, the cross-cutting areas of engaging low income residents, mobilizing for collective action, resourcing collaboratives and sustaining the gain. Mobilizing, resourcing, and sustaining ecosystems would likely fall under the evaluation pillar of Network Health. Engaging low income residents, in particular, is critical to ensure civic data ecosystems do not inadvertently contribute to inequities within communities and could be evaluated as a part of Network Connectivity (“who is participating”) as well as Network Results (“are the participating ecosystem partners reaching low income audiences?”). As part of their project, they also published a Guide to Civic Tech and Data Ecosystem Mapping.

Finally…

Remember that your community’s civic data ecosystem is unique, and part of measuring the health and capacity of your ecosystem is keeping the goals and values of your ecosystem in mind during the process. What you measure depends on what you are collectively trying to achieve. If thinking about the ecosystem as a whole feels overwhelming, you can also think about your own organization and measure whether you are receiving benefits from your space in the ecosystem.