Civic Switchboard Guide
  • Introduction
  • About
  • Context and Concepts
    • An incredibly condensed look at the development of open civic data in the United States
    • Ways of thinking about data: Open data, civic data
    • Defining a data intermediary
    • Where do libraries fit in?
    • Selected Resources
  • Engaging Partners
    • Building libraries into civic data partnerships
    • Finding partners in your ecosystem
    • Common barriers to getting started
    • Selected Resources
  • Understanding Your Ecosystem
    • Mapping your Ecosystem
    • Measuring Health and Capacity
    • Understanding Community Data Needs
    • Selected Resources
  • Library Roles
    • Connecting data users
    • Connecting data producers
    • Showing the importance of civic data
    • Developing civic data literacy
    • Advocating for ethical, responsible, and accessible civic data
    • Making civic data more usable
    • Providing expertise on data management
    • Creating civic data
    • Using civic data
    • Publishing civic data
    • Archiving civic data
    • Increasing the library's capacity to do all of the above
    • Selected Resources
  • Maintaining Momentum
    • Finding resources to support civic data work
    • Support networks and communities of practice
    • Institutionalizing
    • Selected Resources
  • Case Studies 2019
    • Alaska State Library, Juneau, AK
    • Charlotte Mecklenburg Public Library, Charlotte, NC
    • Fondren Library, Rice University, Houston, TX
    • Pioneer Library System, Ontario County, NY
    • Providence Public Library, Providence, RI
    • Queens Public Library, Queens, NY
    • Robert L. Bogomolny Library at the University of Baltimore, MD
    • Saint Paul Public Library, St. Paul, MN
    • Western New York Library Resources Council, Western NY
  • Case Studies 2020
    • The School of Library and Information Sciences Library at NCCU, Durham, NC
    • The University of Chicago Library’s Center for Digital Scholarship, Chicago, IL
    • Indianapolis Public Library, Indianapolis, IN
    • Spokane Public Library, Spokane, WA
    • St. Joseph County Public Library, South Bend, IN
  • Additional Resources
    • Civic Switchboard Workshop Materials
    • Open Data
    • Data Literacies
    • Partnerships and Community-Building
    • Library Roles in Civic Data Ecosystems
    • Guidelines for Creating Open Educational Resources
    • Archiving and Preservation of Civic and Government Data
    • Glossary
Powered by GitBook
On this page
  • Why Measure Ecosystem Health?
  • What is a Healthy Civic Data Ecosystem?
  • Terminology: Ecosystems vs Networks
  • What to Measure
  • How to Measure
  • Who to Work With
  • Projects Related to Ecosystem Mapping
  • Finally…
  1. Understanding Your Ecosystem

Measuring Health and Capacity

PreviousMapping your EcosystemNextUnderstanding Community Data Needs

Last updated 4 years ago

Why Measure Ecosystem Health?

Civic data ecosystems are complex and challenging to assess. But measuring the health and capacity of your ecosystem -- to establish a baseline when you begin your work and monitor through periodic check-ins -- can help you quantify and communicate benefits, identify focus areas as you continue to cultivate your ecosystem, and justify the time and effort of maintaining strong ecosystems.

What is a Healthy Civic Data Ecosystem?

What is a healthy civic data ecosystem? One way of thinking about this question is to consider the ways that a region with a healthy ecosystem might differ from a region without one - what value does having a robust ecosystem lend above and beyond having a single strong partner or several unconnected partners? Cultivating a strong and healthy civic data ecosystem results in a variety of benefits to your work and communities. For example, a community with a strong civic data ecosystem may be more resilient, more connected, more agile, and more inclusive - although all these potential benefits take intentional work to realize.

You can also begin thinking about the value-add of being a member of a connected ecosystem to your own organization’s work. When you have an innovative program idea, are you able to leverage resources (people, spaces, funding) from a diverse set of partners to make it happen? Are you able to quickly ramp up and respond to new challenges and opportunities within your community by sharing knowledge and connections within your ecosystem? Are you able to develop more effective solutions because you can easily obtain feedback from a wider range of user voices via your ecosystem partners? Are you able to address complex, systemic challenges by being done by different ecosystem partners?

Terminology: Ecosystems vs Networks

A civic data ecosystem is “network like” - it can be visualized using a network diagram (that’s essentially what you created if you followed our framework for ) and it shares many qualities of a network. However, it often intentionally lacks the formal structures (membership dues, member accountability structures, centralized communication tools, ongoing shared policy goals) that might be in place for a network, coalition, or collective impact initiative.

Many resources that Civic Switchboard recommends for thinking about ecosystem evaluation were designed to measure the health and capacity of networks. For this section of the guide, we will use the term “network” if the source material uses that term - but we believe these tools and concepts are just as relevant to ecosystems as they are to networks.

Just remember to think through the potential differences for an ecosystem before applying network-specific evaluation tools to your ecosystem. Every civic data community is different - adapt these tools to fit your own needs based on your own ecosystem dynamics and desired collaboration structures.

What to Measure

Even if you are familiar with outcome measurement and evaluation generally, deciding how to evaluate your civic data ecosystem - likely a loose mesh of partners that you may not have close communication with - can pose a challenge. What are you even supposed to measure?

A good starting point can be found in the report , jointly published in 2014 by Network Impact and Center for Evaluation Innovation. The report offers three pillars of network evaluation: connectivity, health, and results.

  • Network Connectivity: The people and organizations participating in the network, the structure, and what information or resources flow through which connections.

  • Network Health: Whether the network is sufficiently resourced and whether the internal systems, processes, and infrastructures are sufficient to support the network.

  • Network Results: Interim outcomes or intended impacts, such as adoption of best practices or policy changes. This pillar is potentially the most complex to evaluate, especially because what results you are looking for will depend on your particular community.

How to Measure

There are a variety of methods to measuring ecosystem health and capacity. Some will be small-scale and free to use while others will require tools and additional staff.

You might decide to create a custom survey to assess a specific quality of your ecosystem. You might decide to conduct interviews or focus groups with ecosystem members to collect qualitative data. You might research your community to identify trends and external strengths, weaknesses, opportunities and threats that may influence how your ecosystem functions.

Who to Work With

Especially at first, you will likely want to think about small, bite-sized measurement techniques you can do with internal ecosystem partner staff. However, it is also worth considering external evaluation consultants if you can obtain funding or if you want to measure an aspect of your ecosystem that may be difficult for partners to talk about openly. A person or organization may not feel comfortable identifying ecosystem weaknesses if the evaluation is being done by a partner they rely on for funding or political support.

  1. Being present in the network

  2. Being an active listener

  3. Recognizing patterns, synthesizing information

  4. Seeing power dynamics and their impacts

  5. Seeing simultaneously through multiple lenses

  6. Identifying different forms of leadership

  7. Giving people a voice to tell their stories

Consider these skills when you look within or outside your ecosystem for someone to assist with evaluation.

Projects Related to Ecosystem Mapping

It is worth taking a closer look at what kinds of results we might expect a strong civic data ecosystem to support or accelerate - either related to shared goals of the ecosystem or, more commonly, the ability of specific partner organizations within the ecosystem to leverage their connections to create stronger, more impactful programs. Below are two potential resources.

Finally…

Remember that your community’s civic data ecosystem is unique, and part of measuring the health and capacity of your ecosystem is keeping the goals and values of your ecosystem in mind during the process. What you measure depends on what you are collectively trying to achieve. If thinking about the ecosystem as a whole feels overwhelming, you can also think about your own organization and measure whether you are receiving benefits from your space in the ecosystem.

We encourage you to take a closer look at the case studies included in the . Each one offers a specific example of what kinds of questions drove the measurements, what tools and methods were used, associated costs, and how the results were evaluated. You may find a case study that resonates with measures that are important to your local ecosystem.

Measuring the health and capacity of your ecosystem can begin with the process of , which is most relevant to measuring network connectivity, but you can also think about measuring ecosystem capacity in ways that are not dependent on a visual map.

Several organizations have created network capacity tools you can use as a starting point, such as the from the Centre for Social Innovation pocket guide's Network Evaluation Cultivating Healthy Networks for Social Change or the from Network Impact.

Regardless of whether you work with existing partners or an outside consultant, the pocket guide referenced earlier identifies some of the skills a network evaluator - whether internal or external - should ideally possess or work to develop:

The Knight Foundation report includes sample measurement techniques for five common civic tech objectives that are also applicable to many civic data ecosystems. These objectives are to (1) build place-based social capital, (2) increase civic engagement, (3) promote deliberative democracy, (4) support open governance, and (5) foster inclusion and diversity. These common objectives are good starting place if you are not yet sure what results you expect from your civic data ecosystem and want to think through how other communities have been thinking about similar work.

The project looked directly at the structure and impact of civic tech and data ecosystems and identified the key ingredients of a civic tech and data ecosystem in their . They emphasize the importance of, and offer resources related to, the cross-cutting areas of , , and . Mobilizing, resourcing, and sustaining ecosystems would likely fall under the evaluation pillar of Network Health. Engaging low income residents, in particular, is critical to ensure civic data ecosystems do not inadvertently contribute to inequities within communities and could be evaluated as a part of Network Connectivity (“who is participating”) as well as Network Results (“are the participating ecosystem partners reaching low income audiences?”). As part of their project, they also published a Guide to .

aligning work
mapping your ecosystem
Evaluating Networks for Social Change: A Casebook
Evaluating Networks for Social Change: A Casebook
interpreting your ecosystem map
Evaluation Framework (beginning on page 47)
Network Health Scorecard
Cultivating Healthy Networks for Social Change
Assessing Civic Tech: Case Studies and Resources for Tracking Outcomes
Civic Tech and Data Collaborative
Civic Tech and Data Collaborative Toolkit
engaging low income residents
mobilizing for collective action
resourcing collaboratives
sustaining the gain
Civic Tech and Data Ecosystem Mapping