Reports & Briefs

Platforms & Publishers

Reports & Briefs

Tow/Knight Projects

NewsLynx: A Tool for Newsroom Impact Measurement

Project Leader:

 

Executive Summary

 

With the rise of nonprofit, foundation-funded newsrooms, the field of Measurement and Evaluation (M&E), which emerged in the international development community, has taken a strong foothold in journalism. As nonprofit newsrooms apply for grants and appeal to donors for funding, they often need to explain in formal reports “how well” their stories performed—not just in terms of impressive traffic but in qualitative evaluations of the impact their reporting had on the world: Did it change a law? Did it move the needle in the conversation? Did it meet the expectations—however defined—the organization had for it? Based on survey research and interviews with newsrooms regarding current impact measurement practices, the researchers designed and built a new analytics platform called NewsLynx to improve upon existing methods of displaying quantitative metrics and to add qualitative information that was previously nonexistent in such tools. Many newsrooms found current analytics tools insufficient for fully capturing their output’s performance. They had trouble seeing comparisons between audience reactions to stories or the effects of their social media and promotional efforts. While they often had multiple data sources—Google Analytics, Omniture, etc.—putting these numbers into context was still difficult.

The NewsLynx Project Implements Three Key Ideas

  • NewsLynx seeks to augment metrics with context. It shows how an article performs in comparison to the average of all a publication’s articles and allows comparisons within subsets—all immigration articles, for example, or within any user-defined category.
  • NewsLynx also provides efficient tools for tracking, categorizing, and assessing indicators of impact aside from audience reach. Such impact indicators might be legislative reform or community action. This has previously proved extremely difficult and time-consuming. NewsLynx’s “Approval River” functionality aims to reduce the effort associated with managing traditional clip searches and social media searches, which newsrooms use to monitor impact. Crucially, it allows users to apply consistent (and therefore comparable) metadata to impact indicators.
  • The NewsLynx developers propose an impact framework that allows for the fact that real-world impact measures are often messy and hard to categorize. NewsLynx implements a framework that offers newsrooms enough structure to categorize “impactful events” across similar boundaries, while also providing enough freedom for them to create their own impact definitions to match particular goals. Importantly, the researchers believe that successful, long-term impact measurement can only result from identifying such organizational goals.

Key Observations and Recommendations

  • Effective impact measurement must be tied to an organization’s goals. No amount of technology can help an organization measure what it hasn’t defined as important. Should a newsroom’s reporting seek to change the narrative around an issue? Does it want to reach certain stakeholders or affect lasting reform? Only after an organization has understood what it wants to achieve can quantitative and qualitative tools assess how close the organization is to that goal.
  • Both quantitative and qualitative metrics have a place in impact measurement. While quantitative metrics are often vilified as leading journalism astray from its true purpose, the researchers found they do help tell the story of a newsroom’s performance. Although this project began with an interest in giving more visibility to qualitative measurements, its founders repeatedly heard reports from newsrooms that quantitative measurements play an important role for organizations wanting to tell a long-term story of audience growth.
  • Newsrooms should better tag their articles. Newsrooms that want to properly understand their own performance over time should put more care into tagging and cataloging their stories. These practices can give an organization a better understanding of its own operations and how much space it devotes to each subject. Tags also offer staff the ability to perform myriad analyses comparing stories and packages. Without differentiating and labeling content, it is difficult to understand patterns in traffic or impact.
  • Newsrooms have metrics, but they also still have many questions—particularly about audience. As one newsroom put it, “Google Analytics feels both too complicated and not powerful enough for the questions we want to answer about readers.” Many existing metrics aren’t designed to help analyze metrics from readers’ perspectives—as in, what did they think about the story? Did they leave after the fifth graf because they understood the newsy part of it and didn’t need anymore, or was the site design wrong or the prose too dense? Nor do common tools provide enough insight into that relationship between the news organization and its audience.
  • Custom analytics solutions have recently become more feasible. With the continuing maturation of open source analytics pipelines, it is now possible for news organizations to own their entire analytics stack and not rely on third-party vendors for the data-collection portion of their metrics. In other words, the next few years could see newsrooms access much more diverse offerings, providing faster analysis and greater detail more relevant to journalism. That being said, these pipelines are largely for data collection, so most newsrooms will need to design and implement their own custom interfaces to interpret this data for the average reporter and editor.

 

Read the full report here or download a PDF

June 4, 2015