Last year, Buzzfeed published lists of the most tweeted, Facebooked, and searched-for stories across its now-defunct partner network, which included sites like The New York Times, the Huffington Post, the Atlantic. The lists provided plentiful (if unsurprising) fodder for anyone who is worried about the future of participatory democracy in an age when we get more and more of our news from Facebook: the top 10 most-shared stories on the social networking site included “27 Shocking And Unexpected Facts You Learn In Your Twenties,” “What Happens If You Text Your Parents Pretending To Be A Drug Dealer?” and the Times’s wildly successful dialect quiz, “How Y’all, Youse, and You Guys Talk.” But the lists were also striking for a different reason: they underscored the fact that the online habits of news readers are being tracked constantly. While Buzzfeed’s lists focused on clicks and shares, many analytics dashboards go far beyond these measures to include time spent reading, scroll depth (how far one scrolls down a page before clicking on something else), recirculation (how many pages one consecutively visits on a particular site), and visit frequency over time.
The sheer ubiquity of this data makes the longstanding professional argument about whether metrics have any place in a newsroom seem almost quaint. The question isn’t whether metrics should be in newsrooms to begin with – they already are, and they’re not going anywhere. The question being debated nowadays is: now that all this data is available, what are the best audience measures of audience behavior and what should journalists do with them?
Important questions, to be sure. But the premise of my Tow project is that before attempting to answer these normative questions, it is helpful to investigate some empirical ones. How do analytics companies produce metrics? What values, assumptions, and ideas about journalism are embedded within these numbers? How do analytics tools interact with established work routines and organizational dynamics in different types of news organizations? Answering these questions requires that we set aside the idea that data can “speak for itself,” and instead examine the creation and use of news metrics as social processes – that is, as active decisions that are carried out by groups of human beings. Over the past year and a half, I’ve been studying metrics this way by observing day-to-day operations and conducting interviews at Chartbeat, Gawker Media, and the New York Times.
Here are some things I’ve discovered:
- News metrics are hard to interpret because journalism has multiple aims. The meaning of metrics is relatively straightforward in fields with a singular, easily measurable goal: the Oakland A’s in Michael Lewis’s Moneyball were trying to maximize wins, and everyone agreed on what constituted a win. Journalism is (and has always been) considerably more complicated, because there are many ways a news story can be said to “win”: it can break news, it can prompt a legal change, it can shatter a damaging myth, it can be read by a huge number of people, and so on. This complexity means that news metrics can come to resemble Rorschach tests; as a reporter at the Times put it, “you rarely have an apples to apples comparison….There are so many other things kind of confounding it. It’s very easy for everybody to read their own agenda into the numbers.” At Chartbeat, employees wrestle with the question of how much the analytics dashboard should interpret metrics and recommend actions. Provide too little interpretation and the clients may see the dashboard as insufficiently useful or “actionable”; provide too much and editors may view the tool as a usurper and resent it. In sum, the way in which metrics are interpreted (and by whom) tells us much about the power dynamics and internal politics of the media field.
- Organizational culture matters – a lot – when it comes to how metrics are used. In other words, the mere existence of data in newsrooms does not tell us much about what journalists do with it. Case in point: the Times and Gawker both use similar Chartbeat dashboards (among other analytics tools) but the organizations use this data in completely different ways. At Gawker, metrics are a prominent presence in the newsroom: large screens displaying real-time traffic rankings of stories and individuals famously loom over writers’ heads in the company’s offices; these rankings are also publicly available online and help determine bonuses. At the Times, access to analytics tools is mostly confined to editors (though this may soon be changing in the wake of the Innovation Report), and reporters’ traffic is not a factor in performance reviews. These different approaches for dealing with the same metrics can be traced back to each organization’s particular history, structure, and culture – yet these factors often get overlooked in conversations about how metrics are affecting news.
- Metrics serve social and emotional functions just as much as rational ones. We tend to think of the analytics dashboard as a dispassionate tool whose purpose is to provide objective data about reader behavior. For this reason, metrics have gained a reputation as ego-busters, as journalists can discover that their readership is smaller and far less attentive than they imagined. While some find this information to be helpful, if humbling, others can sour on a tool if it only tells them things they don’t want to hear. To avoid this, Chartbeat builds into the ostensibly neutral dashboard opportunities for optimism and celebration, sometimes even at the expense of the data’s utility. For instance, the dashboard’s traffic dial is designed to “break” when clients’ traffic is surging past a pre-set cap. As one Chartbeat employee put it to me, when the dial maxes out “the product is broken, but in a fun way… if you didn’t have that excitement, [the product] wouldn’t work.”
These findings, along with others I’ll introduce in the full report (to be launched in late March) illustrate a basic – though easily overlooked – truth: to know the implications of metrics for journalism, we must first understand how this data is created, interpreted, and used by real people in actual organizations.