Scott Klein, Jeff Larson and Jennifer LaFleur wrote the guides, which include a news app [...]]]> Coding the news now has a manifesto. ProPublica’s developers launched a series of news application guides, including a coding manifesto, this morning. The guides, which all live on GitHub, are intended to give insight into the programming ethos of the non-profit investigative journalism outfit. As the manifesto says, “We’re not making any general statements about anything beyond the environment we know: Doing journalism on deadline with code.”

Scott Klein, Jeff Larson and Jennifer LaFleur wrote the guides, which include a news app style guide, a data check-list and a design guide. These resources add to the ever-growing community of news application developers, many of whom are actively blogging about and sharing their working processes.

Read all the guides here.

Our investigation found that marijuana arrests actually rose after [...]]]> The other week, the New York World published a data reporting project with the Guardian examining the NYPD’s controversial Stop, Question and Frisk policy. Last year NYPD Commissioner Kelly issued an order to curtail low-level marijuana arrests following stop-and-frisks. WNYC had previously reported that the NYPD manufactured such arrests by ordering people to remove marijuana from their pockets and then charging them for the more serious crime of possesion in public view.

Our investigation found that marijuana arrests actually rose after Kelly’s order. But finding that story involved diving into the data.

Thanks to a recent lawsuit, the NYPD releases a database each year of every single “stop-and-frisk” that officers make. Unfortunately, the database is so big it can’t easily be opened in Excel and the data also requires some serious “cleaning” to be usable.

To address these issues, we analyzed the data using the open source statistics program R, which can handle data cleaning, interrogation, and visualization in one program. Because R lets you type in commands that apply across multiple files, it removes the need for switching among Excel windows. R also supports the SQL-like queries through the sqldf extension package, which makes more complex database systems so powerful.

Cleaning
Because we were interested in when certain types of stop-and-frisk incidents had taken place, we used R to split the day, month, and year of each incident’s date field into individual columns. This set the data up for the next step in our analysis, which was to count up how many marijuana arrests occurred each month.

Querying
Using SQL queries, we were able to group and count the data by month and crime type. We focused our searches on marijuana possession (which in the NYPD data was spelled “marihuana”).

We ran a number of queries to see month-to-month trends and also compared across years to see how 2011 compared to data to 2008. This gave us valuable context because stops actually dropped in November and December of 2011, but not as much as they did in those same months in prior years. If Kelly’s order impacted officer behavior, we should have seen relatively dramatic decreases during those months, but found only slight declines. This context was vital to our story, and explaining why the 2011 drop was not significant was a high priority for our final visualization.

We also ran queries comparing arrests to stops as well as isolating specific precincts. However, only a few of these queries yielded results that were worthy of inclusion in the final interactive.

Visualizing Part 1:
In order to find the trends mentioned above, though, we first had visualize our query results, which R can do, too. An extension package for R called ggplot2 will generate high-quality, customizable line graphs that could be used directly for print graphics. However, we wanted ours to be interactive, which required some additional work.

Visualizing Part 2:
SVG (Serialized Vector Graphic) is a type of graphic that is drawn dynamically on a computer screen, which means that it can be highlighted, clicked, rolled over, or animated in ways that .jpg, .gif and .png files can’t. The ggplot2 graphics can be converted to SVG, and then published to the web using a javascript library called called Raphaël. Although this requires some copying-and-pasting, the clean, dynamic graphics it produces are worth it.

Putting it all together
To better tell the story, we compiled four sets of charts that we incorporated into a so-called “stepper graphic.” Thanks to the newsapps team at ProPublica, there is a great open-source library (http://www.propublica.org/nerds/item/anatomy-of-a-stepper-graphic) for building these graphics. Turning my four charts into four different “slides” was as easy as creating a function for each of them and then copying in their Raphaël code. The stepper graphic library took care of numbering and transitions. We built the grid and axes with standard HTML and CSS, and made label fades using simple jQuery fadeIn() and fadeOut() methods.

Finally, once we confirmed we were running the story with the Guardian, we adjusted the styles to make sure it would mesh well with their design. So we made the months lowercase, the font Georgia, and the line fuschia – perhaps the most important part.

The analysis in progress

In early February, the Tow Center hosted a Journalism Data Camp with Knight News Challenge winner ScaperWiki, which provides tools and training to journalists working with difficult data. The goal of the camp was to bring together journalists and computer scientists to make data more accessible, analyze it, and create stories around the theme of “New York Accountability”. A group of journalism school students attended [...]]]>

The analysis in progress

In early February, the Tow Center hosted a Journalism Data Camp with Knight News Challenge winner ScaperWiki, which provides tools and training to journalists working with difficult data. The goal of the camp was to bring together journalists and computer scientists to make data more accessible, analyze it, and create stories around the theme of “New York Accountability”. A group of journalism school students attended the event to gain experience with data journalism. Marc Georges, one of the students who was part of the winning team, describes how his group’s project was developed.


Attendees at the event started out by forming groups and identifying a data source to mine for stories. Our group consisted of current journalism school students Curtis Skinner, Eddie Small, Isha Sonni, Trinna Leong, Keldy Ortiz, Salim Essaid, Sara Alvi and myself, as well as recent graduate and New York World fellow Michael Keller, and GSAS statistics student Brian Abelson.

Salim pitched the group a project focused on stop-and-frisks in Shia communities in New York City. A recent AP report showed that in 2006, the NYPD had recommended increased surveillance of Shia communities and had identified nine mosques for possible surveillance. Our team wanted to know if this recommendation had resulted in an increase in stop-and-frisks of Middle-Eastern New Yorkers and if anything in the data would tell us whether or not police actually targeted these communities. Here’s what we learned in trying to put together this story:

Data is Dirty
Just because data is available doesn’t mean you’ll be able to use it quickly and easily. Our first main challenge was accessing and cleaning data on stop-and-frisks in New York City. The NYPD makes this data available on their website but there’s a ton of it–400,000 cells of Excel values for every year.

Curtis, Eddie, Isha, Trinna, Keldy and Sarah researched and collected our data but one of the most basic issues we ran into was trying to determine how many stop and frisks affected people of Middle-Eastern descent. Although the NYPD tracks the race of those it stops, Middle-Eastern people are categorized as whites so it was not possible to isolate that ethnic group directly. As a workaround, we considered using census data to find predominately Middle-Eastern neighborhoods, but ran into the same issue. After reviewing the information we did have, we came up with the idea of using proximity to the mosques mentioned in the AP report as a marker for ethnicity. We thought it was fair to assume the closer a stop was to a mosque, the more likely the person being stopped was Middle-Eastern. We decided to look at a radius of 900ft, the average length of a New York City block.

Once we were able to isolate our data set, we realized that working with such large amounts of data wasn’t feasible without some type of automation. The coders at the event were really helpful in writing a script that scraped the data for the variables we needed. That let us isolate the key aspects of the stop-and frisks we wanted to use and move forward in mapping our data.

Mapping is Hard
One of our main goals in the project was comparing the incidences of stop-and-frisks near these 9 mosques with other areas in New York City. To do that, we needed to be able to map our cleaned data. Sounds simple enough, right?

Our initial map of stop-and-frisks for 2006, color-coded by race.

Creating our maps turned out to be one of the most difficult and time-consuming aspects of our weekend. Our main problem was that our location data for our stop-and-frisks was in a format which the NYPD uses called State Plane while the location data for our mosques was in longitude and latitude. Brian Abelson, a graduate student at Columbia pursuing a degree in Quantitative Methods in the Social Sciences saved the day by converting our data and then mapping it.

Brian used a mapping tool called ArcGis to solve the conversion problem so we could view the stop-and-frisks and mosques on the same map.  Brian, Michael and I then used a program called R, to further scrape the data and isolate points by specific variables, like race.  Brian then used ArcGis to isolate all the points within a 900ft radius of a mosque so we could see how the rates of stop-and-frisks changed over time. Mike Dewar, one of the coders at the event, was also very helpful in nailing down our approach to identifying stop-and-frisks near mosques.  Mike wrote an algorithm for us to measure the distance between any one point and a mosque.  We didn’t end up using Mike’s algorithm but talking the problem over with him and discussing various approaches was a great help in tackling the larger issue of working with such a large data set.

An initial map of stop-and-frisks for 2006. White dots show the number of stops, red dots the number of arrests.

Analyzing Data leads to More Analyzing Data
Once we were able to map our data, we could see how many stop-and-frisks occurred near these nine mosques in 2006. When we compared the first three months of the year, before the recommendation for surveillance was made, with the next nine months, we did see a small increase. However, to know if this is statistically relevant or markedly different from stop-and-frisks in other areas of New York, we have to do a lot more research and analysis.

Stop-and-frisks for two mosques prior to March 2006

Stop-and-frisks for two mosques after March 2006

Jeremy Baron of WikiMedia New York City and Thomas Levine of ScraperWiki, two coders from the event, worked with us after the event to help automate our workflow. Jeremy wrote a script which aggregated our data and put into a database while Thomas helped us throughout the process in fixing and checking our sql and javascript code.

Our next step has been designing the right control group against which to test our data, and we received great feedback after the event on the best way to do so. One surprising thing has been how rich our data set is. Further analysis may show that stop-and-frisks near these mosques wasn’t unusual, but as we continue working with the data, it has already given us ideas for more stories we can work on.

Published on the cover of the Chinese business magazine Caixin and by iWatch News, the story details how CIF’s network of more than 64 companies struck opaque deals with African [...]]]> In the fall of 2010, students from the Stabile Center for Investigative Journalism started what became a yearlong investigation into the multi-billion dollar deals of The China International Fund (CIF), a Hong Kong-based company with investments in African oil, diamonds and minerals.

Published on the cover of the Chinese business magazine Caixin and by iWatch News, the story details how CIF’s network of more than 64 companies struck opaque deals with African leaders such as Robert Mugabe in Zimbabwe and Eduardo dos Santos in Angola.  They followed CIF to Guinea, where the company is connected to the country’s former military regime.  

In addition to analyzing international oil sales, locating public documents in foreign countries, and finding sources overseas, two students went to Guinea to investigate CIF’s investments on the ground.

Below, the four reporters describe their methods.




Understanding the World of Oil Sales - Himanshu Ojha

In October, The New York World began a collaboration with WNYC and the Brian Lehrer Show to crowdsource ratings of New York City’s several hundred privately-owned public spaces. To facilitate and catalog this process, The New York World’s Michael Keller developed an interactive map of these spaces, which served as both a guide and a repository for audience contributions. Below, he discusses the process and technologies used to create this piece. Complete coverage of the project can be found here.

1. Where did the idea come from?

The Occupy Wall Street in late September was probably what brought the term “privately owned public space” to most New Yorkers’ attention. These spaces — like Zuccotti Park —were created from land deals made between developers and the city: if developers ceded some land for public use such as a plaza, they could build taller skyscrapers or get other incentives potentially worth millions of dollars. For my part, though, the sudden notoriety of Zucotti Park reminded me of a file I had seen a few months ago on the NYC Datamine entitled “Privately Owned Public Spaces” — or POPS.

I mentioned this find to Yolanne Almanzar, the New York World’s public space reporter, who was looking into other POPS and found that a 2008 Manhattan Community Board 6 survey found that many such spaces in midtown were actually closed to the public or in poor condition.

After locating and converting the Datamine Access file to a spreadsheet, I found it contained the 391 addresses of all of the POPS in the city. From there we had a question to answer: were landlords living up to their end of the deal in providing useable public space? Our editors Alyssa Katz and Amanda Hickman pitched the idea to editors at WNYC that week as a collaborative crowdsourcing project. They liked it and we went from there.

2. Who worked on it?

Working with WNYC’s map wizard John Keefe, I used Google Fusion Tables to map the addresses from the Datamine and then used JavaScript to add features like the progress bar, address finder, and GPS locator.

Once the map was developed, Yolanne and I worked with the WNYC producers Jody Avirgan and Paige Cowett to figure out what specific questions we wanted New York World readers and WNYC listeners to answer. Yolanne also went on the Brian Lehrer show to discuss the project twice — at the beginning and end of the month-long crowdsourcing phase.

Once we started getting responses, Yolanne and I used twitter and email to follow up on interesting sites.

3. How long did it take?

The initial map took a few days to conceptualize and code but the whole project is still ongoing. A crowdsourcing project is a differs from a traditional story because you spend a great deal of time planning and reviewing information before you have enough information to know what the story is.

We stopped accepting new reader reports at the beginning of November and are now pulling building records to see exactly what these sites were supposed to offer to the public and what they got in return. For me, it’s a great way to work because you’re in touch with readers, and you’re both invested in the project and anxious to see what comes of it.

4. What processes or technologies were used?

We used Google Fusion Tables to map the POPS, but first I had to format and verify the data.

I used Microsoft Excel’s “=CONCATENATE()” function to add “New York, NY” to the addresses so that Fusion Tables could properly geocode them — convert them to latitude and longitude so that they can be mapped.

Oddly, Zuccotti Park at 1 Liberty Plaza didn’t geocode correctly and was placed in the middle of the South Tower at the WTC memorial. ProPublica’s Al Shaw pointed out that sometimes Google doesn’t work nicely with addresses that start with a single number. By changing the “1” to a “One” it geocoded properly. To be safe, I manually spelled out all of the single number addresses. (This glitch also comes up using normal Google maps. Type in “1 Liberty Plaza New York, NY” and see where it drops the pin.)

We spot checked a few other locations, and manually geocoded their position where necessary. Out of 391 entries, about a dozen were incorrectly placed by Fusion Tables. Reader feedback helped us find a few, too.

One of the big challenges with this dataset was getting the infowindows — the little info bubbles that appear when you click — to appear in a standardized way since the data weren’t consistent: some spaces contained a building name and an address whereas others just showed an address.

To get around this, I made two new columns in my Excel spreadsheet called displayName and displayAddress. displayName would contain the site name if it had one, or the address if it didn’t. displayAddress would have an address if a site had a name, otherwise I left it empty. Although Because Fusion Tables won’t allow JavaScript in infowindows, I used CSS to style the site name and address as elements of an unordered list with each line styled to

float:left;clear:both;”

While the default infowindow layout puts break tags at the end of each line, my list layout meant that if displayAddress were blank it wouldn’t leave an awkward empty line in the text.

I also collapsed additional details of each site into a new column and used that for the infowindow so it would be consistent for all of the entries.

To create the progress bar for the map, I used the Fusion Tables SQL API to query our fusion table and count the number of columns that have been marked as visited.

Through the Google Maps API documentation, I found that you can relatively easily add GPS functionality to a site with javascript. I added this feature to help people use the map when they were out reporting on these spaces. And similar to the address finder, I see it as a way the story can be personalized for people who want to find spaces near where they live or work.

For the address locator, I borrowed some javascript code from John Keefe at WNYC.

Because mobile was a big concern for the map, I made a second stylesheet that showed only the address finder and the GPS button when it was viewed on a mobile device. Though creating the second stylesheet wasn’t difficult, making sure it was showing up correctly across devices required a bit of a hack.

For most devices, I could detect the mobile device using:

media='only screen and (max-device-width:480px)'

For iPad or horizontal views on those devices I added:

media='only screen and (max-device-width:768px) and (max-device-width:1024px)'

To make sure those dogged infowindow didn’t show up super small on mobile devices, Keefe sent along this piece of code to go in theof my page to scale the viewport appropriately:

<meta name="viewport" content="width = device-width,initial-scale=1.0,minimum-scale=1.0,maximum-scale=1.0" />  
<meta name="apple-mobile-web-app-capable" content="yes" />

That did the trick.

5. What was learned from it?

We used Google Forms to get the reports from our readers and listeners, which was a good solution in that WNYC was familiar with it and it’s easily embeddable. But we spent a lot of time curating reader responses so it would have been much nicer to input our responses into a database whose results we could query and sort. The Google Form inputs our results into a spreadsheet that we had to then manually arrange and create lists from, which was a time-consuming process.

It would also have been nice to have a more robust wrapper for our fusion table so that we could easily pull in comments or media from our reader-submitted database and display them alongside the map. This would have meant designing a flexible layout to handle multiple content types as well as a backend database to categorize this information, but this was far beyond what our timeframe allowed.

A recurring lesson on this and other projects has been that designing user-interfaces is much like the scene in Alice in Wonderland where Alice is racing the Red Queen. Alice runs as fast as she can but she doesn’t move from where she stands. The Red Queen says: “Now, here, you see, it takes all the running you can do to keep in the same place.” When starting with messy data, it takes a lot of thinking just to get an interface that works without giving the reader a headache.