Northeastern Wants to Unlock the Secrets of Boston’s Past with Big Data

Photos clockwise from top left: Mayor Raymond Flynn stands with Carmen Pola and two men (1988), Two men carry a banner reading “Boston black and white men together” while marching in a parade (1983), A band performs with young girl in a pink dress during Chinese Progressive Association’s Tenth Anniversary Celebration(1987), Citizens’ Urban Renewal Action Committee meeting; chairman John Jones standing at left; unidentified woman speaking at right (1966) | Courtesy of the Northeastern University Archives and Special Collections

Boston has a long and complicated past, and its many archives and databases each tell little pieces of that story—a photo album here, a map there, a batch of letters and some old news articles somewhere else. But what do we do with all that stuff? What kinds of secrets are buried in all that material scattered around the city?

In the era of Big Data (a term used to describe analysis of giant datasets with powerful computers), a group of researchers at Northeastern’s Snell Library say it’s time to find out. This spring, thanks to a grant from the Andrew W. Mellon Foundation, they’re launching a new Boston Research Center, where they plan to compile and study the city’s history using modern tools, hoping to glean new revelations about the place we live. Big Data, if you’re unfamiliar, is “There tends to be a divide between the archival materials and the data materials in cities like Boston,” says Dan Cohen, Northeastern’s dean of libraries and head of information collaboration. “What Mellon was really interested in here was, can we bring together these modern data sets with historical materials, which can be digitized into one seamless whole?”

Cohen thinks they can. He has set the ambitious goal of assembling all of Boston’s history—as it exists in both paper and digital records—in one database, in collaboration with the city’s universities, libraries, museums, nonprofits, and other storehouses of info. Over the next few months, a group of students, faculty, and other outside experts will gather on the library’s fourth floor to collect, digitize, and assign “metadata” to all that information, to make it easier to find in search engines and use in studies about everything from how climate change has impacted the coast to how waves of immigration shaped communities.

Northeastern is already a storehouse of historic records. In its archives, you can find the entire print and digital run of the Boston Phoenix, as well as the Gay Community News, and the East Boston Community News—along with an extensive history of the Big Dig and activism in Roxbury, which has been used for Boston Public Schools curricula.

Researchers will also draw on municipal data, including via the city-run site Analyze Boston, which tracks everything from crime stats to energy usage at City Hall. “Boston is a smart city, so the city itself produces a lot of data on everything from where potholes are to where trees are planted every day, to new transportation modes,” says Cohen. “There’s a lot of contemporary data that we can synthesize together with the historical data, and that allows a lot of researchers—at Northeastern and also, we expect, from around the world—to come to the center and use that material in new ways to tell us new things about Boston.”

Eventually, he says, the Boston Research Center data will be available for free on Northeastern’s website, in the mold of the nonprofit Digital Public Library of America, which Cohen helped create in the early 2010s. A prototype of the searchable database is expected to be unveiled in August.

“We’re thinking about ways in which an understanding of the past can help inform the research of today, especially when it comes to Boston’s communities,” says Giordana Mecagni, Northeastern’s head of Archives and Special Collections. “We provide this very deep community understanding in the way that Boston’s neighborhoods are built, the ways that they’ve changed.”

One new project the Boston Research Center team is excited about involves placing thousands of historic pictures of Boston on an interactive map, as a way of tracking growth and decay in parts of the city over several decades. They say they plan to release a handful of new data-driven projects by the end of the year, with help from the NULab for Texts, Maps, and Networks and the Boston Area Research Initiative, an inter-university partnership that brought us the Boston Research Map.

“I envision a whole bunch of collaboration,” Mecagni says, and “being able to tell the story of Boston in a much more robust and diverse way.”