Data Base: Boston's Big Data Initiative

Photo by Lucas Zarebinski

“We are going to dominate this field,” Governor Patrick told a crowded room of hackers this past November in Kendall Square. He was talking about Big Data, the large-scale analysis of all the raw information that’s proliferating online. It’s what everybody is talking about these days: a field growing so fast that it has the potential to become a $50 billion industry in the next five years. Last May, sensing a huge opportunity, Patrick launched what he called his Big Data Initiative. And this year he’s orchestrating a push for Massachusetts to become a global leader in this emerging field, with Boston leading the way.

The numbers tell an impressive story. In 2012, the world’s computer users created an average of 23 billion gigabytes of data each day. Walmart alone handles more than a million customer transactions each hour, and has already built up a storehouse of data approximately167 times greater than all of the books in the Library of Congress. Facebook is home to 40 billion photos, and YouTube users upload 48 hours of video each minute. Google, for its part, handles tens of thousands of searches each second.

Perhaps most remarkable of all, some 90 percent of all the data in the world has been created in the past two years. Those who can best make sense of all of this are poised to grab a major piece of what may become the biggest growth industry of the decade—and with his initiative, Patrick hopes to make Massachusetts the “premier destination” for those looking to crunch big numbers.

The flagship of the initiative is something called Hack/Reduce, a “hackspace” launched this fall with $1 million of funding from a variety of area government, tech, venture-capital, and academic sources. The idea is simple: create a central location in the Boston area where anybody who wants access to the power of Big Data can turn. Hack/Reduce has set up 1,000 computers in a 2,500-square-foot space in Kendall Square, and is now making all of that processing power available for free. Anybody interested in making use of it—tech entrepreneurs, scientists, medical professionals, financial analysts, government officials, private individuals—can sign up online to use the facility. “We’re creating space where talent can come and collide,” Patrick said.

At any given time, some 200 hackers will be granted access to Hack/Reduce resources. They’ll be guided—and funded—by major players from Google, IBM, and the city’s VC leaders, who will work in collaboration with academics at schools like MIT and Harvard in the hopes of launching new startups. According to Chris Lynch, a cofounder of the project and partner at Atlas Venture, combining the tech talent of Boston-area students with the experience of regional entrepreneurs will help create “thousands of jobs, and hundreds of companies in 10 years,” and will help Boston establish itself as a world capital of Big Data.

Processing power is the key to Hack/Reduce’s efforts, and those 1,000 computers, outfitted with the latest data-visualization and analytic software, will supply it in spades. “You can’t just crank up your laptop and do Big Data on it,” explains Frederic Lalonde, another cofounder of Hack/Reduce and the CEO of Hopper, a local startup in the emerging field. Lalonde saw that while the city had some of the world’s brightest computing minds, there was no facility where they could actually start running numbers on an immense scale. Until now. “If an undergrad at MIT has an idea for a way to fix global warming,” he says, “he can literally sit down in a chair and run the computers using a million dollars’ worth of resources.”

Since opening, Hack/Reduce has been hosting parties and daylong hack-a-thons to get people excited about Big Data. Its founders have also provided resident hackers with their first challenge: helping Boston reduce traffic congestion by using the data pulled from thousands of sensors that track the density of cars on our streets.

The Hack/Reduce founders hope the hackers’ data mining will help detangle one of the city’s most persistent problems by resetting traffic-light systems and restructuring public transportation and route schedules. Today, Boston. Tomorrow, the data world.