"Big data is the new black"

Submitted on Wed, March 28, 2012

Submitted on Wed, March 28, 2012

Sanjana Hattotuwa, founder of GroundViews and a senior researcher at the Center for Policy Alternatives, has written a very thoughtful analysis of the potential social impact of Open and Big Data. What is "Big Data?" Basically, it's a huge dataset that's made broadly available for public consumption, analysis, and application -- on the assumption, generally, that more information shared by more people will lead to constructive outcomes.

Sanjana describes the current state of play -- while acknowledging that this phenomenon is shifting quickly and, often, unpredictably: "It is clear then that we are awash in data. Yet ironically, in this information glut, knowledge is an increasingly precious commodity. There are also growing concerns about the use and licensing of Big Data, with allegations that big corporations are locking in data generated by users, which in fact should be owned by them to do as they please, to their platforms, services and products. At stake is a global, interoperable, accessible data commons, nourished not just by datasets published by the UN, multinational corporations, civil society, NGOs and regional organisations, but by the aggregate information inflows from the billions of transactions made by citizens today using their mobiles and computers, augmented by remote sensing data as well as satellite imagery. From literally the grassroots to geo-stationary orbit, we are enmeshed in information pathways that are opening up to expose hidden data that governments and other actors are soon realising are too complex and large for themselves to analyse. Big Data, through gaming problems and encouraging citizens to make use of public data through APIs, is helping people help themselves, and through this process, help governments and the UN better understand the dynamics of a complex world."

The potential upside, he writes, is tremendous: "Big Data allows us to extrapolate from large datasets vital patterns and indicators that can influence, progressively or otherwise, the lives of individuals on the ground. At its most benign, it can help, for example, traffic and pedestrian flows in mega-cities so that congestion is eased, energy consumption is reduced and through infrastructure and public services that respond to socio-cultural, seasonal and geographical interaction patterns (e.g. waste disposal, municipal and local government offices, public spaces like parks, parking, entertainment centres, libraries and zoning) to significantly improve the quality of life of residents."

Already, the US Department of Defense has opened up certain military data to academics: "Given that the military and intelligence services have some of the comprehensive datasets on their respective countries, and in some cases, on other countries, we can expect subsets of these big datasets – which are themselves very large – to be parcelled out and openly published. Although commonly associated with big business, ‘data philanthropy’ by governments – the anonymisation and sharing of even sensitive data for public use – will grow, and aid peacebuilding by providing new insights into complex political emergencies, peace negotiations and peace processes, including post-war state-building and reconciliation."

"The impact however of using these datasets can be much more than improving the efficiency and effectiveness of wars. It can contribute to healing, understanding and the identification of socio-economic, ethnic, cultural, religious, partisan, political, tribal and other patterns that help early warning, mitigation, response and recovery. Aid agencies working in some of the most complex political emergencies today are already realising the potential of Open Data to transform their work on the ground."

As Sanjana notes: "Big Data is no panacea," and big challenges remain concerning access, rights, and availability. But its potential -- in providing early warning of conflict and natural disasters, in checking government abuses, in powering investigative journalism, is palpable. Read the full report here.

Blog Thumbnail: