How Big data & Analytics can help Government agencies run better?

Jobs Admin 18 Apr, 2015
8 min read

Over the past decade or so, analytics has undergone a rapid transformation. During its initial stages, analytics was used more as a reactionary measure i.e., to observe the business trends of the past. So, there was a significant time lapse between the pain points and the corresponding course correction. But over the years, analytics bridged this time gap and is currently helping businesses take real-time decisions. Not just that, it is opening up avenues to predict future outcomes so that decision making is more proactive. This paved way for analytics being the backbone of “Strategy” and thus we saw the role of Chief Data/Analytics Officer joining the C-suite. Now with business organizations across the globe embracing Analytics like they did never before, the important question to be asked is “Do citizens have the right to expect their governments to be run on par with the best business organizations?”. The answer to this question is a resounding Yes and this was precisely the vision of Mr. Bloomberg, Mayor of New York city(more on this later in the article), who had setup an Analytics division which helped in making New York a model city in terms of governance.shiva baba

Government agencies have huge volumes of data and more often than not there would be little/no dialogue happening between departments.This essentially makes them work in silos which curbs the possibility of decision making using collective information. The insights that can be derived out of having a common data warehouse and tapping into the collective knowledge of various departments is left to one’s imagination. It would generate outcomes more than sum of its parts. Now let’s look at some of the Why, What and How of a typical city government’s problems and then we will look at how Analytics, with its arsenal of tools, helps tackle the toughest of these problems.

  • How can a government reduce/eliminate financial leakages from social welfare schemes?
  • What are the most crime-prone areas in a city?
  • Why can’t we eliminate Fraud and waste in government agencies?
  • What are the best practices that governments can adapt from private banking/financial sector companies in dealing with Fraud?
  • How can we improve emergency services like 108(or 911 in US) by using predictive analytics?
  • How can we deal with disaster response using predictive analytics?
  • How can we better allocate resources to various departments be it monetary, human or other kind of resources?
  • How can amendments to laws and policies be data driven?

These are just a few ways that Governments, at least in the developed countries, can utilize the potential of analytics and big data to help serve people better. As is the case with business organizations, it would take some time for governments of developing countries to catch up with their developed counterparts. Now let’s look at some of the case studies and real world examples of how Governments addressed some of the issues mentioned above and how they tapped into an ocean of data to discover pearls of insights.


Formally instituted by Michael Bloomberg, Mayor of NYC, at the beginning of 2013, the Mayor’s Office of Data Analytics (MODA) had a clear vision of using data analytics for a more effective and transparent government. MODA represents a paradigm shift in how government works – one that is guided primarily by data and the expertise of the people behind it. MODA created a datawarehouse by the name DataBridge, a common platform to facilitate inter-agency data handshake. The following are some of the major achievements of MODA

  1. 911: Emergency response is a very crucial service offered by city governments and in some cases a split second is all that stands between life and death. With an average of 25,000 calls to the emergency 911, it is imperative for the administration to provide the best-possible service in the shortest possible amount of time. While it is difficult to reduce the time taken by making police cars race even faster through NYC streets, the city is taking action to reduce the time to respond, including shortening the initial operator script, improving the relay between agencies, and more effectively routing services to the location. The information provided by the 911 end-to-end analysis lead to a better understanding of emergency services, and decision making that leads to faster responses at the most critical times for New Yorkers.When a New Yorker dials 911, they receive one, integrated experience. From the initial routing of the call through Verizon to NYPD telephone operator fielding the call until emergency services are on the scene, the caller is served by a staff of trained professionals who are collecting the vital information quickly and efficiently.
  2. FDNY: Based on decades of historical data on fire breakouts in various parts of the city, MODA has developed a statistical model to predict future fire risks. During this exercise, they took key inputs from veteran fire-fighters to refine and improve the model. Focused group discussions were conducted and risk inputs were appropriately weighed in to build the fire risk model. Thus, MODA was able to identifyhigh-risk zones in Harlem, Downtown Manhattan, and the Rockaways.
  3. Hurricane Sandy: The disruption that Sandy caused to NYC was unprecedented in terms of magnitude. It displaced one-eighth of the city’s population and left many more without basic amenities. During these testing times, MODA played a paramount role in the recovery process. Working closely with Deputy Mayor for Operations and his staff, and the leadership of the Office of Emergency Management (OEM), MODA integrated from City agencies, National Guard surveys of affected residents, and daily outages from Consolidated Edison (ConEd), the Long Island Power Authority (LIPA), and National Grid. The day-to-day progress was delivered to City Hall through the “Recovery Report,” a two-page summary of recovery efforts that was used to allocate disaster response resources, and ensure that vulnerable populations received needed attention.
  4. Legislation: Generally, legislative deliberations do not take into complete consideration the potential impact of the changes. This would jeopardize the idea/intent behind the legislation and sometimes they would also run the risk of being counter-productive. This is precisely where MODA comes into the picture. It would give the council members, a first-hand experience of the possible outcomes of a particular policy change. MODA developed a statistical model to factor in the potential impact and thus helped predict the outcome at a near-granular level.
    • To put things into perspective, let us take an example ofa policy change in current budget by the Govt. of India.The excise duty on Gutka (and chewing tobacco) was increased from 60 to 70%.Now let us see how Predictive Modelling helps answer some pertinent questions. How would this policy impact sales of Gutka companies? Is this a hike that the end-consumer can bear, or will it lead to lower sales? Now, if the intent of the government is to bring down the sales (in lieu of a healthy society), is the hike steep enough to discourage consumers from buying?On the flipside, if the policy does indeed lead to a decline in sales(and if it is sharp) are we providing other avenues for Gutka companies to save money and stay afloat? Is there another policy in which we can bake in this impact and thus reduce the impact on these companies? These and a plethora of subsequent questions can be satisfactorily (if not comprehensively) answered using the power of analytics.



shiva baba.1

Using Big Data and Analytics to reduce crime on the streets is also getting traction these days. Predictive Policing is one such endeavor. The PredPol tool was developed over the course of six years by a team of PhD mathematicians and social scientists at UCLA, Santa Clara University, and UC Irvine in close collaboration with crime analysts and line level officers at the Los Angeles and Santa Cruz Police Departments. It comes with a simple mission of having police officers at the right place, at the right time to prevent crime. To accomplish its mission, PredPol processes crime data in order to:

  1. Assign probabilities of future crime events to regions of space and time
  2. Present estimated crime risk in a useable framework to law enforcement decision makers
  3. Lead to more efficient & more accurate resource deployment by local law enforcement agencies

Los Angeles Police Department (LAPD) scored big success by deploying PredPol. Crimes in the Los Angeles foothill division came down by 13% in the 4 months following the rollout of PredPol compared to an increase of 0.4% in the rest of the city where the rollout had not happened. Over this time period Foothill division was a leader in crime rate reduction among LAPD’s divisions. Similar reductions have been seen in other cities that implemented the tool. LAPD said that they enjoyed a “day without crime” on Feb 13 ’14 in the Foothill division, which stretches 50-square miles, home to more than 250,000 people.

Tackling Fraud and Waste:

  1. In the United Kingdom, the National Fraud Authority estimates that £21 billion ($33B USD) is lost to fraud in the public sector each year.
  2. In the United States, improper paymentsby government agencies reached $125 billion in FY10.
  3. In the United Kingdom the estimated Tax Gaps for2011-2012 areapproximately £35 billion.

shiva baba.2

Appalling as they may seem, these numbers merely point us to the tip of the iceberg.It is high time Governments take appropriate measures to keep a check on fraudsters from siphoning of funds. Governments could learn a great deal from the Banking and Financial industry in their efforts to curb fraud using advanced analytics. Research by Accenture indicates that the most effective solution for annihilating government fraud at all levels is achieved when leaders drive an enterprise approachthroughout their agencies, integrating andapplying analytics insights throughoutend-to-end processes.

Examples of corrective measures:

  1. Washington State Department ofLabor and Industries: This department initiated an effortto identify all gaps in its detection andits management of possible workers compensation fraud. The groupautomated the fraud analysis processand improved business analytics tomove the organization from detectingfraud after the fact (and workingtoward recouping payments), to beinga more proactive organization thatcan quickly detect fraud, stop it andprevent it from happening in thefuture.
  2. US Metropolitan Finance Agency: The agency conducted a 14-week pilot project to define a standard for identifying and quantifying non-compliance, and employed analytics to identify likely violators. An initial assessment of 100 properties (of 26,000) from the prioritized candidate list yielded an additional 21 percent noncompliance identification. As a result of identifying non-compliance, the agency stands to collect an estimated $26 to $50 million in additional revenue.

shiva baba.3

MODA, PredPol, Washington State Department’s initiatives are steps in the right direction. With companies like IBM and Accenture envisioning smarter cities and governments, it is just a matter of time for policy and decision making in government agencies to be backed up by statistics and driven by data. Leaders have a big role to play in this transition, to be able to see the big picture and deploy holistic analytic solutions for dealing with the gamut of problems their agencies are facing. I would like to know your views/comments/ideas about the topics mentioned in the article. More precisely, do you see how analytics can empower our municipalities and corporations to serve us better? Do you see how the Govt. of India can lap up data driven decision making? With a PM who stresses on Good Governance and efficient bureaucracy, how can analytics showcase the answers for cutting the costs, increasing ROI?

This article was submitted by Mr. Prashanth Pattamatta as part of his application for Analytics Vidhya Apprentice programme. Prashanth completed his PGDM (MBA) from IMT Ghaziabad, a top rated private B-school in India and has more than 4 years of IT / Analytics experience across several IT giants. He is extremely passionate about learning Analytics and hence wants to be part of Analytics Vidhya Apprentice programme.

If you like what you just read & want to continue your analytics learning, subscribe to our emailsfollow us on twitter or like our facebook page.

Jobs Admin 18 Apr, 2015

Frequently Asked Questions

Lorem ipsum dolor sit amet, consectetur adipiscing elit,

Responses From Readers


Karthikeyan 14 Aug, 2014

Interesting Case studies. Superb stuff. Somewhere I read that one entity who can get big benefit out of implementing analytics are the government agencies. After reading this article, I feel its viable. Thank you for bringing in Prasanth.

Deepankar 15 Aug, 2014

Great article. It is imperative that govt. adopts analytics in managing scarce resources such as water as well as waste management.

Varshul 16 Dec, 2014

I wonder what great effects would data driven policies would have in India given the huge amount of data available wrt India's 1.2 billion population.

Take a note

10 Nov 23 • 08:00pm

View all notes