Tdworld 2441 Bigdata

Is BIG DATA a Big Bust?

Nov. 3, 2014
Utilities have invested billions in smart meters and devices that are generating massive quantities of data. How can the industry insure a return from this data investment?

BIG DATA is attracting big interest. A recent Forbes article said that “87% of enterprises believe BIG DATA analytics will redefine the competitive landscape of their industries within the next three years. 89% believe that companies that do not adopt a BIG DATA analytics strategy in the next year risk losing market share and momentum.”

With the growing deployment of smart meters and other smart devices, utilities are certainly generating their share of BIG DATA.

Given the significance of BIG DATA, we posed this question to our IdeaXchange Xperts: “Is BIG DATA a big bust?” Here is what our Xperts have to say:


It will be if we don't find ways for our utilities to manage, protect and extract value from BIG DATA. From my perspective, we need a holistic, standards-based approach to grid modernization. We need an open architecture that will offer us the best means of data management, protection and value creation. Oh, and did I mention that open standards will ensure that related investments maintain their value?


A huge difference between BIG DATA in the utility industry and that in the retail industry is the amount of data errors. From weather data (900 errors and holes in 8,760 hours of data on my last project from the U.S. Weather Service) to metering data, there are massive holes in the data, data that is wrong, missing and data that are estimated. Then there is the fact that the topology that the data is collected from may or may not be right. For instance, in one case, auditing 1,000 meter-to-transformer connections manually turned up 264 situations where the mappings were wrong. The meter was at the correct account, and the billing was right, but the transformer was wrong.

Fixing the underlying data, grid topologies, grid models, electrical characteristics and other foundation pieces will not be cheap. Until they are fixed, analytics on asset management, impacts of renewables, load forecasting and other items will be only as good as the inputs. Using BIG DATA for customer analytics, non-revenue losses and other single point analytics are the low hanging fruit. Using it for system-wide analytics is also low-hanging fruit because the errors tend to average out in many cases. However, using them to determine the load at a step-down transformer on lateral or to site a voltage regulator will required much more work — most of it in getting the data and the topology right. A non-trivial exercise that is not getting the discussion it needs.

To make BIG DATA really valuable means doing a lot of work and planning to get to answers that are based on good data. Transactive energy, integration and operation of renewables, demand response and other advanced topics are going to depend on getting these foundations right.


One of the significant concerns for utilities in utilizing BIG DATA containing customer information is security. As an example, there are currently a number of public utilities in the U.S. that must share customer data under sunshine laws. Because of this, there have been a number of incidents where customers have been harmed. In one case, a customer was actually murdered as a result of someone being able to access their information.

Admittedly this is an extreme example, but it reflects the importance of guarding information especially with the stakes being elevated in the world of BIG DATA.


I agree with John McDonald that BIG DATA has a lot of potential. It also has potential to be a significant cost and operational burden.

Utilities already have a lot of data. Many are awash in data, and it shows up as a cost burden and a liability (if you have the data and don’t see what it is telling you then you are at risk for claims of negligence).

BIG DATA by itself is not the answer. The value lies in what we do with it. Earlier in my career, I was involved in some of the best industry-wide benchmarking. Through that work I learned a number of things, and one of the most important things I learned was this: Data on its own is worthless. 

Having processes, tools and systems to convert the data into meaningful information helps. Where it becomes truly valuable is when you can combine the meaningful information with knowledge and experience to create insight. It is the insight that enables us to make different and better decisions.

I am concerned that the notion of BIG DATA is being pushed by data people and that we have not thought enough about how to leverage the data to create insight. This requires investments not only in tools but in people, particularly those that can see the implications of what the information is telling them.

I know that this may be a contrary view and it is based on my experience working with a large number of utilities around the globe.

I hope that this helps push the conversation forward.


I agree with Stewart Ramsay and John McDonald: BIG DATA by itself has no value. It only takes up space, and everyone is just afraid to do anything with it. The key to this is in the insights that it can deliver and what the utility (or a different entity) does with these insights. I believe the following steps need to be taken by the entity:

1. Figure out what insights (outcomes) are important to you — no need to dredge the entire ocean here

2. Use these outcomes to decide how to store the data in the most optimum mechanism

3.  As and when the business case is justified, implement the specific analytics to drive the specific set of outcomes. The outcomes are there in the utility industry in droves:  

  • Customer insights
  • Operational insights
  • Asset management/maintenance insights
  • Corporate insights.

Lastly, this is not a technology-only solution. This includes people and processes, because if people and processes do not change, then real transformation and real delivery of business benefits will not take place.


I agree that the notion of BIG DATA may be influenced by those that benefit from calling it out. To add to the contrarian view:

BIG DATA is a solvable problem, and perhaps this is small for utilities compared to BIG DATA driving financial markets. What makes this unsolvable is the boundaries we refuse to push out. Today, PMU data is relegated to parallel systems because state-of-the-art EMS software cannot handle it. A next-generation EMS system is required to begin to deal with BIG DATA for direct use by state estimators and for direct use in closed-loop systems. Herein lies the problem: Closed-loop systems on the grid are typically relegated to local devices with local input. Imagine if the installed power electronics at generating plants can be tweaked to avoid oscillatory system behavior and dampen, or HVDC systems able to deal with loop flows in real time, etc.

At the distribution level, we await the day when aggregators can make smart use of customer data while creating grid value for roof-top solar, switching devices such as water heaters without customer impact, etc. We used to call this smart house technology in the 1990s.

One point raised earlier regarding customer information for municipal utilities is that Ohio passed a law protecting customer energy information that takes the data out of the realm of public records. 

OK, perhaps I moved off topic a bit, as Stewart can attest I sometimes do (and he does too), but I think BIG DATA is an enormous opportunity to solve TECHNICALLY the duck curve issue, inter-area oscillations, state-estimation blind spots, loop flow issues and with MARKET AGGREGATOR SMARTS make roof-top and pole-top solar play more economically without subsidy for consumers.

Obviously, I can go on, but I will stop here.


I view the concept of BIG DATA in much the same light as I viewed the concepts of Deregulation and Smart Grid. On the surface and amid the hype, these ideas may initially appear to be the next greatest thing, but in reality, they are generally oversold and pushed out into the mainstream before being thoroughly vetted or understood. 

With regards to BIG DATA and its application(s) within the electric power industry, I would suggest a more precise concept of “Smart Use of Smart Data.” Managing a gross database containing massive amounts of information can be an overwhelming burden to those charged with the responsibility of overseeing utility operations, maintenance and other similar functions.  Many readers will no doubt have witnessed examples of equipment, systems and programs dedicated to the collection of large aggregate databases that start out with the best of intentions, only to be shelved or abandoned because they are far too cumbersome or impractical to use and maintain. 

Rather than promote a concept of Big Data, I advocate a well-planned strategic approach that provides the end user(s) with clear, concise, actionable reports that contain useful data. For example, mining circuit breaker data such as counter operations, gas pressure, contact wear, motor function, speed and travel, and other historical and real-time data can be extremely valuable for use in smart condition-based maintenance (CBM), as opposed to generic time-based maintenance (TBM).  Other data that may be available such as status of trip and close coils, auxiliary contacts, phase currents and voltages, etc., may or may not be useful for apparatus condition assessment, and strategic use of this data can greatly reduce and minimize file size while focusing on critical, necessary data only. 

The elephant in the room is data security. Cyber security is a significant concern within the community.  Data security is tantamount to mainstream acceptance and implementation. Data has value and as such requires systems and methods that protect it from theft, hacking, corruption and attack. All parties involved in data sharing must have their privacy and security issues addressed and made bulletproof. 

As chairman of an IEEE Working Group that has recently collected a vast database for support of standards development, I have first-hand experience of the difficulties involved in satisfying the commercial concerns and security needs of the parties involved; the general overwhelming aversion of data holders to contribute any data that may compromise a contributor’s commercial interests; and difficulties associated with data central storage and archival protocols, usage rules, and access security. 

These are some of the more significant issues that I believe to be associated with the advent of a data-rich environment.  There are many others issues associated with the generation, usage and storage of data, and while I do expect that the electric power industry will continue to gradually move to a smarter grid, I personally do not anticipate an overnight dramatic shift.  The industry has not historically ever moved rapidly in any direction.

To join the BIG DATA conversation, please comment in the box below.

Related content:

Data in Hand. Now What?

Over the last 10 years, Maquoketa Valley Electric Cooperative (MVEC) has been able to reduce its outage hours by 37% using technology. Many of the cooperative’s systems have been in place for more than a decade, but recent efforts to integrate data from its SCADA system, AMI, OMS, GIS and CIS systems  have resulted in service reliability. on

Managing Big Data

In the digital era, grid modernization involves the addition of devices and systems that produce data — and lots of it. That trend inevitably leads to the use of apt terms such as “Big Data” and the colorful if daunting metaphor “tsunami.”

Pertinent questions thus arise from those charged with a utility’s technology roadmap: How can a utility manage, protect and extract value from Big Data? What approaches offer investment protection? on

Voice your opinion!

To join the conversation, and become an exclusive member of T&D World, create an account today!