The challenge and promise of unstructured data

Utilities navigating more complex world of data

Mike Smith | Jun 10, 2012

Share/Save  

As utilities have waded into the data-heavy post-Smart Grid era, utility leaders from across the enterprise are seeing that there is value in all of that data beyond simply managing the volumes of data and getting it to the right people. This, of course, is where analytics is proving to be a "game changer" at many utility organizations. 

We'll be discussing these and other issues at Utility Analytics Week

Turning the volumes of data into actionable intelligence is what the analytics game is all about, and utilities are finding more ways to do score points in this game on a seemingly daily basis. The examples range from new business processes for managing assets predictively to segment and targeting customers for new programs to re-shaping the load curve, and more. These examples—and countless others—are all dependent on the ability to move the organization from using all of that data to report "what happened" to operating predictively by taking the knowledge from the historical data and changing business processes based on this knowledge. When done well, the improvements across customer, financial, operational, and regulatory requirements can be significant.

Inherent in these opportunities are a new set of challenges, including the emergence of unstructured data in utility customer and grid databases.  Unstructured Data (or unstructured information) is information that either does not "fit" into a pre-defined data model and/or does not "play well" with relational tables. Unstructured data is typically characterized as being text-heavy, but may also have numeric information as well, such as dates and raw numbers. This lack of structure creates irregularities and ambiguities that make it difficult to understand using traditional computer programs that would pull data from more tabular types of databases structures. 

Examples of how different unstructured data can be include text data from various social media platforms, voice data from customer service telephone calls that have been converted to text, or possibly even data from synchro-phasers out on the distribution grid. Nuggets of quantitative and qualitative information are buried in these unstructured piles of data, but they require different management, manipulation, and application tools to be effective. 

Another example of unstructured data that most utilities encounter is from free text fields in systems and survey applications.  Customer service representatives or customers populate these fields with valuable customer data in an un-codified format that's hard to "mine."  Mari Vandewettering at Portland General Electric in (Portland, OR) tells of their challenges in working with this type of unstructured data: "A lot of the unstructured data that we see is in free text fields in our operational systems and from surveys that customers fill out directly on our website, or in our community offices. We're still learning how to work with this and are using some new text-analytics software tools to get there, but we are starting to use this data to decipher customer sentiment and digest and categorize customer feedback."

Going a bit broader and deeper to yield new insights and tangible, measurable improvements is where the use of unstructured data is heading for utility analytics professionals. Alyssa Farrell, Global Energy Product Strategy Lead at SAS (Cary, NC), provides a few more examples of how utilities can benefit from analyzing unstructured data: "We see a variety of applications for unstructured data at utilities—some of which are in use today, others which are being used in other industries that can be applied to utilities."

Alyssa points out that areas with immediate value include using call center data to improve outage root cause analysis, or even the use of technician's notes from standard maintenance or emergency replacements of field assets to build predictive models of future asset failures. Reiterating the challenges noted above, Ms. Farrell points out that "...utilities have to turn to data storage mechanisms such as Hadoop that do not impose `structure on the data," and that tools built for analyzing the sentiment in text are called for to help capture the value in this new data.

Pointing to how critical unstructured data can be for customer service, Bryan Truex, Utility Industry Director at Teradata (Dayton, OH), notes that "...up to 80 percent of how customers communicate is unstructured. Utilities need to ask if they can afford to ignore up to 80 percent of their customer contact data."

Continuing with the customer management challenges and how utilities are using unstructured data to meet these challenges, Dan Burgess, an Analyst at Avista Utilities (Spokane, WA) adds an interesting perspective: "We have historically looked at our customer interactions in two dimensions, typically based on time and cost. These are helpful, and help us build some quantitative metrics, but this neglects the third dimension, that of the customer perspective." Dan also comments that this new type of customer data has added a qualitative element to their customer service metrics.

As utilities continue to drive forward in this new era of analytics-centric management, the ways that utilities are using all of this data—be it traditional, tabular data or the more complex unstructured data—continue to evolve. Social didn't even exist until a few short years ago, nor did interval meter data or deeper, broader availability of asset data via sensors deployed across the grid. As utilities learn to move from simply managing this data to leveraging it for predictive operations, the impact will be seen and felt across the enterprise and from the executive suite to the field crews.

Mike Smith
Vice President
Utility Analytics Institute
msmith@energycentral.com 

Related Topics