The challenge and promise of unstructured data
Utilities navigating a more complex world of data
Published In: Intelligent Utility Magazine July / August 2012
AS UTILITIES HAVE WADED INTO THE DATA-HEAVY POST-smart grid era, utility leaders from across the enterprise are seeing that there is value in all of that data beyond simply managing the volumes of data and getting it to the right people. This, of course, is where analytics is proving to be a game changer at many utility organizations.
We'll be discussing these and other issues at Utility Analytics Week in September.
Turning the volumes of data into actionable intelligence is what the analytics game is all about, and utilities are finding more ways to score points in this game on a seemingly daily basis. The examples range from new business processes for managing assets predictively to segmentation, and targeting customers for new programs to re-shaping the load curve, and more.
These examples-and countless others-are all dependent on the ability to move the organization from using all of that data to report what happened to operating predictively by taking the knowledge from the historical data and changing business processes based on this knowledge. When done well, the improvements across customer, financial, operational and regulatory requirements can be significant.
Opportunities and challenges
Inherent in these opportunities are a new set of challenges, including the emergence of unstructured data in utility customer and grid databases. Unstructured data (or unstructured information) is information that either does not fit into a pre-defined data model and/or does not play well with relational tables. Unstructured data is typically characterized as being text-heavy, but may also have numeric information, such as dates and raw numbers. This lack of structure creates irregularities and ambiguities that make it difficult to understand using traditional computer programs that would pull data from more tabular types of database structures.
Examples of how different unstructured data can be include text data from various social media platforms, voice data from customer service telephone calls that have been converted to text, or possibly even data from synchrophasers out on the distribution grid. Nuggets of quantitative and qualitative information are buried in these unstructured piles of data, but they require different management, manipulation and application tools to be effective.
Mining free text fields
Another example of unstructured data that most utilities encounter is from free text fields in systems and survey applications. Customer service representatives or customers populate these fields with valuable customer data in an uncodified format that's hard to mine.
Mari Vandewettering at Portland General Electric in Portland, Oregon, tells of the utility's challenges in working with this type of unstructured data: "A lot of the unstructured data that we see is in free text fields in our operational systems and from surveys that customers fill out directly on our website, or in our community offices. We're still learning how to work with this and are using some new text-analytics software tools to get there, but we are starting to use this data to decipher customer sentiment and digest and categorize customer feedback."
Going a bit broader and deeper to yield new insights and tangible, measurable improvements is where the use of unstructured data is heading for utility analytics professionals. Alyssa Farrell, global energy product strategy lead at SAS, provides a few more examples of how utilities can benefit from analyzing unstructured data. "We see a variety of applications for unstructured data at utilities-some of which are in use today, others of which are being used in other industries that can be applied to utilities," she said.
Areas with immediate value, she said, include using call center data to improve outage root cause analysis, or even the use of technician's notes from standard maintenance or emergency replacements of field assets to build predictive models of future asset failures. Reiterating the challenges noted above, Farrell points out that "utilities have to turn to data storage mechanisms such as Hadoop that do not impose structure on the data," and that tools built for analyzing the sentiment in text are called for to help capture the value in this new data.
Pointing to how critical unstructured data can be for customer service, Bryan Truex, Teradata's utility industry director, noted that "up to 80 percent of how customers communicate is unstructured. Utilities need to ask if they can afford to ignore up to 80 percent of their customer contact data."
A third customer interaction dimension
Continuing with the customer management challenges and how utilities are using unstructured data to meet these challenges, Dan Burgess, an analyst at Avista Utilities, based in Spokane, Washington, added an interesting perspective. "We have historically looked at our customer interactions in two dimensions, typically based on time and cost," he said. "These are helpful, and help us build some quantitative metrics, but this neglects the third dimension, that of the customer perspective." Burgess also noted that this new type of customer data has added a qualitative element to the utility's customer service metrics.
As utilities continue to drive forward in this new era of analytics-centric management, the ways that they are using all of this data-be it traditional, tabular data or the more complex unstructured data-continue to evolve. Social data didn't even exist until a few years ago, nor did interval meter data or deeper, broader availability of asset data via sensors deployed across the grid.
As utilities learn to move from simply managing this data to leveraging it for predictive operations, the impact will be seen and felt across the enterprise and from the executive suite to the field crews.