Big data: threat or opportunity?
McKinsey report touts benefits
We have often written about the importance of data management and analytics, especially given the proliferation of data that often results from utility smart grid investments.
A recent report from the McKinsey Global Institute shows that our industry is not alone. With computers and cell phones continuing to pervade our daily activities and as millions of networked sensors are being embedded in numerous devices (such as automobiles, smart meters and other machines), the amount of data available for analysis is exploding across all sectors.
McKinsey's report, "Big data: The next frontier for innovation, competition, and productivity," looks at the vast amount of enterprise information that exists, and the challenges that organizations will face in trying to manage it. The report explores topics such as the state of digital data and how organizations can use large data sets to create value.
McKinsey's analysis suggests that companies and policy makers must tackle significant hurdles to fully capture data's potential. The United States alone faces a shortage of 140,000 to 190,000 people with analytical and managerial expertise and 1.5 million managers and analysts with the skills to understand and make decisions based on the study of data.
The McKinsey report also identified five broadly applicable ways to leverage big data:
- Make big data more accessible and timely. Transparency, enabled by big data, can unlock value.
- Use data and experiments to expose variability and raise performance. As organizations create and store more transactional data in digital form, they can collect more accurate and detailed performance information on everything from product inventories to sick days.
- Segment populations to customize. More data allows organizations to create ever-narrowing segmentations and to tailor services precisely to meet customer needs.
- Use automated algorithms to replace and support human decision making. Sophisticated analytics can substantially improve decision making, minimize risks and unearth valuable insights that would otherwise remain hidden.
- Innovate with new business models, products and services. To improve the development of future offerings to customers, leverage data to better understand the use of current products and services.
The McKinsey report should prove quite useful to utilities looking to manage the flood of information resulting from smart grid endeavors. One thing for our industry to keep in mind is that with an increase in data, there is also a large increase in noise. The challenge is to extract the insights or wisdom from the data that will lead to greater success.
Christopher Perdue
Vice President, Sierra Energy Group
cperdue@energycentral.com
310.471.7396
Twitter: @chris_perdue
Comments
From data to intelligence
I think Chris and the McKinsey report summed up the challenge well, and certainly there is a hugh opportunity in parsing the data from the various IEDs into a normalized, addressable fashion. This is where companies like SAS, CapGemini, Accenture, etc. shine. The last line in Chris' column struck me as well, since at some point someone with utility experience and insight will still have to analyze the data to see what intelligence it can render. This is not a trivial exercise, since even deciding what data to group and view requires some insight into what data may have some correlation to measurable outputs. Therein lies the need for skilled, experienced, and insightful managers. Additionally, my experience has been that senior managers want dashboards that can give them real-time analytics of fairly straightforward and normally accepted measurements they need to run their business; I think the real challenge is just below these accepted KPIs, where middle managers have to make decisions on investment, how best to accomodate load growth in an environment where the generation mix is really more like managing a portfolio of options, introduction of microgrids, DG, DSM, etc. And these are just the utility-facing options. Sprinkle in ways to provide data to customers to aid their choices, and the flexibility to provide choices to them, and the calculus can be intimidating. I see future middle managers in the utility space (both those working for utilities and those working for companies that provide products and services to utilities) needing to be more skilled in business intelligence to complement their knowledge of the generation, transmission, and distribution business. And, oh by the way, the normal T&D business is rapidly changing as well. Not sure of the answers, but I am beginning to better understand the questions.
Charlie Nobles
Big Data First Mile and Last Mile Challenges
This is a good framing of the big picture issues for big data, Chris. But I would add the following. While technology has evolved rapidly to enable the scaling of our data capabilities the most difficult problems faced are still the first mile ETL issues of extracting data from many fragmented databases, cleaning and transforming it into a useful format, and loading it into the data warehouse where is is accessible. And the last mile issues of big data gaps such as access control by various users when data is comingled in one large 'bit bucket' and finding the expertise to frame the questions that produce useful insight which still requires vertical knowledge.