October 26, 2014

Analysts are mostly working on the “known unknowns”, while the “unknown unknowns” remain the “holy grail” of analytics.

The concept of “knowns and unknowns” is attributed to Donald Rumsfeld, former US Defense Secretary, back in 2002:

“… there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns -- the ones we don't know we don't know.” (see for instance Wikipedia)

Rumsfeld spoke in reference to counter-terrorism intelligence, but the concept is applicable in “civilian” analytics, research, and Business Intelligence as well.

An unscientific model of knowns and unknowns, and the efforts, methods and tools existing in each space:

Obviously there are a lot of “known knowns” out there, often as openly available information on the internet (news reports, blog and web sites, wikis and so forth). Inside corporations there’s usually an abundance of reports and statistics on various business parameters like revenue, turnover, logistics and more.

Organizations are capable of developing “blind spots” that might be considered “unknown knowns”. This is knowledge that is forgotten, displaced, overlooked or misunderstood/misinterpreted, and needs to be rediscovered or re-learned. Modern information systems, databases and intranets help build a “collective memory” for the organization. Findability is often key in keeping “unknown knowns” from becoming a problem.

In analytics and Business Intelligence, the focus so far has been on the “known unknowns”. That is, analysts, researchers and data scientists are attempting to find answers to specific challenges facing the organization (or society at large). There are many tools and methods available: search engines, data mining, text mining, NLP, sentiment analysis, and more. Analysis of “known unknowns” can, for instance, find valuable information on customers and markets, identify causes of revenue changes, or map the spread of viral diseases. By using historical data and predictive modelling, analysts can forecast future developments and enable businesses to stay ahead of the game.

The “holy grail” of analytics, though, is to find the “unknown unknowns”: identify patterns in data that we didn’t set out looking for; get early warning of events we don’t know will occur; discover relations between people, places or entities we’re not even aware of yet.

In the era of Big Data, the quest for the “unknown unknowns” is gathering pace. Endless amounts of data, vast computing resources, and increasingly sophisticated algorithms make it possible to uncover hidden patterns and reveal unseen relationships.

Machine Learning (ML) and Artificial Intelligence (AI) are being developed as the primary technologies for dealing with the “unknown unknowns”. Open source software, standards for data formats and integration, cloud storage and cloud computing are enablers in this process.

Still, the human analyst has an important role to play in identifying “unknown unknowns”, determining their importance, and alerting decision makers. Employing Agile analytics methods, with the ability to “pivot” when data points you in a new direction, will be necessary.

By combining top-notch analytical capabilities with smart tools and Big Data, the “holy grail” of analytics may be within reach.


February 16, 2014

A recent article in Harvard Business Review (HBR) about disruption in the consulting industry positions Big Data as one of the main forces of disruption. Will data scientists, fast computers, and advanced software replace experienced consultants and tried-and-trusted methodology?

Disruption, or fundamental changes to an industry, has hit many traditional businesses over the past years. Since the internet became commonplace in the mid- to late-90s, disruption has been a fact of life. Established – and often stale - industries like retail, travel, and media, to name a few, have been transformed by the arrival of internet-based services. New players such as Google, Amazon, and eBay have built their success largely by using Big Data at the core of their businesses.

Disruption in consumer markets is getting most of the attention, but Business-to-Business (B2B) is also ripe for change. Trade, transportation, manufacturing and many other industries are affected by not just globalization, but also forces like connectivity, digitalization, and technological leaps.

Many B2B service sectors have also gotten their share of disruption. Advertising and media has seen a lot of change since the internet arrived in full, with new media channels, media products and advertising networks. IT, customer service, accounting and others have been transformed by global outsourcing. In the legal industry, mentioned in the HBR article, traditionally exclusive bonds between clients and their law firms are loosening.

Consulting, though, has been thought to be immune against disruption, maybe because it’s usually consultants that promote change in other industries. Why shouldn’t they be able to stay ahead in their own game?

Odds are that they won't. The HBR article provides McKinsey Solutions as a sign that disruption is coming to the consulting industry as well. McKinsey Solutions, established in 2007, delivers analytics software, tools that can be embedded at clients and used in shorter projects for faster results. While McKinsey are trying to fight disruption this way, there's any number of startups, niche players, and crossovers that want to get in on the action.

And there is no doubt that Big Data, combined with Agile analytics, is a serious challenge to established thinking in management consulting and business development. Rather than spending months on studying internal work flows, interviewing customers, or discussing with key personnel, you could check the (Big) data.

Businesses today continuously create new data - from sales and marketing, customer relations, production, logistics and more. In addition, there’s a huge amount of data generated by web sites, public databases, social media, and, increasingly, the Internet of Things. By tapping into Big Data, using Agile analytics methods, analysts can rapidly respond to business needs and deliver timely results to decision makers.

For instance, to find out what customers think about a new product, a company can search social media data for mentions and analyze sentiments (positive/negative mentions). This can be done in real-time, while the traditional methods of focus groups and phone interviews might take weeks or even months to produce sufficient results for decision-making. In many ways, the mindset necessary to operate in the age of Big Data is so fundamentally different that only new, tech-savvy players will be able to grasp it.

Still, the arrival of data scientists and Big Data analytics doesn’t eliminate the need for the traditional business consultant. Rather, the business side, IT and analytics must work together to achieve great results. Deep knowledge of business processes, markets, and customer behavior is required to ask the right questions and pose the right hypotheses. Tech-based newcomers in the consulting services marketplace will have to acquire these business skills, or collaborate with someone that already possess them.

At this point it remains to be seen if the traditional consulting houses will be able to respond to the threat of disruption by developing their services and customer relationships - or if new and more agile players with completely new offerings will take over. Recent history definitely suggests that the changes that are coming will be fundamental - disruptive.

Reference: Harvard Business Review, Oct 2013: “Consulting on the Cusp of Disruption”, by Clayton M. Christensen, Dina Wang, and Derek van Bever.