Connect, Connecting, Connected Should be the Mantra; not Big, Bigger, Even Bigger!

To emphasize the title of my blog, let me lay out two pivotal events in US history. They were both significant, tragic and tsunami events for humanity and global security. The first was the attack on Pearl Harbor, the second were the attacks of 9/11. There was no lack of data leading up to both these catastrophic events. In fact, the data collected was large, from a diverse variety of sources and was growing at increasing rates. This was the ‘Big Data’ of its time. What made both events tragic was the inability of experts who were receiving the information to connect across the data assets so they could identify patterns, trends and associations. Let me be clear, it was not for the lack of competent experts. In fact, at the time of both events the US intelligence community had the world’s best analysts; it still does. The point I am trying to make is that the data was sitting in organizational and technical silos that hindered connecting them and inferring signals.

Let me explain a bit further. There is a tremendous amount of analysis on the Pearl Harbor attack of December 7, 1941. However, the most compelling and captivating contribution, in my opinion, comes from Roberta Wohlstetter, via her book, Pearl Harbor: Warning and Decision (Stanford, CA: Stanford University Press, 1962). Ms. Wohlstetter was a military intelligence historian who worked for the RAND Corporation and was presented with the Presidential Medal of Freedom by President Ronald Reagan in 1985. Her body of work was very influential in laying out the framework that most intelligence failures are assessed with; that intelligence failures happen when warning signals are available but get missed for other background/noise. She wrote, in connection to the data available before the Dec 7, 1941 attack, “Never before have we had so complete an intelligence picture of the enemy,” “In short, we failed to anticipate Pearl Harbor not for want of the relevant materials, but because of a plethora of irrelevant ones.” She goes on to state that, “it is much easier after the event to sort the relevant from the irrelevant signals. After the event, of course, a signal is always crystal clear; we can now see what disaster it was signaling since the disaster has occurred. But before the event it is obscure and pregnant with conflicting meanings.”

In the executive summary of the 9/11 commission’ (National Commission on Terrorist Attacks Upon the United States) final report there is a section titled “Unity of Effort: Sharing Information”. The commission stated at the time that, the U.S. government has access to a vast amount of information. But it has a weak system for processing and using what it has. The system of “need to know” should be replaced by a system of “need to share.” The section went on to state as well that, official after official has urged us to call attention to problems with the unglamorous “back office” side of government operations…. to build the network requires an effort that transcends old divides…in tackling information issues, America needs unity of effort.

When we miss signals in our datasets and analysis it does not happen because we have limited information that is not arriving fast enough; in my experience, we often miss signals because there is too much information moving too fast. The tendency to take an “everything by everything” approach to data collection and harmonization is the definition of what Ms. Wohlstetter wrote almost 50 years ago and I paraphrase here; we will fail to identify signals “not for want of relevant materials, but because of a plethora of irrelevant ones.”

We need to be fiercely mindful of collecting the right data, harmonize only the key parts that need to be connected and then analyzed for greater impact. When we take this mindful approach, and only when we do this, can we provide actionable insights to decision makers who then need to be receptive and courageous to act on that information.

Great analytics teams know how to communicate, relate and contextualize information from multiple sources. They never, let me repeat, never live in “one data source” to make inferences. When information is connected and triangulated in meaningful ways one can identify signals and land insights more effectively. In my observation organizations that are benefiting from the big data boom were also leading the pack when they had ‘small data’ 😊! It was never about ‘big data’ or ‘small data’; it was always about how teams connect these data assets so they may be analyzed to reveal patterns and trends.

Mindful analytics teams “address the unglamorous 'back office' side of their data operations to connect data and build the network required to transcend old divides and to tackle information issues, they demonstrate a unity of effort to see through the fog of pregnant and conflicting meanings.”

Be that team!