A New Approach to Self- Service Data Analytics Architecture
Colonel John Boyd is considered one of the Air Force’s greatest military strategists, due in part to his development of the OODA Loop; the recurring cycle of Observe Orient Decide Act that describes how decision making happens.
By studying combat aviation, Boyd observed the entity who processes this cycle quickly, observing and reacting to unfolding events more rapidly than an opponent, will “get inside” the opponent’s decision cycle and gain advantage.
Putting his methodology in a business context; companies gain significant advantages when they make decisions faster and more accurately than their competitors.
However, for companies that adopt a self-service data analytics culture, we need to apply a redefined version of the OODA Loop to get ahead of business forces.
The Linear Process of Self Service Data Analytics
The below diagram outlines the flow of data within a self-service data analytics platform
Sources: The origination point for all data in a system. Sources are classified in terms of accessibility (public or private) and type (cloud, file, or database).
Storage Area: Storing and versioning data in a separate, business-owned location allows access for historical trending, and ownership of data when a contract is canceled with a third-party vendor. Data is moved to the Storage Area via data pipelining tools with light ELT (Extract, Load, Transform) or ETL methodology (Extract, Transform, Load).
Processing Area: A crucial step in analysis; where data is merged, cleansed, or enriched to better relate to your business. ELT & ETL is primarily used in this area.
Modeling Area: A sub-area of processing, where data is related to each other using a data model to streamline analysis, serving as the foundation of the analytical layer.
Analytical Area: Also known as the semantic layer, this area allows users to explore and understand the data available to them for analysis. Analytic views, adhoc querying, and automation processes live here for reporting ease of use.
Reporting Area: User-friendly areas of your business intelligence environment. Through applications, refreshable reports, or dashboards, users leverage the previous stages to make actionable decisions.
The architecture is optimized so multiple analysts can interact with data concurrently, allowing data-driven decision making and an extreme competitive advantage in speed to act against unexpected business forces.
However, companies want to predict their environment, not just react to it. OODA can’t support this and needs to be augmented to meet this business need.
OODA Loop’s Missing Link
OODA is inherently a reactive decision-making cycle (due to the nature of fighter pilots), not a proactive one. In translation to the business context, the cycle can only be used to react to external or internal forces.
However, companies that lead their industries are those who better predict business forces, rather than react to them. To achieve this next step in maturity, a new component must be incorporated early in the decision cycle: Investigation and Questioning.
Companies now have a framework that pivots questions from the reactive “how can we address current competing products in the marketplace?” to the proactive “what do we do with our products most vulnerable to new offerings from competitors?”
QOODA and the Short Comings in Traditional Self-Service Data Analytics
With the introduction of questioning into the decision cycle, OODA become QOODA. However, QOODA reveals longstanding issues in traditional architectures: multiple versions of the truth, and duplicate work efforts.
Taking the example of forecasting product revenue against new offerings from competitors, executives might instruct multiple analytical teams to investigate the impact from within their respective business groups. Without an avenue for analysts to search for related reports, metrics, and data sources, they could be forced to intake isolated data sources to support their reports.
Many of these data sources may have the same metrics names, like “product”, “market share”, or “forecast”, but have different business logic associated with each unique data source and team. These differing business rules create different “truths” for the same metric in each team’s report, and frustrate executives because they cannot coherently answer a seemingly simple question.
An improved, modern self-service data analytics architecture is needed.
Supporting the QOODA Loop with Modern Self Service Data Architecture
Expanding analytics platforms that satisfy the need to quickly answer new questions requires the addition of two new elements: A data catalog, and a data governance committee.
Data Catalog: Data catalogs contain metadata and data lineage from data sources, as well as reports in a company’s data environment. Think of it as the internal data search engine for an organization. Users can now query what data and reports are available for consumption, preventing duplicate data being ingested, replication of reports, and enforce a single version of truth.
Data Governance Committee: Data governance committees regulate and approve data sets, reports, access, and uphold all data & report quality standards.
In our example, the data catalog could have been used to query which data source had the appropriate forecast measure, and the data governance committee would have prevented both reports from being exposed with two differing metrics.
To achieve the QOODA Loop, these two elements are critical to the data driven decision making process. Without them, the integrity of decisions made and the insights discovered will be in question.
Without foundational trust in data, rapid decision making in the form of an OODA or QOODA Loop will leave users in disarray rather than gaining a competitive advantage.
The speed of business accelerates each year and decision making processes must adapt. Moving to the proactive QOODA loop, and adding data cataloging and a data governance committee to the traditional self-service architecture will lead to major competitive advantage. Helping your organization make decisions faster and more accurately with all data at your disposal will be the differentiator between companies that lead in the future and those that lag in the antiquated.