Introducing Data Core Architecture: The evolution of Data Fabric

Introducing Data Core Architecture: The evolution of Data Fabric

Data Core Architecture. Centralized Source of Truth for a Data Assets Democratization

1) Motivation - Why is Data Fabric not enough?

In 2014, Jay Kreps introduced Kappa Architecture, a software approach designed for data streaming and system decoupling. 

The Data Fabric concept, aimed at connecting data across organizations, was first introduced by Forrester in 2013. Its core principle is data integration, which addressed the predominant challenge of the 2010s—siloed data. However, today’s organizations manage hundreds or even thousands of integration interfaces, struggling to support existing systems while developing new use cases. This growing complexity has made it evident that the traditional approach to data integration is outdated. 

In 2023, Forrester introduced “Data Fabric 2.0 for Connected Intelligence”, a reference architecture that incorporates real-time data management. This approach leaves gaps in real-time data streaming, which is essential for both data processing and temporary persistence. By integrating data streaming, organizations can achieve real-time data democratization for both analytics and operational use cases. 

The post-COVID technological acceleration—particularly in AI and automation—demonstrated that traditional data integration alone is insufficient for agile business innovation. A significant challenge lies in the communication gap among various organizational roles, including Business Process Owners, Functional Consultants, Developers, Data Engineers, Data Analysts, and Data Scientists. Without a shared understanding of data and data logic, aligning on a single use case often takes months of meetings and iterations. 

To complement data integration, organizations turned to Data Governance, which encouraged a paradigm shift—thinking of business processes in terms of Data Assets, structured as Data Entities and Logical Data Assets. This approach established a ubiquitous language for each business process domain, fostering better collaboration.  

To evolve and enhance collaboration, Data Governance required an operational framework, leading to the emergence of Data Operations (DataOps).  

Together, Technical Data Architecture, Data Governance, and DataOps form the foundation of an Enterprise Data Strategy. In the digital economy, an Enterprise Data Strategy’s primary goal is to accelerate business objectives and drive value.  

Organizations quickly recognized the need for a dedicated role to manage the Enterprise Data Strategy and lead the transition from process-centric to data-centric operations. This led to the rise of the Chief Data Officer (CDO/CDAO).   

CDOs soon encountered key challenges: Evangelizing the adoption of an Enterprise Data Strategy based on Data Assets, Data Governance, and DataOps. 

  • Demonstrating business value through a data-centric approach. 

  • Providing a clear roadmap for transitioning from process-driven to data-driven operations. 

2) Business Challenge: The Digital Economy's Demand for Agile Business Innovation & Technology Transformation  


Every industry—from manufacturing to banking and healthcare—is undergoing a digital transformation. Every digital interaction generates a data footprint, which must be connected and transformed into democratized Data Assets to create business value. These assets enable real-time decision-making, such as available-to-sell inventory, personalized product recommendations, and predictive plant maintenance.   
 
The rapid expansion of emerging technologies continues to drive an increasing number of business requirements, each of which must be translated into technology use cases. Every new use case demands specific data sets and business logic, adding to the complexity of modern data ecosystems.   
With AI-driven software development, technology will evolve at an unprecedented pace. Only companies that can adapt quickly will be able to meet the growing expectations of users and consumers, ensuring their relevance in an increasingly digital world.  

3) The Solution: Data Core Architecture

The Digital Economy demands hyperconnected Data Assets to drive innovation and agility.  

Data Core Architecture is more than just a data processing framework—it is a comprehensive approach that enables organizations to become truly data-driven.  



3.1 Technical Perspective: Enabling a Centralized Source of Truth

By democratizing data through event-driven messaging, while maintaining governance for Data Assets, Data Contracts, and Streaming Data Lineage, Data Core Architecture establishes a centralized, trustworthy source from which the entire technology landscape can consume reliable information. 
Traditionally, business logic has been encapsulated within functional specifications that define tightly coupled interface programs. This process-centric approach has resulted in a massive number of rigid, hard-to-maintain interfaces, limiting scalability and reusability. 
In contrast, a data-centric approach represents each use case as a combination of: 

  • Atomic Data Assets – The foundational data model structured by business domain, following a Domain-Driven Design (DDD) approach. Each entity is uniquely identifiable. 

  1. Example: Customers, Products, Orders, etc. 

  • Logical Data Assets – The business logic layer, responsible for calculations and system interactions (e.g., interfaces, reports, or analytics). 

  1. Example: SAP-to-Salesforce Customer Integration, Available-to-Sell Inventory, Customer 360 View. 

A deeper dive into the Data Core Streaming Layers will be provided in a later section. 

3.2 Business Perspective: Agility, Improved User Experiences & Democratized Decision-Making

To stay competitive, Business Process Owners across the enterprise continuously define new business requirements, which must be efficiently translated into technical use cases.
A Data-Centric approach enables organizations to “plug in” new use cases as consumers of existing Atomic and Logical Data Assets, while also producing new Data Assets. This modular architecture accelerates the integration of new business capabilities, fostering agility, scalability, and data-driven decision-making.

4) Streaming Data Assets: Data Core Layered Approach to Data Democratization 


Data Core structures data into incrementally optimized layers for usability.

Data Core Streaming Layers

1. Raw Data Ingestion:

  1. Bronze Layer: Tables, JSON, XML foundational level for data products.   
  2. Structured data: Data is organized with a defined schema, easily stored, and query-friendly. 
  3. Semi-Structured data: Data with some organizational structure, like JSON or XML, but no fixed schema.   
  4. Example: Tables, CDS views, JSON

2. Atomic Data Assets 

  1. Silver Layer: Emphasizes their role as the building blocks for higher-level or derived data products.
  2. Data Entities: Refers to structured representations of domain objects (e.g., customers, products, orders) within a system or bounded context.
  3. Canonical Data Models: Refers to standardized representations of data entities across systems or domains.
  4. Example: Customer, Material, Sales Order, Delivery, Shipment, Invoice.

3. Logical Data Assets

  1. Data Products: Discoverable, reusable datasets designed for specific domain use cases. 
  2. Interface Data Contracts: Last mile layer for integration endpoints with other APIs or Operational Systems. 
  3. Calculated Data Entities: Data entities derived from transformations or computations on atomic or composite data.
  4. Example: Customer 360, Available to Sell Inventory Situation, Order Status   

5) The Five Core Components of Data Core Architecture 

The foundation of Data Core Architecture is its ability to ingest event messages from all data sources—including operational and analytical systems across on-premises, cloud, and edge environments. 

While Data Core may include additional components for enhanced data management (such as ontology), five fundamental components ensure a Centralized Source of Truth: 

  1. Event-Driven Messaging – Enables real-time communication between systems. 

  1. Publish and Subscribe Persistence – Facilitates system decoupling for seamless data flow. 

  1. Stream Processing – Transforms raw data into real-time Logical Data Assets. 

  1. Data Governance – Ensures data and metadata discoverability and manages Data Assets. 

  1. Data Lineage – Provides traceability and quality assurance for reliable data. 

These components work together to establish trustworthy, democratized data that drives business value and innovation.  

6) Bridging the Gap – Data Asset Modeling with Domain-Driven Design for Enterprise Integrations  


One of the biggest challenges in enterprise data initiatives is the communication gap between Business Process Owners, Functional Consultants, Developers, Data Engineers, Data Analysts, and Data Scientists. Misalignment in data definitions, business logic, and technical implementations often results in prolonged development cycles and inefficiencies. 
Domain-Driven Design (DDD) for Data Asset Modeling offers a structured approach to address this issue by establishing a ubiquitous language—a common set of terms and definitions shared between business and technical teams. By defining Atomic Data Entities and Logical Data Assets, organizations can create a shared understanding of data and data logic, enabling seamless collaboration, faster alignment on use cases, and more efficient enterprise integrations. 

The following key concepts form the foundation of Domain-Driven Design (DDD) for Data Asset Modeling: 

Domain   
A domain refers to the area of expertise or the business domain. It represents the problem space or the capabilities that the business unit aims to address. In other words, it's the core business unit or primary focus of the Business Process in the business unit.

Subdomain   
A subdomain is a subset or a specific area within the larger domain. It represents a specialized part of the domain that requires distinct knowledge, rules, or processes. Subdomains often correspond to specific business capabilities, departments, or teams within a Business Process.

To illustrate this concept, consider an SAP Sales and Distribution example: 

• Domain: 
• Order to Cash – B2C Sales 

• Subdomains:

• Product Catalog Management  
• Order Management  
• Inventory Management  
• Payment Processing  
• Customer Relationship Management 


Entities

Entity is a core concept in Data Core Architecture. An Entity represents objects within the domain (Business Process) that have a unique identity that persists over time, regardless of changes to their attributes. 

  • Identity: The most important feature of an entity is its identity, which distinguishes it from other entities. Even if two entities have the same attributes, they are considered distinct because they have different identities.   

  • Lifecycle: Entities typically have a lifecycle; they are created, modified, and eventually archived or deleted. The business rules dictate how entities evolve over time.   

  • Examples: Common examples of entities include a Customer, Order, or Product. These objects are distinct in the system, even if their details change, such as a customer’s address or a product’s price.   

7) Initiating the Data-Driven Journey

When an organization embarks on the transition to becoming data-driven, several key questions often arise: 
1. How do we transition from a Process-Centric to a Data-Centric approach? 

2. Which entities should we model first? 

3. Which roles and profiles should be involved in the project? 

    To address these challenges, the One Journey methodology, which will be published in a separate document, provides a systematic and guided approach to achieving data democratization.


The Four Principles of the One Journey Methodology 

1. Emerging Design: Based on Agile development, this principle ensures that data entities and logical data assets evolve organically as teams iterate and refine their understanding of the domain. 

2. Refining Models Through Iteration: As teams progress, they may uncover hidden complexities or opportunities for simplification, leading to continuous refinement of the domain model. 

3. Agile Governance: A feedback-driven approach ensures that both data entities and logical data assets evolve with the business, keeping teams aligned with organizational goals. 

4. Retrospectives for Domain Learning: Teams should regularly review and reassess the domain model to ensure it accurately reflects current business realities, resolve misunderstandings, and iterate on the design as needed. 

The Two Phases of the One Journey Methodology

Adopting a Data-Centric approach, the methodology consists of two key phases:

    1. Model: Discover and allow the data model to emerge naturally from the use case, while also defining the governance policy.
    2. Connect: Generate the Streaming Logical Data Assets needed to integrate and operationalize the use case.



Reference Architecture: Enabling a Unified, Real-Time Data Ecosystem


A modern reference architecture leverages event-driven data collection and stream processing to transform raw events into Logical Data Assets, creating a centralized and trusted source of truth. By continuously processing and enriching data in motion, organizations can ensure real-time availability, accuracy, and consistency across all systems. This approach democratizes Data Assets, enabling seamless access to reliable, domain-driven insights for both operational and analytical use cases, ultimately driving business agility and innovation. 


About Onibex


Gustavo Estrada, founder of Onibex, is a passionate technologist with over two decades of hands-on experience in SAP technology. Since 2003, he has been at the forefront of SAP’s evolution, working with R/3, ECC, XI, PI, CPI, Portal, SAP S/4HANA Rise, HANA, and BTP. In 2014, as SAP Practice Director of Technical Solutions at Idhasoft, he played a key role in the company's recognition with the SAP Americas Award for Most Net New Logos. Facing persistent integration challenges, Gustavo founded Onibex in December 2015 with the vision of simplifying and democratizing SAP data access. 

For nearly a decade, Onibex has remained relentless in its mission, drawing inspiration from technology pioneers like Netflix, studying business transformation journeys from companies like Walmart, and learning from industry leaders about Cloud Computing, Event-Driven Architecture, Stream Processing, Data Governance, Data Lakehouses, and Real-Time Data Warehousing. 

Today, Onibex stands at the intersection of business and technology, leading the way in designing cohesive, value-driven technology architectures. This vision led to the creation of two key concepts: 

  • Data Core – A centralized data framework designed to democratize Data Assets across the organization. 
  • One Journey – A structured methodology that helps organizations of all sizes and industries become truly Data-Driven and thrive in the Digital Economy. 



    • Related Articles

    • Introduction to eCommerce360

      ONIBEX's B2B (Business to Business) system comprises all the most advanced features of any E-Commerce system; but besides, our 360° E-Commerce system is fully integrated to the SAP Enterprise Resource Planning (ERP) system. This chapter describes in ...