Modern data storage formats and working with Apache Parquet
26 February 2024
Data Governance, Data Minimalism and Data Swamp
26 February 2024

Data Warehouses and Data Lakehouse architecture

Musical compositions are made up of notes that combine to create complex pieces of music, while words are created from a primitive unit, the letter-sound. Whether it is nature, science, economics, art or technology, the world shows a remarkable unity and harmony in its quest for destruction, structure, cyclicity and creation.

In the same way, processes in costing systems are broken down into the smallest structured units - resource items - at the level of calculations and scheduling. These units are then used to form more complex calculations and schedules. CAD (BIM) systems work on the same principle, where complex architectural and engineering projects are built from basic elements - individual smallest elements or library components, from which a full 3D model of a complex building or structure is created.

The smallest elements with their finite indivisibility are the basic building blocks in all business processes. It is important to carefully consider from the very beginning how to collect and store these smallest building blocks from different sources. Decisions will also have to be made between approaches such as data warehouses (DWH) and data lakes (Data Lakes). These issues first arose in the telecoms industry, then became relevant for financial organisations, and now companies in the construction industry will face similar issues.

The concept of cyclicality and structure found in nature and science is also reflected in the modern world of data. Just as in nature all living things return to atoms and molecules, so in the world of modern data processing tools, information tends to move to the most primitive, molecular form possible.

This idea is embodied in the Data Lakehouse architecture, where data, like drops of water, merge into a single entity, forming a "lake" of heterogeneous information. The development of the Data Lakehouse concept represents an evolutionary step in the field of data management, emerging at the intersection of Data Warehouses and Data Lakes.

  • In traditional Data Warehouses, data is typically pre-processed, transformed and cleaned (ETL - Extract, Transform, Load) before it is loaded into the warehouse. This implies that the data is structured and optimized for specific analytics and reporting tasks. The main focus is on maintaining high query performance and data integrity. However, this approach can be costly and less flexible in terms of integrating new data types and rapidly changing data schemas.
  • Data Lakes, on the other hand, are designed to store large amounts of raw data in its original format. The ETL process is being replaced by ELT (Extract, Load, Transform), where data is first loaded into storage "as is" and only then can it be transformed and analyzed as required. This provides greater flexibility and the ability to store heterogeneous data, including unstructured data such as text, images and logs. The downside can be more complex query performance optimization and data management.

Traditional Data Warehouses, designed to store structured data in a format optimized for analytical queries, faced limitations in handling unstructured data and scalability. In response to these challenges, Data Lakes have emerged, offering flexible storage for large amounts of heterogeneous data.

Data Lakehouse is an architectural approach that seeks to combine the flexibility and scalability of Data Lakes with the manageability and query performance of Data Warehouses. Key features of Data Lakehouse include:

  • Open storage format: The use of open formats for data storage, such as Parquet (Apache), provides efficiency and query optimization.
  • Read-only schema: Unlike the traditional write-only schema approach in DWH, Lakehouse supports read-only schema, allowing more flexibility in managing data structure.
  • Flexibility and Scalability: Supports storage and analysis of both structured and unstructured data, while providing high query performance through storage-level optimization.

Thus, Data Lakehouse offers a compromise solution that combines the advantages of both approaches, making it ideal for modern analytical workloads that require flexibility in processing

Data Lakehouse is a synthesis of data lakes and data warehouses, combining the flexibility and scalability of the former with the query manageability and optimization of the latter.

Before data storage and infrastructure can be developed, Data Governance policies and standards must first be developed and approved at the data governance level.

Leave a Reply

Change language

Post's Highlights

    Stay updated: news and insights



    We’re Here to Help

    Fresh solutions are released through our social channels

    UNLOCK THE POWER OF DATA
     IN CONSTRUCTION

    Dive into the world of data-driven construction with this accessible guide, perfect for professionals and novices alike.
    From the basics of data management to cutting-edge trends in digital transformation, this book
    will be your comprehensive guide to using data in the construction industry.

    Related posts 

    Focus Areas

    navigate
    • ALL THE CHAPTERS IN THIS PART
    • A PRACTICAL GUIDE TO IMPLEMENTING A DATA-DRIVEN APPROACH (8)
    • CLASSIFICATION AND INTEGRATION: A COMMON LANGUAGE FOR CONSTRUCTION DATA (8)
    • DATA FLOW WITHOUT MANUAL EFFORT: WHY ETL (8)
    • DATA INFRASTRUCTURE: FROM STORAGE FORMATS TO DIGITAL REPOSITORIES (8)
    • DATA UNIFICATION AND STRUCTURING (7)
    • SYSTEMATIZATION OF REQUIREMENTS AND VALIDATION OF INFORMATION (7)
    • COST CALCULATIONS AND ESTIMATES FOR CONSTRUCTION PROJECTS (6)
    • EMERGENCE OF BIM-CONCEPTS IN THE CONSTRUCTION INDUSTRY (6)
    • MACHINE LEARNING AND PREDICTIONS (6)
    • BIG DATA AND ITS ANALYSIS (5)
    • DATA ANALYTICS AND DATA-DRIVEN DECISION-MAKING (5)
    • DATA CONVERSION INTO A STRUCTURED FORM (5)
    • DESIGN PARAMETERIZATION AND USE OF LLM FOR CAD OPERATION (5)
    • GEOMETRY IN CONSTRUCTION: FROM LINES TO CUBIC METERS (5)
    • LLM AND THEIR ROLE IN DATA PROCESSING AND BUSINESS PROCESSES (5)
    • ORCHESTRATION OF ETL AND WORKFLOWS: PRACTICAL SOLUTIONS (5)
    • SURVIVAL STRATEGIES: BUILDING COMPETITIVE ADVANTAGE (5)
    • 4D-6D and Calculation of Carbon Dioxide Emissions (4)
    • CONSTRUCTION ERP AND PMIS SYSTEMS (4)
    • COST AND SCHEDULE FORECASTING USING MACHINE LEARNING (4)
    • DATA WAREHOUSE MANAGEMENT AND CHAOS PREVENTION (4)
    • EVOLUTION OF DATA USE IN THE CONSTRUCTION INDUSTRY (4)
    • IDE WITH LLM SUPPORT AND FUTURE PROGRAMMING CHANGES (4)
    • QUANTITY TAKE-OFF AND AUTOMATIC CREATION OF ESTIMATES AND SCHEDULES (4)
    • THE DIGITAL REVOLUTION AND THE EXPLOSION OF DATA (4)
    • Uncategorized (4)
    • CLOSED PROJECT FORMATS AND INTEROPERABILITY ISSUES (3)
    • MANAGEMENT SYSTEMS IN CONSTRUCTION (3)
    • AUTOMATIC ETL CONVEYOR (PIPELINE) (2)

    Search

    Search

    057 Speed of decision making depends on data quality

    Today’s design data architecture is undergoing fundamental changes. The industry is moving away from bulky, isolated models and closed formats towards more flexible, machine-readable structures focused on analytics, integration and process automation. However, the transition...

    060 A common language of construction the role of classifiers in digital transformation

    In the context of digitalization and automation of inspection and processing processes, a special role is played by classification systems elements – a kind of “digital dictionaries” that ensure uniformity in the description and parameterization...

    061 Masterformat, OmniClass, Uniclass and CoClass the evolution of classification systems

    Historically, construction element and work classifiers have evolved in three generations, each reflecting the level of available technology and the current needs of the industry in a particular time period (Fig. 4.2-8): First generation (early...

    Don't miss the new solutions

     

     

    Linux

    macOS

    Looking for the Linux or MAC version? Send us a quick message using the button below, and we’ll guide you through the process!


    📥 Download OnePager

    Welcome to DataDrivenConstruction—where data meets innovation in the construction industry. Our One-Pager offers a concise overview of how our data-driven solutions can transform your projects, enhance efficiency, and drive sustainable growth. 

    🚀 Welcome to the future of data in construction!

    You're taking your first step into the world of open data, working with normalized, structured data—the foundation of data analytics and modern automation tools.

    By downloading, you agree to the DataDrivenConstruction terms of use 

    Stay ahead with the latest updates on converters, tools, AI, LLM
    and data analytics in construction — Subscribe now!

    🚀 Welcome to the future of data in construction!

    You're taking your first step into the world of open data, working with normalized, structured data—the foundation of data analytics and modern automation tools.

    By downloading, you agree to the DataDrivenConstruction terms of use 

    Stay ahead with the latest updates on converters, tools, AI, LLM
    and data analytics in construction — Subscribe now!

    🚀 Welcome to the future of data in construction!

    You're taking your first step into the world of open data, working with normalized, structured data—the foundation of data analytics and modern automation tools.

    By downloading, you agree to the DataDrivenConstruction terms of use 

    Stay ahead with the latest updates on converters, tools, AI, LLM
    and data analytics in construction — Subscribe now!

    🚀 Welcome to the future of data in construction!

    You're taking your first step into the world of open data, working with normalized, structured data—the foundation of data analytics and modern automation tools.

    By downloading, you agree to the DataDrivenConstruction terms of use 

    Stay ahead with the latest updates on converters, tools, AI, LLM
    and data analytics in construction — Subscribe now!

    🚀 Welcome to the future of data in construction!

    You're taking your first step into the world of open data, working with normalized, structured data—the foundation of data analytics and modern automation tools.

    By downloading, you agree to the DDC terms of use 

    🚀 Welcome to the future of data in construction!

    You're taking your first step into the world of open data, working with normalized, structured data—the foundation of data analytics and modern automation tools.

    By downloading, you agree to the DataDrivenConstruction terms of use 

    Stay ahead with the latest updates on converters, tools, AI, LLM
    and data analytics in construction — Subscribe now!

    DataDrivenConstruction offers workshops tested and practiced on global leaders in the construction industry to help your team navigate and leverage the power of data and artificial intelligence in your company's decision making.

    Reserve your spot now to rethink your
    approach to decision making!

     

    🚀 Welcome to the future of data in construction!

    By downloading, you agree to the DataDrivenConstruction terms of use 

    Stay ahead with the latest updates on converters, tools, AI, LLM
    and data analytics in construction — Subscribe now!

    Have a question or need more information? Reach out to us directly!
    Schedule a time to discuss your needs with our team.
    Tailored sessions to help your team grow — let's plan together!
    Have you attended one of our workshops, read our book, or used our solutions? Share your thoughts with us!
    Name
    Data Maturity Diagnostics

    🧰 Data-Driven Readiness Check

    This short assessment will help you identify your company's data management pain points and offer solutions to improve project efficiency. It takes only 1–2 minutes to complete and you will receive personalized recommendations tailored to your needs.

    Clean & Organized Data

    Theoretical Chapters:

    Practical Chapters:

    What You'll Find on
    DDC Solutions:

    • CAD/BIM to spreadsheet/database converters (Revit, AutoCAD, IFC, Microstation)
    • Ready-to-deploy n8n workflows for construction processes
    • ETL pipelines for data synchronization between systems
    • Customizable Python scripts for repetitive tasks
    • Intelligent data validation and error detection
    • Real-time dashboard connectors
    • Automated reporting systems

    Connect Everything

    Theoretical Chapters:

    Practical Chapters:

    What You'll Find on
    DDC Solutions:

    • CAD/BIM to spreadsheet/database converters (Revit, AutoCAD, IFC, Microstation)
    • Ready-to-deploy n8n workflows for construction processes
    • ETL pipelines for data synchronization between systems
    • Customizable Python scripts for repetitive tasks
    • Intelligent data validation and error detection
    • Real-time dashboard connectors
    • Automated reporting systems

    Add AI & LLM Brain

    Theoretical Chapters:

    Practical Chapters:

    What You'll Find on
    DDC Solutions:

    • CAD/BIM to spreadsheet/database converters (Revit, AutoCAD, IFC, Microstation)
    • Ready-to-deploy n8n workflows for construction processes
    • ETL pipelines for data synchronization between systems
    • Customizable Python scripts for repetitive tasks
    • Intelligent data validation and error detection
    • Real-time dashboard connectors
    • Automated reporting systems
    Data Warehouses and Data Lakehouse architecture
    This website uses cookies to improve your experience. By using this website you agree to our Data Protection Policy.
    Read more
    ×