136 Big Data Storage Analyzing Popular Formats and Their Effectiveness

10 June 2025

138 DWH Data Warehouse data warehouses

10 June 2025

CHAPTERS 1-12

HOW INFORMATION HAS EVOLVED IN CONSTRUCTION

CHAPTERS 13-22

HOW THE CONSTRUCTION BUSINESS IS DROWNING IN DATA CHAOS

137 Optimize storage with Apache Parquet

One of the popular formats for storing and processing big data is Apache Parquet. This format is designed specifically for columnar storage (similar to Pandas), which allows you to significantly reduce memory footprint and increase the speed of analytical queries. Unlike traditional formats such as CSV and XLSX, Parquet supports native compression and is optimized for big data systems including Spark, Hadoop, and cloud storage.

Key features of Parquet include support for data compression and encoding, which significantly reduces storage size and speeds up data read operations by working directly with the desired columns rather than all rows of data.

For an illustrative example of how easy it is to get the necessary code to convert data to Apache Parquet, let’s use the LLM.

Send a text request to LLM chat (CHATGP, LlaMa, Mistral DeepSeek, Grok, Claude, QWEN):
Write code to save data from Pandas DataFrame to Apache Parquet. ⏎

LLM’s response:

Fig. 8.1-4 Transferring Dataframe data from RAM to a storage-efficient format Apache Parquet using a few lines of Python.

Next example: let’s simulate the ETL process with data stored in Parquet format to filter projects by a certain value of one of the attributes “cost_million_usd” (Fig.‎ 8.1-4).

As the chat continues, send a text request to LLM:
Write code where we want to filter the data in a table and save only those projects (table rows) from Apache Parquet data, whose cost (parameter cost_million_usd) exceeds 150 million dollars. ⏎

LLM’s response:

Fig. 8.1-5 The ETL process when working with data in the Apache Parquet format looks the same as with other structured formats.

Using the Parquet format (in relation to XLSX, CSV, etc.) significantly reduces the amount of information stored and speeds up search operations. This makes it excellent for both storing and analyzing data. Parquet integrates with various processing systems, providing efficient access in hybrid architectures.

However, an efficient storage format is only one element of a complete data experience. To create a sustainable and scalable environment, a well-designed data management architecture is required. This is exactly what DWH (Data Warehouse) class systems do. They provide aggregation of data from heterogeneous sources, transparency of business processes and the possibility of complex analysis using BI-tools and machine learning algorithms.

Change language

Post's Highlights

Stay updated: news and insights

Focus Areas

architecture Construction cost data digital EVOLUTION OF DATA USE IN THE CONSTRUCTION INDUSTRY information intelligence logic

001 The birth of the data era in construction

About 10,000 years ago, in the Neolithic era, mankind made a revolutionary transition in its development, abandoning the nomadic lifestyle in favor of sedentary life, which led to the appearance of the first primitive buildings...

application architecture cost decision design EVOLUTION OF DATA USE IN THE CONSTRUCTION INDUSTRY information planning process record tool

002 From clay and papyrus to digital technology

The first documentary evidence in construction dates back to the period of pyramid building, around 3000-4000 BC (“Papyrus, 3rd century B.C. Language is Greek,” 2024). Since then, the keeping of written records has facilitated and...

Analytics decision digitalization EVOLUTION OF DATA USE IN THE CONSTRUCTION INDUSTRY Forecasting information planning process structure tool transformation

003 Process as a tool for data-driven experience

At the heart of any process is the transformation of past experience into a tool for planning the future. Experience in the modern sense is a structured set of data, the analysis of which allows...

Analytics cost decision design digitalization EVOLUTION OF DATA USE IN THE CONSTRUCTION INDUSTRY format information learning text Verification

004 Digitalization of construction process information

For millennia, the amount of information recorded in construction has barely changed, but it has grown rapidly in recent decades (Fig. 1.1-5). According to the PwC study® “Managed Data. What Students Need to Succeed in...

architecture asset cost digitalization information MANAGEMENT SYSTEMS IN CONSTRUCTION platform report risk source structure

005 Digital revolution and the emergence of modular MRPERP-systems

The era of modern digital data storage and processing began with the advent of magnetic tape in the 1950s, which opened up the possibility of storing and utilizing large amounts of information. The next breakthrough...

architecture asset cost design information MANAGEMENT SYSTEMS IN CONSTRUCTION manager network performance planning procurement

006 Data management systems from data mining to business challenges

Today’s companies are faced with the need to integrate multiple data management systems. Selecting data management systems, managing these systems well, and integrating disparate data sources is becoming critical to business performance. In the mid-2020s,...

analyst Analytics decision design ERP information MANAGEMENT SYSTEMS IN CONSTRUCTION network procurement structure tool

007 Corporate mycelium how data connects to business processes

The process of integrating data into applications and databases relies on the aggregation of information from a variety of sources, including different departments and specialists (Fig. 1.2-4). Specialists search for relevant data, process it, and...

Construction data digital digitalization information intelligence learning machine role THE DIGITAL REVOLUTION AND THE EXPLOSION OF DATA value

008 The beginning of the data volume boom as an evolutionary wave

The construction industry is experiencing an unprecedented information explosion. If we think of business as a knowledge tree (Fig.‎ 1.2-5) fed by data, the current stage of digitalization can be compared to the rapid growth...

AI Construction cost data digital drawings information management storage THE DIGITAL REVOLUTION AND THE EXPLOSION OF DATA transformation

009 The amount of data generated in a modern company

In the last two years, 90% of all existing data in the world has been created (B. Marr, “How much data do we create every day? The Mind-Blowing Stats Everyone Should Read,” 2018). As of...

AI architecture cost decision digitalization ERP information intelligence process THE DIGITAL REVOLUTION AND THE EXPLOSION OF DATA trend

010 Cost of data storage the economic aspect

In recent years, more and more companies are outsourcing data storage to cloud services. For example, if a company hosts half of its data in the cloud, at an average price of $0.015 per gigabyte...

Audit automation Construction decision engineer information resource role structure system THE DIGITAL REVOLUTION AND THE EXPLOSION OF DATA

012 Next steps from data theory to practical change

The evolution of data in construction is a journey from clay tablets to modern modular platforms. The challenge today is not to collect information, but to create a framework that turns disparate and diverse data...

application asset cost DATA UNIFICATION AND STRUCTURING decision information performance process text transformation trend

034 Filling systems with data in the construction industry

Whether it is large corporations or medium-sized companies, specialists are daily engaged in filling program systems and databases with various interfaces with multiformat information (Fig.‎ 3.2-1), which, with the help of managers, must interact with...

analyst DATA UNIFICATION AND STRUCTURING decision format information Interoperability learning manager report structure text

035 Data transformation the critical foundation of modern business analysis

Today, most companies are facing a paradox: about 80% of their daily processes still rely on classic structured data – familiar Excel spreadsheets and relational databases (RDBMS) (М. Shacklett, “Structured and unstructured data: Key differences,”...

Analytics architecture CSV DATA UNIFICATION AND STRUCTURING entity format information ontology record structure table

036 Data models relationships in data and relationships between elements

Data in information systems are organized in different ways – depending on the tasks and requirements for storing, processing and transmitting information. The key difference between the types of data models, the form in which...

API architecture asset DATA UNIFICATION AND STRUCTURING digitalization format information interface Interoperability structure table

037 Proprietary formats and their impact on digital processes

One of the key challenges faced by construction companies during digitalization is limited access to data. This makes it difficult to integrate systems, reduces the quality of information and complicates the organization of efficient processes....

AI DATA UNIFICATION AND STRUCTURING design digitalization format information Interoperability performance report source Workflow

038 Open formats are changing the approach to digitalization

The construction industry was one of the last to address the problem of closed and proprietary data. Unlike other sectors of the economy, digitalization has been slow to develop here. The reasons for this include...

Analytics architecture cost DATA UNIFICATION AND STRUCTURING design information monitor orchestration report structure transformation

039 Paradigm Shift Open Source as the End of the Era of Software Vendor Dominance

The construction industry is undergoing a shift that cannot be monetized in the usual way. The concept of data-driven, data-centric approach and the use of Open Source tools is leading to a rethinking of the...

AI Analytics architecture DATA UNIFICATION AND STRUCTURING decision information report source structure transformation unification

040 Structured open data the foundation of digital transformation

While in past decades business sustainability was largely determined by the choice of software solutions and dependence on specific vendors, in today’s digital economy the key factor is data quality and the ability to work...

AI information intelligence LLM AND THEIR ROLE IN DATA PROCESSING AND BUSINESS PROCESSES Pandas python source sql table text tool

041 LLM chat rooms ChatGPT, LlaMa, Mistral, Claude, DeepSeek, QWEN, Grok for automating data processing processes

The emergence of Large Language Models (LLMs) was a natural extension of the movement towards structured open data and the Open Source philosophy. When data becomes organized, accessible and machine-readable, the next step is a...

decision design format information learning LLM AND THEIR ROLE IN DATA PROCESSING AND BUSINESS PROCESSES monitor neural query structure text

042 Large Language Models LLM how it works

Big language models (ChatGPT, LlaMa, Mistral, Claude, DeepSeek, QWEN, Grok) are neural networks trained on huge amounts of textual data from the Internet, books, articles and other sources. Their main task is to understand the...

AI architecture cost design Forecasting information LLM AND THEIR ROLE IN DATA PROCESSING AND BUSINESS PROCESSES performance report source text

043 Utilizing local LLMs for sensitive company data

The appearance of the first chat-LLMs in 2022 marked a new stage in the development of artificial intelligence. However, immediately after the widespread adoption of these models, a legitimate question arose: how secure is it...

accuracy AI application information interface LLM AND THEIR ROLE IN DATA PROCESSING AND BUSINESS PROCESSES performance query risk source text

044 Full control of AI in the company and how to deploy your own LLM

Modern tools allow companies to deploy a large language model (LLM) locally in just a few hours. This gives complete control over data and infrastructure, eliminating dependence on external cloud services and minimizing the risk...

AI architecture assistant information interface langchain LLM AND THEIR ROLE IN DATA PROCESSING AND BUSINESS PROCESSES python query RAG text

045 RAG Intelligent LLM -assistants with access to corporate data

The next stage in the evolution of LLM application in business is the integration of models with actual real-time corporate data. This approach is called RAG (Retrieval-Augmented Generation) – Retrieval-Augmented Generation. In this architecture, the...

decision format IDE WITH LLM SUPPORT AND FUTURE PROGRAMMING CHANGES information interface learning network NiFi platform RAG structure

046 Choosing an IDE from LLM experiments to business solutions

When diving into the world of automation, data analysis, and artificial intelligence – especially when working with large language models (LLMs) – it is critical to choose the right integrated development environment (IDE). This IDE...

Analytics CSV decision format IDE WITH LLM SUPPORT AND FUTURE PROGRAMMING CHANGES information learning neural Pandas platform query

048 Python Pandas an indispensable tool for working with data

Pandas occupies a special place in the world of data analysis and automation. It is one of the most popular and widely used libraries of the Python programming language(“Python Packages Download Stats,” 2024), designed to...

Analytics CSV format IDE WITH LLM SUPPORT AND FUTURE PROGRAMMING CHANGES information label Pandas query structure table text

049 DataFrame universal tabular data format

DataFrame is the central structure in the Pandas library, which is a two-dimensional table (Fig. 3.4-6) where rows correspond to individual objects or records and columns correspond to their characteristics, parameters, or categories. This structure...

asset Audit decision framework IDE WITH LLM SUPPORT AND FUTURE PROGRAMMING CHANGES information Pandas process python structure transformation

050 Next steps building a sustainable data framework

In this part, we reviewed the key types of data used in the construction industry, got acquainted with different formats of their storage and analyzed the role of modern tools, including LLM and IDEs, in...

analyst CSV DATA CONVERSION INTO A STRUCTURED FORM decision format information JSON learning platform structure text

051 Learning how to turn documents, PDF, pictures and texts into structured formats

In the era of the data-driven economy, data is becoming the basis for decision-making rather than an obstacle. Instead of constantly adapting information to each new system and its formats, companies are increasingly striving to...

AI CSV DATA CONVERSION INTO A STRUCTURED FORM feedback format learning Pandas query source table text

052 Example of converting a PDF -document into a table

One of the most common tasks in construction projects is to process specifications in PDF format. To demonstrate the transition from unstructured data to a structured format, let’s consider a practical example: extracting a table...

CSV DATA CONVERSION INTO A STRUCTURED FORM decision format information manager Pandas query structure table taxonomy

054 Converting text data into a structured form

In addition to PDF documents with tables (Fig.‎ 4.1-2) and scanned versions of tabular forms (Fig.‎ 4.1-5), a significant part of information in project documentation is presented in text form. It can be both coherent...

API cost DATA CONVERSION INTO A STRUCTURED FORM design dwg format information Pandas source table transformation

055 Translation of CAD data (BIM) into a structured form

Structuring and categorizing CAD data (BIM) is more challenging because data stored from CAD (BIM) databases are almost always in closed or complex parametric formats, often combining geometric data elements (semi-structured) and metainformation elements (semi-structured...

Analytics architecture DATA CONVERSION INTO A STRUCTURED FORM design digitalization entity format information Interoperability platform structure

056 CAD solution vendors move to structured data

From 2024, the design and construction industry is undergoing a significant technological shift in the use and processing of data. Instead of free access to design data, CAD -system vendors are focusing on promoting the...

automation CLASSIFICATION AND INTEGRATION: A COMMON LANGUAGE FOR CONSTRUCTION DATA Construction decision design digitalization information Integration project real-time Verification

057 Speed of decision making depends on data quality

Today’s design data architecture is undergoing fundamental changes. The industry is moving away from bulky, isolated models and closed formats towards more flexible, machine-readable structures focused on analytics, integration and process automation. However, the transition...

accuracy CLASSIFICATION AND INTEGRATION: A COMMON LANGUAGE FOR CONSTRUCTION DATA decision design format information intelligence process structure unification Verification

058 Data standardization and integration

Effective data management requires a clear standardization strategy. Only with clear requirements for data structure and quality can data validation be automated, manual operations reduced and informed decision making accelerated at all stages of a...

CLASSIFICATION AND INTEGRATION: A COMMON LANGUAGE FOR CONSTRUCTION DATA cost design entity format information Interoperability report scheduling structure Verification

136 Big Data Storage Analyzing Popular Formats and Their Effectiveness

138 DWH Data Warehouse data warehouses

CHAPTERS 1-12

CHAPTERS 23-50

CHAPTERS 51-72

CHAPTERS 73-93

CHAPTERS 94-112

CHAPTERS 113-132

CHAPTERS 133-145

CHAPTERS 146-160

137 Optimize storage with Apache Parquet

Leave a Reply Cancel reply

Focus Areas

Search

Don't miss the new solutions

Looking for the Linux or MAC version? Send us a quick message using the button below, and we’ll guide you through the process!

📥 Download OnePager

Reserve your spot now to rethink your approach to decision making!

🧰 Data-Driven Readiness Check

🚀 Goals and Pain Points

Build your automation pipeline

Understand and organize your data

Automate your key process

Define a digital strategy

Move from CAD (BIM) to databases and analytics

Combine BIM, ERP and Excel

Convince leadership to invest in data

📘 What to Read in Data-Driven Construction Guidebook

Chapters 1.2, 4.1–4.3 – Technologies, Data Conversion, Structuring, Modeling:

Centralized vs fragmented data

Principles of data structure

Roles of Excel, DWH, and databases

Chapters 5.2, 7.2 – QTO Automation, ETL with Python:

Data filtering and grouping

Automating QTO and quantity takeoff

Python scripts and ETL logic

Chapter 10.2 – Roadmap for Digital Transformation:

Strategic stages of digital change

Organizational setup

Prioritization and execution paths

Chapters 4.1, 8.1–8.2 – From CAD (BIM) to Storage & Analytics:

Translating Revit/IFC to structured tables

BIM as a database

Building analytical backends

Chapters 7.3, 10.2 – Building ETL Pipelines + Strategic Integration:

Combining Excel, BIM, ERP

Automating flows between tools

Connecting scattered data sources

Chapters 7.3, 7.4 – ETL Pipelines and Orchestration (Airflow, n8n):

Building pipelines

Scheduling jobs

Using tools like Airflow or n8n to control the flow

Chapters 2.1, 10.1 – Fragmentation, ROI, Survival Strategy:

Hidden costs of bad data

Risk of inaction

ROI of data initiatives

Convincing stakeholders

🎯 DDC Workshop That Solves Your Puzzle

Module 1 – Data Automation and Workflows in Construction:Overview of data sourcesExcel vs systemsTypical data flows in constructionFoundational data logic

Module 3 – Automated Data Processing Workflow:Setting up ETL workflowsCAD/BIM extractionAutomation in Excel/PDF reporting

Module 8 – Converting Unstructured CAD into Structured Formats From IFC/Revit to tablesGeometric vs semantic dataTools for parsing and transforming CAD models

Module 13 – Key Stages of Transformation Transformation roadmapChange managementRoles and responsibilitiesKPIs and success metrics

Module 8 – Integrating Diverse Data Systems and FormatsExcel, ERP, BIM integrationData connection and file exchangeStructuring hybrid pipelines

Module 7 – Automating Data Quality Assurance Processes Rules and checksDashboardsReport validationAutomated exception handling

Module 10 – Challenges of Digitalization in the Industry How to justify investment in dataStakeholder concernsROI examplesFailure risks

💬 Individual Consultation – What We'll Discuss

Audit of your data landscape

We'll review how data is stored and shared in your company and identify key improvement areas.

Select a process for automation

We'll pick one process in your company that can be automated and outline a step-by-step plan.

Strategic roadmap planning

Together we’ll map your digital transformation priorities and build a realistic roadmap.

CAD (BIM) - IFC/Revit model review

We'll review your Revit/IFC/DWG data and show how to convert it into clean, structured datasets.

Mapping integrations across tools

We’ll identify your main data sources and define how they could be connected into one workflow.

Plan a pilot pipeline (PoC)

We'll plan a pilot pipeline: where to start, what tools to use, and what benefits to expect.

ROI and stakeholder alignment

We’ll discuss how to justify data investments with ROI examples and stakeholder alignment tips.

Reserve your spot now to rethink your
approach to decision making!

Module 1 – Data Automation and Workflows in Construction:
Overview of data sources
Excel vs systems
Typical data flows in construction
Foundational data logic

Module 3 – Automated Data Processing Workflow:
Setting up ETL workflows
CAD/BIM extraction
Automation in Excel/PDF reporting

Module 8 – Converting Unstructured CAD into Structured Formats
From IFC/Revit to tables
Geometric vs semantic data
Tools for parsing and transforming CAD models

Module 13 – Key Stages of Transformation
Transformation roadmap
Change management
Roles and responsibilities
KPIs and success metrics

Module 8 – Integrating Diverse Data Systems and Formats
Excel, ERP, BIM integration
Data connection and file exchange
Structuring hybrid pipelines

Module 7 – Automating Data Quality Assurance Processes
Rules and checks
Dashboards
Report validation
Automated exception handling

Module 10 – Challenges of Digitalization in the Industry
How to justify investment in data
Stakeholder concerns
ROI examples
Failure risks