Dive into the Code
Every week, fresh solutions are released through our social channels
Don't miss the new solutions
Facing Issues? Reach Out!
Offer support for any troubleshooting or advanced use-cases
2 August 2023
Install Jupyter Notebook
Change language
Post's Highlights
Stay updated: news and insights
We’re Here to Help
Fresh solutions are released through our social channels
UNLOCK THE POWER OF DATA
IN CONSTRUCTION
Dive into the world of data-driven construction with this accessible guide, perfect for professionals and novices alike.
From the basics of data management to cutting-edge trends in digital transformation, this book
will be your comprehensive guide to using data in the construction industry.
Related posts
Focus Areas
- XML
- Workflow
- Work Schedules
- Word
- Windows
- VR
- Visualization Techniques
- Visualization
- Verification
- value
- Validation
- USD
- Unity
- unification
- trend
- transformation
- traceability
- tool
- Timeline
- text
- taxonomy
- task
- Tables
- table
- system
- Sustainability
- structure
- storage
- standardization
- sql
- SPARQL
- source
- software
- Sharepoint
- sensor
- scheduling
- scenario
- rvt converter
- role
- risk
- revit to xlsx
- revit to excel
- revit
- resource
- report
- record
- real-time
- RDF
- RAG
- query
- Quantification
- QTO
- python
- Project Versions
- Project Integration
- Project Collaboration
- project
- procurement
- process
- prediction
- Point Clouds
- PNG
- platform
- planning
- pipeline
- performance
- parsing
- Pandas
- orchestration
- operation
- OpenBIM
- ontology
- Omniverse
- numpy
- noBIM
- NLP
- NiFi
- neural
- network
- MySQL
- monitor
- modeling
- model
- ML
- mesh
- Merging Workflows
- manager
- management
- machine
- logistics
- logic
- log
- loading
- llm
- linking
- learning
- langchain
- label
- KPI
- JSON
- JPEG
- Interoperability
- interface
- intelligence
- Integration
- input
- information
- Image
- IFC Integration
- IFC Converter
- ifc convert
- IFC
- Graph Chart
- graph
- Geometry
- framework
- format
- Forecasting
- feedback
- extraction
- Excel
- evaluation
- etl
- Estimation
- ERP
- Environmental Impact
- entity
- engineer
- dwg to excel
- dwg export to excel
- dwg converter
- dwg
- drawings
- digitalization
- Digital Modeling
- digital
- DGN
- devops
- design
- deep
- decision
- dataset
- Database
- Data Visualization
- Data Validation
- data mining
- Data Interoperability
- Data Frames
- Data Format Conversion
- Data Extraction
- Data Export
- Data Conversion
- Data Analytics
- data
- Dashboard
- CSV
- Cost Calculations
- cost
- correlation
- converter
- Conversion
- consultant
- Construction Simulation
- Construction Projects
- Construction Models
- Construction
- Comparison
- Communication
- CO2 Footprint
- ClosedBIM
- ChatGPT
- Charting Techniques
- calculation
- cad
- Budgeting
- Bounding Box
- BlenderBIM
- BIM
- Big Data
- Bentley
- Batch Processing
- batch
- automation
- Autodesk
- AutoCAD
- Audit
- attribute
- assistant
- asset
- architecture
- application
- API
- Analytics
- analyst
- algorithm
- AI
- accuracy
- 6D
- 5D
- 4D
- 3D Scanning
- 3D Printing
Search
AnalyticsdesignDESIGN PARAMETERIZATION AND USE OF LLM FOR CAD OPERATIONentityinformationinterfacemanagerprocesstooltransformationtrend
109 Design through parameters the future of CAD and BIM
No construction project in the world has ever started in a CAD program. Before a drawing or model takes shape in CAD, it passes through the conceptualization stage (Fig. 6.4-1, stages 1-2), where the focus...
APIdesignDESIGN PARAMETERIZATION AND USE OF LLM FOR CAD OPERATIONformatinformationPandaspythonquerysqltabletext
110 Emergence of LLM in design CAD data processing processes
In addition to the development of CAD database access tools and open and simplified CAD -formats, the emergence of LLM -tools (Large Language Models) is revolutionizing the processing of design data. Whereas previously the access...
APIdesignDESIGN PARAMETERIZATION AND USE OF LLM FOR CAD OPERATIONformatinformationPandasplatformquerystructuretextWorkflow
111 Automated analysis of DWG -files with LLM and Pandas
The process of data processing from DWG -files due to the unstructured nature of the information – has always been a complex task, requiring specialized software and often manual analysis. However, with the development of...
AnalyticsarchitecturedecisiondesignDESIGN PARAMETERIZATION AND USE OF LLM FOR CAD OPERATIONextractioninformationlearningmodelingsoftwaresource
112 Next steps moving from closed formats to open data
When working with the design data of the future, it is unlikely that anyone really needs to understand the geometric kernels of proprietary tools or learn hundreds of incompatible formats containing the same information. However,...
AIAnalyticscostDATA ANALYTICS AND DATA-DRIVEN DECISION-MAKINGdecisioninformationlearningmanagerrisktexttransformation
113 Data as a resource in decision making
After the steps of collecting, structuring, cleaning and verifying the information, a coherent and analyzable data set has been formed. The previous parts of the book covered the systematization and structuring of heterogeneous sources –...
costDATA ANALYTICS AND DATA-DRIVEN DECISION-MAKINGdecisioninformationperformanceprocessreportriskstructuretexttool
114 Visualizing data the key to understanding and decision making
In today’s construction industry, where project data is characterized by complexity and multi-level structure, visualization plays a key role. Visualization of data allows project managers and engineers to visualize complex patterns and trends hidden in...
automationConstructioncostdataDATA ANALYTICS AND DATA-DRIVEN DECISION-MAKINGdigitalKPImanagementperformanceprojectrole
115 KPIs and ROI
In today’s construction industry, the management of performance indicators (KPI and ROI) and their visualization through reports and dashboards play a key role in improving productivity and project management efficiency. As in any business, in...
applicationDATA ANALYTICS AND DATA-DRIVEN DECISION-MAKINGinformationKPImonitorplanningprocesspythonsourceTimelinetool
116 Dashboards and dashboards visualization of indicators for effective management
A variety of charts and graphs are used to visualize indicators and metrics, which are typically combined into data showcases and dashboards. These dashboards provide a centralized view of the status of a project or...
AnalyticsassetcostDATA ANALYTICS AND DATA-DRIVEN DECISION-MAKINGdecisionForecastinginformationperformancereportrisktransformation
117 Analyzing data and the art of asking questions
Data interpretation is the final stage of analysis, where information makes sense and begins to “speak”. This is where the answers to the key questions are formulated: “what to do?” and “how to do?” (Fig....
APIDATA FLOW WITHOUT MANUAL EFFORT: WHY ETLdesignformatinformationPandasplatformreportsensorstructuretable
118 ETL automation lower costs and faster data handling
When key performance indicators (KPI) stop growing, despite the increase in data volumes and team size, company management inevitably comes to the realization of the need to automate processes. Sooner or later this realization becomes...
CSVDATA FLOW WITHOUT MANUAL EFFORT: WHY ETLformatinformationJSONPandaspythonsqltabletexttransformation
119 ETL Extract data collection
The first stage of the ETL process – Extract) – starts with writing code to collect data sets to be further checked and processed. To do this, we scan all the folders of the production...
CSVDATA FLOW WITHOUT MANUAL EFFORT: WHY ETLdesignevaluationformatinformationmanagerqueryrecordtabletext
120 ETL Transform application of validation and transformation rules
The Transform step is where the data is processed and transformed. This process may include correctness checking, normalization, filling in missing values and validation using automated tools According to the PwC study “Data-Driven. What Students...
DATA FLOW WITHOUT MANUAL EFFORT: WHY ETLdecisiongraphinformationmodelpythonqueryreportsourcetexttool
121 ETL Load Visualize results in charts and graphs
After completion of the Transform stage, when the data have been brought to a structured form and verified, the final stage – Load, where the data can be both loaded into target system and visualized...
122 ETL Load Automatic creation of PDF documents
At the data loading stage it is possible not only to visualize data, upload them to tables or databases, but also to automatically generate reports, including the necessary graphs, charts and key analytical indicators that...
APIcostDATA FLOW WITHOUT MANUAL EFFORT: WHY ETLformatinformationinterfacereportstructuretabletextVerification
123 ETL Load automatic document generation from FPDF
Automating reporting at the ETL stage Load is an important step in data processing, especially when the results of the analysis need to be presented in a format that is easy to communicate and understand....
CSVDATA FLOW WITHOUT MANUAL EFFORT: WHY ETLERPformatinformationinterfaceJSONqueryreporttooltransformation
124 ETL Load Reporting and loading to other systems
At the Load stage, the results were generated in the form of tables, graphs and final PDF reports prepared in accordance with the established requirements. Further it is possible to export this data into machine-readable...
AIAnalyticscostDATA FLOW WITHOUT MANUAL EFFORT: WHY ETLinformationreportscenariosourcetabletexttransformation
125 ETL with LLM Visualize data from PDF -documents
It’s time to move on to building a full-fledged ETL process that covers all key stages of data handling in a single scenario – extraction, transformation and loading. Let’s build an automated ETL-Pipeline that allows...
AnalyticsAUTOMATIC ETL CONVEYOR (PIPELINE)CSVformatpythonqueryreportsourcestructuretexttransformation
127 Pipeline -ETL data validation process with LLM
In the previous chapters on creating data requirements and automating ETL, we step-by-step broke down the process of data preparation, transformation, validation, and visualization. These activities were implemented as separate code blocks (Fig. 7.2-18 –...
AUTOMATIC ETL CONVEYOR (PIPELINE)costdesigninformationInteroperabilitymanagerNiFiPandasreportschedulingVerification
128 Pipeline-ETL verification of data and information of project elements in CAD (BIM)
Data from CAD systems and databases (BIM) are some of the most sophisticated and dynamically updated data sources in the business of construction companies. These applications not only describe the project using geometry, but also...
formatinformationinterfacemonitorORCHESTRATION OF ETL AND WORKFLOWS: PRACTICAL SOLUTIONSplatformriskschedulingstructureVerificationWorkflow
129 DAG and Apache Airflow workflow automation and orchestration
Apache Airflow is a free and open source platform, designed to automate, orchestrate and monitor workflows (ETL -conveyors). Working with large amounts of data is required every day: Download files from different sources – Extract...
architectureinformationinterfacemonitorNiFiORCHESTRATION OF ETL AND WORKFLOWS: PRACTICAL SOLUTIONSPandasplatformreportstructuretable
130 Apache Airflow practical application on ETL automation
Apache Airflow is widely used to organize complex data processing processes, allowing to build flexible ETL -conveyors. Apache Airflow can be run either through a web interface or programmatically through Python code (Fig. 7.4-2). In...
AnalyticsbatchinformationinterfaceNiFiorchestrationORCHESTRATION OF ETL AND WORKFLOWS: PRACTICAL SOLUTIONSplatformsourcetooltransformation
131 Apache NiFi for routing and data conversion
Apache NiFi is a powerful open source platform,designed to automate data flows between different systems. Originally developed in 2006 by the US National Security Agency (NSA) under the name “Niagara Files” for internal use. In...
APIarchitecturedecisioninformationinterfaceNiFiORCHESTRATION OF ETL AND WORKFLOWS: PRACTICAL SOLUTIONSplatformreportscenariotext
132 n8n Low-Code, No-Code process orchestration
n8n is an Open Source Low-Code / No-Code platform for building automated workflows, characterized by ease of use, flexibility and the ability to quickly integrate with a wide range of external services. No-Code is a...
AnalyticsarchitectureformatinformationmonitorNiFiORCHESTRATION OF ETL AND WORKFLOWS: PRACTICAL SOLUTIONSperformancepythontooltransformation
133 Next steps moving from manual operations to analytics-based solutions
Today’s construction companies operate in an environment of high uncertainty: changing material prices, delayed deliveries, labor shortages and tight project deadlines. The use of analytical dashboards, ETL -conveyors and BI systems helps companies quickly find...
dataDATA INFRASTRUCTURE: FROM STORAGE FORMATS TO DIGITAL REPOSITORIESdecisiondesignformatinformationmanagementprojectresourcestoragestructure
134 Data atoms the foundation of effective information management
Everything in the Universe consists of the smallest building blocks – atoms and molecules, and over time all living and non-living things inevitably return to this initial state. In nature, this process occurs with astonishing...
AnalyticsDATA INFRASTRUCTURE: FROM STORAGE FORMATS TO DIGITAL REPOSITORIESdwginformationperformanceprocessquerysqlstructuretabletool
135 Information storage files or data
Data warehouses allow companies to collect and combine information from different systems, creating a single center for subsequent analytics. Collected historical data enables not only deeper analysis of processes, but also the identification of patterns...
AnalyticsCSVDATA INFRASTRUCTURE: FROM STORAGE FORMATS TO DIGITAL REPOSITORIESformatJSONPandasperformancepythonstructuretabletext
136 Big Data Storage Analyzing Popular Formats and Their Effectiveness
Storage formats play a key role in the scalability, reliability, and performance of analytics infrastructure. For data analysis and processing – such as filtering, grouping, and aggregation – our examples used Pandas DataFrame – a...
architecturecostCSVDATA INFRASTRUCTURE: FROM STORAGE FORMATS TO DIGITAL REPOSITORIESformatinformationlearningPandaspythontabletext
137 Optimize storage with Apache Parquet
One of the popular formats for storing and processing big data is Apache Parquet. This format is designed specifically for columnar storage (similar to Pandas), which allows you to significantly reduce memory footprint and increase...
AnalyticsDATA INFRASTRUCTURE: FROM STORAGE FORMATS TO DIGITAL REPOSITORIESdecisionForecastingformatinformationintelligencelearningperformancestructuretransformation
138 DWH Data Warehouse data warehouses
Just as the Parquet format is optimized for efficient storage of large amounts of information, the Data Warehouse is optimized for integrating and structuring data to support analytics, forecasting and management decision making. In today’s...
AnalyticsarchitectureDATA INFRASTRUCTURE: FROM STORAGE FORMATS TO DIGITAL REPOSITORIESdecisionformatinformationlearningquerystructuretexttransformation
139 Data Lake – evolution of ETL to ELT from traditional cleaning to flexible processing
Classic DWH – data warehouses, designed to store structured data in a format optimized for analytical queries, have faced limitations in handling unstructured data and scalability. In response to these challenges, Data Lakes) have emerged,...
AnalyticsarchitectureDATA INFRASTRUCTURE: FROM STORAGE FORMATS TO DIGITAL REPOSITORIESERPformatinformationlearningperformancequerystructuretraceability
140 Data Lakehouse architecture synergy of warehouses and data lakes
To combine the best features of DWH (structured, manageable, high performance analytics) and Data Lake (scalability, handling heterogeneous data), the Data Lakehouse approach was developed. This architecture combines the flexibility of data lakes with the...
AnalyticscostDATA INFRASTRUCTURE: FROM STORAGE FORMATS TO DIGITAL REPOSITORIESdesigninformationintelligencelearningplanningprocurementsensorstructure
141 CDE, PMIS, ERP or DWH and Data Lake
Some construction and engineering companies are already using the concept of Common Data Environment (CDE) according to ISO 19650. In essence, the CDE performs the same functions as a data warehouse (DWH) in other industries:...
DATA WAREHOUSE MANAGEMENT AND CHAOS PREVENTIONdecisiondesignformatinformationlearningneuralNLPPandasquerystructure
142 Vector Databases and the Bounding Box
Vector databases are a new class of repositories that do not just store data, but allow searching by meaning, comparing objects by semantic proximity, and creating intelligent systems: from recommendations to automatic analysis and context...
AIAnalyticsDATA WAREHOUSE MANAGEMENT AND CHAOS PREVENTIONdecisioninformationlearningmachineprocessstandardizationstructuretransformation
143 Data Governance, Data Minimalism and Data Swamp
Understanding and implementing the concepts of Data Governance, Data Minimalism, and preventing Data Swamp are key to successfully managing data warehouses and delivering business value (Fig. 8.2-3). According to a study by Gartner (2017), 85%...
DATA WAREHOUSE MANAGEMENT AND CHAOS PREVENTIONdesignformatinformationlearningmonitornetworkneuralNiFiriskWorkflow
144 DataOps and VectorOps new data standards
While Data Governance is responsible for controlling and organizing data, DataOps helps ensure its accuracy, consistency and smooth flow within the company. This is especially critical for a number of business cases in construction, where...
architectureassetCSVDATA WAREHOUSE MANAGEMENT AND CHAOS PREVENTIONdecisiondesigninformationintelligencesourcestructuretransformation
145 Next steps from chaotic storage to structured storage
Traditional approaches to building data warehousing often result in the creation of disparate “silos of information” where important insights are inaccessible for analysis and decision making. Modern storage concepts, such as Data Warehouse, Data Lake...
AnalyticsapplicationBIG DATA AND ITS ANALYSISdesignForecastinginformationintelligencelearningmonitorPandasperformance
146 Big data in construction from intuition to predictability
The term “big data” does not have a strict definition. The concept originally appeared when the volume of information began to exceed the capabilities of traditional methods of its processing. Today, the volume and complexity...
AnalyticsBIG DATA AND ITS ANALYSIScorrelationcostdecisionformatinformationmonitorperformanceriskstructure
147 Questioning the feasibility of big data correlation, statistics and data sampling
Traditionally, construction was based on subjective hypotheses and personal experience. Engineers assumed – with a certain degree of probability – how the material would behave, what loads the structure would withstand and how long the...
148 Big data analyzing data from San Francisco’s million building permit dataset
Working with open datasets provides a unique opportunity to put into practice the principles discussed in previous chapters: judicious feature selection, representative sampling, visualization, and critical analysis. In this chapter, we will explore how complex...
149 Example of big data based on CAD data (BIM)
In the following example we will analyze a large dataset using data from different CAD tools (BIM). To collect and create the large dataset, a specialized automated web crawler (script) was used, configured to automatically...
