Unstructured data in the construction industry represents information that is not organized according to a specific pre-defined structure. One of the key sources of such data are scanned documents, images and videos captured on construction sites: this data is critical for monitoring construction progress, monitoring safety compliance and analyzing work processes.
Internet of Things (IoT) devices that collect information on a variety of attributes such as temperature, humidity, noise levels and equipment location also provide a significant amount of unstructured data (being partly also semi-structured data). This data is critical for monitoring building operations and equipment performance.
It is the automated collection, processing, analysis of unstructured data and its integration into construction management processes that opens up opportunities to reduce the human factor in many processes that today involve a large number of managers and support staff.
The process of transforming unstructured data into useful information is one of the most complex and labor-intensive processes in data handling
Text data is also a type of unstructured data. However, we will separate this type into a separate category, given its widespread use and importance in the business processes of construction companies.
In data management, images represent one of the most common forms of unstructured data. Images are full of complex information because of their varied content, colours, textures and the need for specialised processing to extract useful information.
The difficulty in using images as a data source is the lack of structure. They do not convey meaning in a direct, easily quantifiable way that a computer can immediately understand or process, as a spreadsheet or database table does. To convert this unstructured data into a structured form, it is necessary to use software libraries capable of interpreting the visual information it contains.
Transfer of scanned documents or images into structured and semi-structured formats is possible using special OCR tools
One of the main technologies used in working with image data is Optical Character Recognition, or OCR, which is a process that converts different types of documents, such as scanned paper, PDF files or images, into editable and searchable data.
OCR is versatile enough to be integrated into a variety of workflows, offering a powerful tool for developers looking to incorporate recognition capabilities into their applications.
Let's ask ChatGPT to write us some sample code to get data from a scanned or photographed table in a structured way.
❏ Text request to ChatGPT:
Write minimal code that allows to convert a JPEG image that has a table, translate to a DataFrame table. ⏎
➤ ChatGPT Answer:
Using the Pandas library, we convert the extracted text we convert an image or photo table into a structured table view
The code written by ChatGPT uses the pytesseract library (Tesseract for Python) to convert an image into text using OCR (optical character recognition) and the pandas library to convert that text into a structured form, i.e. a DataFrame.
Tesseract is one of the leading open source OCR engines, originally developed by HP and now supported by Google. Known for its accuracy and ability to recognise over 100 languages, Tesseract is widely used for OCR tasks.
The conversion process usually involves pre-processing to improve image quality, after which various algorithms are applied for pattern detection, feature extraction or object recognition. As a result, unstructured visual information is converted into structured data.
