A Descriptive Study on Data Profiling: Focusing on Attribute Value Quality Index

In the era of the Fourth Industrial Revolution, companies are focusing on securing artificial intelligence
(AI) technology to enhance their competitiveness via machine learning, which is the core technology
of AI, and to allow computers to acquire a high level of quality data through self-learning. Securing
good-quality big data is becoming a very important asset for companies to enhance their
competitiveness. The volume of digital information is expected to grow rapidly around the world,
reaching 90 zettabytes (ZB) by 2020. It is very meaningful to present the value quality index on each
data attribute as it may be desirable to evaluate the data quality for a user with regard to whether the
data is suitable for use from the user’s point of view. As a result, this allows the user to determine
whether they would take the data or not based on the data quality index. In this study, we propose a
quality index calculation model with structured and unstructured data, as well as a calculation method
for the attribute value quality index (AVQI) and the structured data value quality index (SDVQI).
SDVQI was calculated using the attribute value quality index (AVQI). As unstructured data are
increasing, it is expected that the calculation of the unstructured data quality index will be helpful to
determine the usefulness of unstructured data. In the future, we plan on completing the data profiling
model using neural network and statistical analysis (DPNS).