期刊名称：International Journal of Japan Association for Management Systems
印刷版ISSN：1884-2089
电子版ISSN：2188-2460
出版年度：2018
卷号：10
期号：1
页码：53-58
DOI：10.14790/ijams.10.53
语种：English
出版社：日本経営システム学会
摘要：Recent years, have witnessed rapid developments in the digital archive, communication, and sensing technologies. Manufacturing industries, governments, etc., can now utilize data generated either from the public or Internet of Things devices for decision making purposes. Thus, big data analysis techniques are becoming increasingly important. However, contamination of the source data with noise often leads to erroneous analysis results. Data cleansing has been one of the main challenges in carrying out data analysis because it is difficult to remove noise data from the real data especially when there is no welldefined metrics for distinguishing between the noise data and the real data. Hence, the purpose of this study is to propose an effective method for cleaning the data before they are processed. This method involves generating the metadata and simulation model according to the purpose of application and finally conducting the simulation. We validated the experimental data cleansing method using the probe person survey data. As a result, even if the necessary data items are lacking they are can be compensated for by setting a formula to estimate them. The results of our experiments show that this method gives relatively accurate results.
关键词：Data Cleansing;Big Data;Probe Person Survey Data
