期刊名称：Bulletin of the Technical Committee on Data Engineering
出版年度：2016
卷号：39
期号：2
页码：47
出版社：IEEE Computer Society
摘要：Data curation includes the many tasks needed to ensure data maintains its value over time. Given thematurity of many data curation tasks, including data transformation and data cleaning, it is surprisingthat rigorous empirical evaluations of research ideas are so scarce. In this work, we argue that thoroughevaluation of data curation systems imposes several major obstacles that need to be overcome. First, weconsider the outputs generated by a data curation system (for example, an integrated or cleaned databaseor a set of constraints produced by a schema discovery system). To compare the results of differentsystems, measures of output quality should be agreed upon by the community and, since such measurescan be quite complex, publicly available implementations of these measures should be developed, shared,and optimized. Second, we consider the inputs to the data curation system. New techniques are neededto generate and control the metadata and data that are the input to curation systems. For a thoroughevaluation, it must be possible to control (and systematically vary) input characteristics such as thenumber of errors in data cleaning or the complexity of a schema mapping in data transformation. Finally,we consider benchmarks. Data and metadata generators must support the creation of reasonable gold-standard outputs for different curation tasks and must promote productivity by enabling the creation ofa large number of inputs with little manual effort. In this work, we overview some recent advances inaddressing these important obstacles. We argue that evaluation of curation systems is itself a fascinatingand important research area and challenges the curation community to tackle some of the remainingopen research problems.
