Mining Imperfect Data: Dealing with Contamination and Incomplete Records

Data mining is concerned with the analysis of databases large enough that various anomalies, including outliers, incomplete data records, and more subtle phenomena such as misalignment errors, are virtually certain to be present. Mining Imperfect Data: Dealing with Contamination and Incomplete Records describes in detail a number of these problems, as well as their sources, their consequences, their detection, and their treatment. Specific strategies for data pretreatment and analytical validation that are broadly applicable are described, making them useful in conjunction with most data mining analysis methods. Examples are presented to illustrate the performance of the pretreatment and validation methods in a variety of situations; these include simulation-based examples in which correct results are known unambiguously as well as real data examples that illustrate typical cases met in practice. Mining Imperfect Data, which deals with a wider range of data anomalies than are usually treated in one book, includes a discussion of detecting anomalies through generalized sensitivity analysis (GSA), a process of identifying inconsistencies using systematic and extensive comparisons of results obtained by analysis of exchangeable datasets or subsets. The book makes extensive use of real data, both in the form of a detailed analysis of a few real datasets and various published examples. Also included is a succinct introduction to functional equations that illustrates their utility in describing various forms of qualitative behavior for useful data characterizations.