Data Quality

by Wang, Richard Y.; Ziad, Mostapha; Lee, Yang W.

• ISBN13: 9780792372158
• ISBN10: 0792372158
• Format: Hardcover
• Copyright: 2000-10-01
• Publisher: Kluwer Academic Pub

Summary

Data Quality provides an exposé of research and practice in the data quality field for technically oriented readers. It is based on the research conducted at the MIT Total Data Quality Management (TDQM) program and work from other leading research institutions. This book is intended primarily for researchers, practitioners, educators and graduate students in the fields of Computer Science, Information Technology, and other interdisciplinary areas. It forms a theoretical foundation that is both rigorous and relevant for dealing with advanced issues related to data quality. Written with the goal to provide an overview of the cumulated research results from the MIT TDQM research perspective as it relates to database research, this book is an excellent introduction to Ph.D. who wish to further pursue their research in the data quality area. It is also an excellent theoretical introduction to IT professionals who wish to gain insight into theoretical results in the technically-oriented data quality area, and apply some of the key concepts to their practice.

Author Biography

Mostapha Ziad is an assistant professor at the Sawyer School of Management, Suffolk University, Boston.

Table of Contents

Preface	xiii
Introduction	1	(18)
Fundamental Concepts	2	(4)
Data vs. Information	2	(1)
Product vs. Information manufacturing	3	(1)
The Information Manufacturing System	4	(1)
Information Quality is a Multi-Dimensional Concept	4	(1)
The TDQM Cycle	5	(1)
A Framework for TDQM	6	(8)
Define IP	7	(5)
Measure IP	12	(1)
Analyze IP	13	(1)
Improve IP	14	(1)
Summary	14	(1)
Book Organization	14	(5)
Extending the Relational Model to Capture Data Quality Attributes	19	(18)
The Polygen Model	20	(5)
Architecture	21	(1)
Polygen Data Structure	22	(1)
Polygen Algebra	23	(2)
The Attribute-based Model	25	(3)
Data structure	26	(1)
Data manipulation	27	(1)
QI-Compatibility and QIV-Equal	28	(1)
Quality Indicator Algebra	29	(3)
Data Integrity	31	(1)
Conclusion	32	(5)
Extending the ER Model to Represent Data Quality Requirements	37	(12)
Motivating Example	38	(1)
Quality Requirements Identification	39	(1)
Requirements Modeling	40	(1)
Modeling Data Quality Requirements	41	(6)
Data Quality Dimension Entity	42	(1)
Data Quality Measure Entity	42	(1)
Attribute Gerund Representation	43	(1)
Conceptual Design Example	44	(3)
Concluding Remarks	47	(2)
Automating Data Quality Judgment	49	(14)
Introduction	50	(1)
Quality Indicators and Quality Parameters	51	(1)
Overview	51	(1)
Related Work	52	(1)
Data Quality Reasoner	52	(5)
Representation of Local Dominance Relationships	53	(1)
Reasoning Component of DQR	54	(3)
First-Order Data Quality Reasoner	57	(4)
The Q-Reduction Algorithm	58	(2)
Q-Merge Algorithm	60	(1)
Conclusion	61	(2)
Developing a Data Quality Algebra	63	(16)
Assumptions	64	(1)
Notation	65	(1)
Definitions	65	(4)
A Data Quality Algebra	69	(1)
Accuracy Estimation of Data Derived by Selection	69	(1)
Worst case when error distribution is non-uniform	70	(1)
Best case when error distribution is non-uniform	71	(1)
Accuracy Estimation of Data Derived by Projection	71	(1)
Worst case when error distribution is non-uniform	72	(1)
Best case when error distribution is non-uniform	73	(6)
Illustrative Example	73	(3)
Conclusion	76	(3)
The MIT Context Interchange Project	79	(14)
Integrating Heterogeneous Sources and Uses	80	(1)
The Information Integration Challenge	81	(3)
Information Extraction and Dissemination Challenges	82	(1)
Information Interpretation Challenges	82	(2)
Overview of the Context Interchange Approach	84	(7)
Context Interchange Architecture	84	(1)
Wrapping	85	(1)
Mediation	86	(5)
Conclusion	91	(2)
The European Union Data Warehouse Quality Project	93	(26)
Introduction	93	(2)
The DWQ Project	95	(5)
DWQ Project Structure	95	(1)
DWQ Objectives	96	(1)
Data Warehouse Components	97	(1)
The Linkage to Data Quality	98	(2)
DWQ Architectural Framework	100	(3)
The Conceptual Perspective	101	(1)
The Logical Perspective	101	(1)
The Physical Perspective	102	(1)
Relationships between the Perspectives	103	(1)
Research Issues	103	(5)
Rich Data Warehouse Architecture Modeling Languages	104	(1)
Data Extraction and Reconciliation	105	(1)
Data Aggregation and Customization	105	(1)
Query Optimization	106	(1)
Update Propagation	107	(1)
Schema and Instance Evolution	108	(1)
Quantitative Design Optimization	108	(1)
Overview of the DWQ demonstration	108	(6)
Source Integration (Steps 1,2)	109	(4)
Aggregation and OLAP Query Generation (Steps 3, 4)	113	(1)
Design Optimization and Data Reconciliation (Steps 5, 6)	113	(1)
Summary and Conclusions	114	(5)
The Purdue University Data Quality Project	119	(20)
Introduction	120	(1)
Background	121	(1)
Related Work	122	(2)
Searching Process	122	(1)
Matching Process	123	(1)
Methodology	124	(6)
Data Pre-processing	125	(1)
Sorting, Sampling & the Sorted Neighborhood Approach	126	(1)
Clustering and Classification	127	(1)
Decision Tree Transformation and Simplification	128	(2)
Experiments and Observations	130	(5)
Conclusions	135	(4)
Conclusion	139	(10)
Follow-up Research	140	(3)
Other Research	143	(1)
Future Directions	143	(6)
Bibliography	149	(14)
Index	163