Impossible Data Warehouse Situations Solutions from the Experts

by Adelman, Sid; Bischoff, Joyce; Dyché, Jill; Hackney, Douglas; Ivoghli, Sean; Kelley, Chuck; Marco, David; Moss, Larissa T.; Rehm, Clay

• ISBN13: 9780201760330
• ISBN10: 0201760339
• Edition: 1st
• Format: Paperback
• Copyright: 2002-10-01
• Publisher: Addison-Wesley Professional

Summary

This book takes a unique approach to the problems faced by data warehouse professionals. The author and the contributors have gathered over 90 situations that they have been asked about in their seminars and presentations, that they have faced in their own work, and that have been submitted to the very popular "Ask the Experts" forum at DMReview. These are all real situations, but they have been disguised to protect the guilty. Topics covered include staffing, budgeting, security, vendors, architecture, and data quality. Each of the "impossible" situations will have one or more solutions contributed by the expert panel. Their different answers and viewpoints, especially when they disagree with one another, provide enlightening reading, as well as useful ideas. This approach should appeal to a broad range of people involved in all aspects of Data Warehouses.

Author Biography

Sid Adelman is founder of Sid Adelman & Associates, an organization specializing in planning and implementing data warehouses. He presents regularly at data warehouse conferences and conducts a Data Warehouse Project Management seminar. Sid is also a founding member of the BIAlliance. He jointly developed a methodology that provides a master plan for implementing data warehouses. He wrote Data Warehouse Project Management (Addison-Wesley, 2000) with Larissa Moss.

Joyce Bischoff, president of Bischoff Consulting, Inc., is an internationally recognized consultant, writer, and lecturer specializing in all aspects of data warehousing, database design, and design methodologies. She has been involved in planning, designing, implementing, and performing design reviews of data warehouses in more than 50 companies in the credit card, chemical, pharmaceutical, insurance, financial, oil refining, publishing, and hospital industries. She is the lead author of the book Data Warehouse: Practical Advice from the Experts 1997, which brings together opinions from 20 contributing authors, and a member of the expert panel for the monthly column "Ask the Experts" at http://www.dmreview.com. She is the author of numerous articles and frequently presents at data warehousing conferences all over the world. She may be reached at [e-mail].

Jill Dyché is a partner with Baseline Consulting Group, a specialty consulting firm focusing on the delivery of business intelligence solutions across industries. Since 1985 she has been working with Fortune 1000 companies worldwide to help align strategic technology initiatives with corporate business objectives. Jill is a frequent speaker at technology and marketing conferences, and her articles have been featured in a variety of publications: Information Week, Oracle magazine, Teradata Review, Telephony Magazine, The Washington Times, and The Chicago Tribune.

Douglas Hackney is President of Enterprise Group, Ltd. and is a monthly columnist for Data Management Review. He has contributed to Computer World and speaks regularly at industry events, including DCI's Data Warehousing Conference, Bill Inmon's Data Warehousing Conference, and the Data Warehouse Institute.

Sean Ivoghli is the founder and president of Digital Symmetry, Inc., formerly the Data Warehouse Consulting Group, a consulting firm that specializes in providing end-to-end data warehousing, business intelligence, and data/application integration solutions. He has over 12 years of experience in full life-cycle data warehouse and information systems development, and he provides expert consulting services in data warehouse design, development, project management, and information management strategies. Mr. Ivoghli is the coauthor of Compass, a comprehensive data warehousing methodology that offers multiple tracks for developing scalable, flexible, and high-performance data warehousing and data mart solutions in a cost-effective manner. He can be reached at [e-mail] and at [e-mail].

Chuck Kelley is an internationally known expert in database technology. He has over 25 years of experience in designing and implementing operational/production systems and data warehouses. Mr. Kelley has worked in some facet of the implementation process of over 45 data warehouses. Mr. Kelley teaches seminars on SQL, Database Internals, Implementing the Data Warehouse, Designing and Implementing the Star Schema from Your Operational System, and other database and data warehousing topics. He has been a speaker at Database World, Client/Server World, UniForum, COMDEX, Rdb Conference, DECUS Symposia, and many data warehouse conferences. He coauthored a book with W. H. Inmon 1994 on data warehouses and is a member of the panel on the "Ask the Experts" column at http://www.dmreview.com. Mr. Kelley has been published in many trade magazines on database technology, data warehousing, and enterprise data strategies. He can be reached at [e-mail] or at his Web site at http://www.excellenceindata.com.

David Marco is an internationally recognized expert in the fields of data warehousing, e-business, XML, and business intelligence, and he is the world's foremost authority on meta data. He authored the book Building and Managing the Meta Data Repository 2000b. Mr. Marco also serves as the editor of Real-World Decision Support, an electronic newsletter focusing on business intelligence and e-business topics (http://www.EWSolutions.com/newsletter.asp). Mr. Marco has published over 80 articles and is a columnist for Application Development Trends, Database Trends, and DM Review magazines. Mr. Marco has been selected as a judge for the 1998-2002 DM Review World-Class Solutions, 2002 TDWI Pioneering Solutions, and 1999-2002 Microsoft Industry Solutions awards. In addition, Mr. Marco was a finalist for the 2000 DAMA Individual IT Achievement award.

Mr. Marco has presented over 70 keynote addresses and courses at all the major data warehousing and meta data repository conferences throughout the world. He also cosponsors with Pennsylvania State University a certified series of courses on data warehousing and business intelligence, and he teaches data warehousing at the University of Chicago. Mr. Marco is the founder and President of the Chicago-headquartered Enterprise Warehousing Solutions, Inc. (EWS), a strategic partner and systems integrator dedicated to providing clients with best-in-class business intelligence solutions using data warehousing and meta data repository technologies. EWS provides strategic consulting services and full life-cycle implementation services for Global 2000 corporations and government institutions and has been awarded a government GSA Schedule. In addition, EWS presents the Marco Master's Series, which is the industry's first and only certified meta data training course. Visit http://www.EWSolutions.com for more information about EWS and the Marco Master's Series. Mr. Marco may be reached at [e-mail].

Larissa Moss is founder and president of Method Focus, Inc., a consulting firm specializing in business intelligence and data warehousing. She is a frequent lecturer and speaker at conferences in the United States, Europe, and Asia on data warehousing, project management, development methodologies, and organizational and cultural issues. Her articles on these topics are regularly published in magazines such as DM Review and Journal of Data Warehousing. She is coauthor of Data Warehouse Project Management (Addison-Wesley, 2000) and Impossible Data Warehouse Situations (Addison-Wesley, 2003). She is a senior consultant at the Cutter Consortium and one of the authors of their Business Intelligence Executive Reports.

Clay Rehm, CCP, PMP, is president of Rehm Technology (http://www.rehmtech.com), a consulting firm specializing in data integration solutions. He provides hands-on expertise in project leadership and management, assessments, methodologies, data modeling, database design, meta data and systems analysis, design, and development. He has worked in multiple platforms, and his experience spans operational and data warehouse environments. He is a technical book editor and author, instructor, and speaker. He serves on the panel of the DM Review "Ask the Experts" Web site (http://www.dmreview.com) and on the Carroll College Business Advisory Council.

With a passion for data architecture, he is well versed in DB2, SQL Server, and Access. He is Access 2000 Microsoft Office User Specialist (MOUS) Certified, a Certified Computing Professional (CCP), and a certified Project Management Professional (PMP). He holds a bachelor's of science degree in computer science from Carroll College and is currently working on his master's degree in software engineering.

He is a member of the Data Management Association (DAMA), the Professional Society of SQL Server (PASS), and the Project Management Institute (PMI). He can be reached at [e-mail].

0201760339AB09262002

Table of Contents

Preface	xi
Credits	xvii
PART I IMPOSSIBLE MANAGEMENT SITUATIONS	1	(196)
Management Issues	3	(24)
Overview	3	(2)
The Data Warehouse Has a Record of Failure	5	(4)
IT Is Unresponsive	9	(2)
Management Constantly Changes	11	(3)
IT Is the Assassin	14	(2)
The Pilot Must Be Perfect	16	(3)
User Departments Don't Want to Share Data	19	(3)
Senior Management Doesn't Know What the Data Warehouse Team Does	22	(5)
Changing Requirements and Objectives	27	(24)
Overview	27	(2)
The Operational System Is Changing	29	(3)
The Source System Constantly Changes	32	(5)
The Data Warehouse Vision Has Become Blurred	37	(3)
The Objectives Are Misunderstood	40	(3)
The Prototype Becomes Production	43	(5)
Management Doesn't Recognize the Success of the Data Warehouse Project	48	(3)
Justification and Budget	51	(24)
Overview	51	(2)
User Productivity Justification Is Not Allowed	53	(4)
How Can the Company Identify Infrastructure Benefits?	57	(2)
Does a Retailer Need a Data Warehouse?	59	(3)
How Can Costs Be Allocated Fairly?	62	(4)
Historical Data Must Be Justified	66	(3)
No Money Exists for a Prototype	69	(6)
Organization and Staffing	75	(34)
Overview	75	(3)
To Whom Should the Data Warehouse Team Report?	78	(3)
The Organization Uses Matrix Management	81	(4)
The Project Has No Consistent Business Sponsor	85	(3)
Should a Line of Business Build Its Own Data Mart?	88	(5)
The Project Has No Dedicated Staff	93	(6)
The Project Manager Has Baggage	99	(3)
No One Wants to Work for the Company	102	(2)
The Organization Is Not Ready for a Data Warehouse	104	(5)
User Issues	109	(22)
Overview	109	(3)
The Users Want It Now	112	(4)
The Business Does Not Support the Project	116	(3)
Web-Based Implementation Doesn't Impress the Users	119	(2)
Management Rejects Multidimensional Tools as Being Too Complex	121	(2)
The Users Have High Data Quality Expectations	123	(4)
The Users Don't Know What They Want	127	(4)
Team Issues	131	(44)
Overview	131	(3)
A Heat-Seeking Employee Threatens the Project	134	(3)
Management Assigned Dysfunctional Team Members to the Data Warehouse Project	137	(4)
Management Requires Team Consensus	141	(3)
Prima Donnas on the Team Create Dissension	144	(3)
Team Members Aren't Honest about Progress on Assignments	147	(3)
A Consultant Offers to Come to the Rescue	150	(3)
The Consultants Are Running the Show	153	(4)
The Contractors Have Fled	157	(4)
Knowledge Transfer Is Not Happening	161	(4)
How Can Data Warehouse Managers Best Use Consultants?	165	(6)
Management Wants to Outsource the Data Warehouse Activities	171	(4)
Project Planning and Scheduling	175	(22)
Overview	175	(2)
Management Requires Substantiation of Estimates	177	(3)
IT Management Sets Unrealistic Deadlines	180	(4)
The Sponsor Changes the Scope But Doesn't Want to Change the Schedule	184	(4)
The Users Want the First Data Warehouse Delivery to Include Everything	188	(4)
The Project Manager Severely Underestimates the Schedule	192	(5)
PART II IMPOSSIBLE TECHNICAL SITUATIONS	197	(148)
Data Warehouse Standards	199	(18)
Overview	199	(1)
The Organization Has No Experience with Methodologies	200	(2)
Database Administration Standards Are Inappropriate for the Data Warehouse	202	(4)
The Employees Misuse Data Warehouse Terminology	206	(4)
It's All Data Mining	210	(2)
A Multinational Company Needs to Build a Business Intelligence Environment	212	(5)
Tools and Vendors	217	(22)
Overview	217	(1)
What Are the Best Practices for Writing a Request for Proposals?	218	(2)
The Users Don't Like the Query and Reporting Tool	220	(3)
OO Is the Answer (But What's the Question?)	223	(2)
IT Has Already Chosen the Tool	225	(3)
Will the Tools Perform Well?	228	(3)
The Vendor Has Undue Influence	231	(2)
The Rejected Vendor Doesn't Understand ``No''	233	(2)
The Vendor's Acquiring Company Provides Poor Support	235	(4)
Security	239	(10)
Overview	239	(2)
The Data Warehouse Has No Security Plan	241	(2)
Responsibility for Security Must Be Established	243	(2)
Where Should a New Security Administrator Start?	245	(4)
Data Quality	249	(26)
Overview	249	(2)
How Should Sampling Be Applied to Data Quality?	251	(5)
Redundant Data Needs to Be Eliminated	256	(3)
Management Underestimated the Amount of Dirty Data	259	(4)
Management Doesn't Recognize the Value of Data Quality	263	(3)
The Data Warehouse Architect Is Obsessed with Data Quality	266	(3)
The ETL Process Partially Fails	269	(3)
Source Data Errors Cause Massive Updates	272	(3)
Integration	275	(28)
Overview	275	(1)
Multiple Source Systems Require Major Data Integration	276	(5)
The Enterprise Model Is Delaying Progress	281	(6)
Should a Company Decentralize?	287	(2)
The Business Sponsor Wants Real-Time Customer Updates	289	(3)
The Company Doesn't Want Stovepipe Systems	292	(5)
Reports from the Data Warehouse and the Operational Systems Don't Match	297	(2)
Should the Data Warehouse Team Fix an Inadequate Operational System?	299	(4)
Data Warehouse Architecture	303	(30)
Overview	303	(2)
The Data Warehouse Architecture Is Inadequate	305	(4)
Stovepipes Are Impeding Integration	309	(9)
Should Backdated Transactions Change Values in the Data Warehouse?	318	(1)
A Click-Stream Data Warehouse Will Be Huge	319	(1)
Time-Variant Analysis Requires Special Designs	320	(1)
Management Wants to Develop a Data Warehouse Simultaneously with a New Operational System	321	(4)
The Data Waterhouse Gets Assigned the Role of a Reporting System	325	(1)
Meta Data Needs to Be Integrated Across Multiple Products	326	(4)
How Can UPC Code Changes Be Reconciled?	330	(3)
Performance	333	(12)
Overview	333	(1)
The Software Does Not Perform Properly	334	(2)
The Data Warehouse Grows Faster Than the Source Data	336	(4)
Loading the Fact Table Takes Too Long	340	(5)
Appendix A Data Warehouse Glossary	345	(24)
Appendix B Colloquialism Glossary	369	(14)
Bibliography	383	(8)
Experts' Bios	391	(6)
Index	397

Excerpts

In the seminars, presentations, and classes we teach on data warehousing, we are often subjected to what appear to be "impossible situations." Likewise, in the DM Review "Ask the Experts" forum (http://www.dmreview.com) we are confronted with questions that, at first glance, appear to have no answers or solutions. However, they do have solutions, and that's what this book is all about. We took the 91 impossible situations discussed in this book from our classes, from the DM Review "Ask the Experts" forum, from data warehouse consultants, and from colleagues in the field who have experienced these situations. These are all real situations, but we have disguised them to protect the authors as well as to protect the organizations experiencing the situations from the attendant shame and humiliation. As a side note, reviewers of specific situations in our manuscript were quick to say, "I know what company this describes," and they were almost always wrong. The Purpose of This Book There is no reason that each organization, as it begins and continues to develop data warehouse projects, must wrestle with many of the very difficult situations that have confounded other organizations. The same impossible situations continue to raise their ugly heads, often with surprisingly little relation to the industry, the size of the organization, or the organizational structure. In this book we let you know you are not alone and your problems are not unique. We also offer hope to the perplexed who see no obvious solutions to their problems. Some of the situations should resonate with those of you planning to enhance your data warehouse by adding new data, additional users, or new applications. It may be that the impossible situation has not yet emerged, but you definitely see it just around the bend. After reading this book, you should be able to avoid the situation rather than needing to fix it after it has developed. Who Should Read This Book Every stakeholder, data warehouse architect, data warehouse project manager, and user liaison responsible for any portion of a data warehouse faces the challenges identified in these pages. These people are looking for solutions to situations that, at first, appear to have no possible answer. This book does not present an introduction to data warehousing. To benefit most from this book, you should have some level of familiarity with data warehousing through practical experience, conferences, or previous reading of data warehouse texts. This book is also not geared to any primary topic such as meta data or data quality; instead, it covers a broad range of areas. The reference section lists both introductory and more advanced suggested reading material. User liaisons and managers may wish to read only Part I (Impossible Management Situations). All others will want to read both Part I and Part II (Impossible Technical Situations). How This Book Is Organized As mentioned, the first part of the book deals with managerial situations and the second part with technical situations. The order within these sections is very roughly the order in which projects are developed and situations are encountered, but each chapter stands alone without depending on those that precede it. You can read the book from front to back, but more likely you will be drawn to the chapters describing the problems that cause you the most pain. For example, if you struggle with data quality issues, Chapter 11, Data Quality, is the place to start. Each subsection of a chapter presents a different impossible situation related to the chapter's topic, followed by the experts' suggested solutions (presented in the alphabetical order of the experts' last names). The Data Warehouse Glossary at the back of the book clarifies some terms and helps keep you from going down the wrong path. Misunderstanding the terminology