Data Virtualization for Business Intelligence Systems

by van der Lans

• ISBN13: 9780123944252
• ISBN10: 0123944252
• Format: Paperback
• Copyright: 2012-07-25
• Publisher: Elsevier Science

Summary

Data virtualization can help you accomplish your goals with more flexibility and agility. Learn what it is and how and why it should be used with Data Virtualization for Business Intelligence Systems. In this book, expert author Rick van der Lans explains how data virtualization servers work, what techniques to use to optimize access to various data sources and how these products can be applied in different projects. You'll learn the difference is between this new form of data integration and older forms, such as ETL and replication, and gain a clear understanding of how data virtualization really works. Data Virtualization for Business Intelligence Systems outlines the advantages and disadvantages of data virtualization and illustrates how data virtualization should be applied in data warehouse environments. You'll come away with a comprehensive understanding of how data virtualization will make data warehouse environments more flexible and how it make developing operational BI applications easier. Van der Lans also describes the relationship between data virtualization and related topics, such as master data management, governance, and information management, so you come away with a big-picture understanding as well as all the practical know-how you need to virtualize your data. First independent book on data virtualization that explains in a product-independent way how data virtualization technology works. Illustrates concepts using examples developed with commercially available products. Shows you how to solve common data integration challenges such as data quality, system interference, and overall performance by following practical guidelines on using data virtualization. Apply data virtualization right away with three chapters full of practical implementation guidance. Understand the big picture of data virtualization and its relationship with data governance and information management.

Author Biography

Rick van der Lans Independent BI analyst Managing Director R20/Consultancy B.V.

Table of Contents

Foreword	p. xiii
Preface	p. xv
About the Author	p. xix
Introduction to Data Virtualization	p. 1
Introduction	p. 1
The World of Business Intelligence Is Changing	p. 1
Introduction to Virtualization	p. 3
What Is Data Virtualization?	p. 4
Data Virtualization and Related Concepts	p. 5
Data Virtualization versus Encapsulation and Information Hiding	p. 5
Data Virtualization versus Abstraction	p. 6
Data Virtualization versus Data Federation	p. 7
Data Virtualization versus Data Integration	p. 8
Data Virtualization versus Enterprise Information Integration	p. 9
Definition of Data Virtualization	p. 9
Technical Advantages of Data Virtualization	p. 10
Different Implementations of Data Virtualization	p. 14
Overview of Data Virtualization Servers	p. 14
Open versus Closed Data Virtualization Servers	p. 15
Other Forms of Data Integration	p. 16
The Modules of a Data Virtualization Server	p. 18
The History of Data Virtualization	p. 19
The Sample Database: World Class Movies	p. 22
Structure of This Book	p. 25
Business Intelligence and Data Warehousing	p. 27
Introduction	p. 27
What Is Business Intelligence?	p. 27
Management Levels and Decision Making	p. 28
Business Intelligence Systems	p. 29
The Data Stores of a Business Intelligence System	p. 30
The Data Warehouse	p. 30
The Data Marts	p. 34
The Data Staging Area	p. 35
The Operational Data Store	p. 37
The Personal Data Stores	p. 38
A Comparison of the Different Types of Data Stores	p. 38
Normalized Schemas, Star Schemas, and Snowflake Schemas	p. 39
Normalized Schemas	p. 40
Denormalized Schemas	p. 40
Star Schemas	p. 41
Snowflake Schemas	p. 43
Data Transformation with Extract Transform Load, Extract Load Transform, and Replication	p. 44
Extract Transform Load	p. 44
Extract Load Transform	p. 45
Replication	p. 46
Overview of Business Intelligence Architectures	p. 47
New Forms of Reporting and Analytics	p. 48
Operational Reporting and Analytics	p. 48
Deep and Big Data Analytics	p. 49
Self-Service Reporting and Analytics	p. 49
Unrestricted Ad-Hoc Analysis	p. 50
360-Degree Reporting	p. 51
Exploratory Analysis	p. 51
Text-Based Analysis	p. 52
Disadvantages of Classic Business Intelligence Systems	p. 53
Summary	p. 56
Data Virtualization Server: The Building Blocks	p. 59
Introduction	p. 59
The High-Level Architecture of a Data Virtualization Server	p. 59
Importing Source Tables and Defining Wrappers	p. 60
Defining Virtual Tables and Mappings	p. 62
Examples of Virtual Tables and Mappings	p. 66
Virtual Tables and Data Modeling	p. 76
Nesting Virtual Tables and Shared Specifications	p. 77
Importing Nonrelational Data	p. 79
XML and JSON Documents	p. 79
Web Services	p. 84
Spreadsheets	p. 86
NoSQL Databases	p. 86
Multidimensional Cubes and MDX	p. 89
Semistructured Data	p. 92
Unstructured Data	p. 95
Publishing Virtual Tables	p. 96
The Internal Data Model	p. 101
Updatable Virtual Tables and Transaction Management	p. 106
Data Virtualization Server: Management and Security	p. 109
Introduction	p. 109
Impact and Lineage Analysis	p. 109
Synchronization of Source Tables, Wrapper Tables, and Virtual Tables	p. 110
Security of Data: Authentication and Authorization	p. 112
Monitoring, Management, and Administration	p. 114
Data Virtualization Server: Caching of Virtual Tables	p. 119
Introduction	p. 119
The Cache of a Virtual Table	p. 119
When to Use Caching	p. 120
Caches versus Data Marts	p. 122
Where Is the Cache Kept?	p. 122
Refreshing Caches	p. 123
Full Refreshing, Incremental Refreshing, and Live Refreshing	p. 124
Online Refreshing and Offline Refreshing	p. 125
Cache Replication	p. 126
Data Virtualization Server: Query Optimization Techniques	p. 127
Introduction	p. 127
A Refresher Course on Query Optimization	p. 128
The Ten Stages of Query Processing by a Data Virtualization Server	p. 132
The Intelligence Level of the Data Stores	p. 134
Optimization through Query Substitution	p. 134
Optimization through Pushdown	p. 137
Optimization through Query Expansion (Query Injection)	p. 139
Optimization through Ship Joins	p. 140
Optimization through Sort-Merge Joins	p. 141
Optimization by Caching	p. 142
Optimization and Statistical Data	p. 142
Optimization through Hints	p. 143
Optimization through SQL Override	p. 143
Explaining the Processing Strategy	p. 145
Deploying Data Virtualization in Business Intelligence Systems	p. 147
Introduction	p. 147
A Business Intelligence System Based on Data Virtualization	p. 147
Advantages of Deploying Data Virtualization	p. 148
Disadvantages of Deploying Data Virtualization	p. 151
Strategies for Adopting Data Virtualization	p. 151
Strategy 1: Introducing Data Virtualization in an Existing Business Intelligence System	p. 152
Strategy 2: Developing a New Business Intelligence System with Data Virtualization	p. 157
Strategy 3: Developing a New Business Intelligence System Combining Source and Transformed Data	p. 161
Application Areas of Data Virtualization	p. 163
Unified Data Access	p. 163
Virtual Data Mart	p. 163
Virtual Data Warehouse-Based on Data Marts	p. 165
Virtual Data Warehouse-Based on Production Databases	p. 165
Extended Data Warehouse	p. 167
Operational Reporting and Analytics	p. 167
Operational Data Warehouse	p. 168
Virtual Corporate Data Warehouse	p. 169
Self-Service Reporting and Analytics	p. 170
Virtual Sandbox	p. 171
Prototyping	p. 171
Analyzing Semistructured and Unstructured Data	p. 172
Disposable Reports	p. 173
Extending Business Intelligence Systems with External Users	p. 173
Myths on Data Virtualization	p. 174
Design Guidelines for Data Virtualization	p. 177
Introduction	p. 177
Incorrect Data and Data Quality	p. 177
Different Forms of Incorrect Data	p. 178
Integrity Rules and Incorrect Data	p. 179
Filtering, Flagging, and Restoring Incorrect Data	p. 179
Examples of Filtering Incorrect Data	p. 180
Examples of Flagging Incorrect Data	p. 184
Examples of Restoring Misspelled Data	p. 186
Complex and Irregular Data Structures	p. 188
Codes without Names	p. 188
Inconsistent Key Values	p. 190
Repeating Groups	p. 192
Recursive Data Structures	p. 192
Implementing Transformations in Wrappers or Mappings	p. 197
Analyzing Incorrect Data	p. 197
Different Users and Different Definitions	p. 198
Time Inconsistency of Data	p. 199
Data Stores and Data Transmission	p. 200
Retrieving Data from Production Systems	p. 202
Joining Historical and Operational Data	p. 203
Dealing with Organizational Changes	p. 204
Archiving Data	p. 205
Data Virtualization and Service-Oriented Architecture	p. 207
Introduction	p. 207
Service-Oriented Architectures in a Nutshell	p. 207
Basic Services, Composite Services, Business Process Services, and Data Services	p. 209
Developing Data Services with a Data Virtualization Server	p. 211
Developing Composite Services with a Data Virtualization Server	p. 213
Services and the Internal Data Model	p. 215
Data Virtualization and Master Data Management	p. 217
Introduction	p. 217
Data Is a Critical Asset for Every Organization	p. 217
The Need for a 360-Degree View of Business Objects	p. 219
What Is Master Data?	p. 219
What Is Master Data Management?	p. 221
A Master Data Management System	p. 222
Master Data Management for Integrating Data	p. 224
Integrating Master Data Management and Data Virtualization	p. 224
Data Virtualization, Information Management, and Data Governance	p. 231
Introduction	p. 231
Impact of Data Virtualization on Information Modeling and Database Design	p. 231
Impact of Data Virtualization on Data Profiling	p. 234
Impact of Data Virtualization on Data Cleansing	p. 239
Impact of Data Virtualization on Data Governance	p. 239
The Data Delivery Platform-A New Architecture for Business Intelligence Systems	p. 243
Introduction	p. 243
The Data Delivery Platform in a Nutshell	p. 243
The Definition of the Data Delivery Platform	p. 244
The Data Delivery Platform and Other Business Intelligence Architectures	p. 245
The Requirements of the Data Delivery Platform	p. 247
The Data Delivery Platform versus Data Virtualization	p. 249
Explanation of the Name	p. 250
A Personal Note	p. 251
The Future of Data Virtualization	p. 253
Introduction	p. 253
The Future of Data Virtualization According to Rick F. van der Lans	p. 254
New and Enhanced Query Optimization Techniques	p. 254
Exploiting New Hardware Technology	p. 255
Extending the Design Module	p. 256
Data Quality Features	p. 258
Support for the Push-Model for Data Access	p. 258
Blending of Data Virtualization, Extract Transform Load, Extract Load Transform, and Replication	p. 259
The Future of Data Virtualization According to David Besemer, CTO of Composite Software	p. 260
The Empowered Consumer Gains Ubiquitous Data Access	p. 261
IT's Back Office Becomes the Cloud	p. 261
Data Virtualization of the Future Is a Global Data Fabric	p. 261
Conclusion	p. 262
The Future of Data Virtualization According to Alberto Pan, CTO of Denodo Technologies	p. 262
The Future of Data Virtualization According to James Markarian, CTO of Informatica Corporation	p. 264
How to Maximize Return on Data with Data Virtualization	p. 265
Beyond Looking Under the Hood	p. 266
Bibliography	p. 267
Index	p. 269
Table of Contents provided by Ingram. All Rights Reserved.