Filtering the Web to Feed Data Warehouses

by Abramowicz, Witold; Kalczynski, Pawel; Wecel, Krzysztof

• ISBN13: 9781852335793
• ISBN10: 1852335793
• Format: Hardcover
• Copyright: 2002-10-01
• Publisher: Springer-Nature New York Inc

Summary

Information is a key factor in business today, and data warehousing has become a major activity in the development and management of information systems to support the proper flow of information. Unfortunately, the majority of information systems are based on structured information stored in organizational databases, which means that the company is isolated from the business environment by concentrating on their internal data sources only. It is therefore vital that organizations take advantage of external business information, which can be retrieved from Internet services and mechanically organized within the existing information structures. Such a continuously extending integrated collection of documents and data could facilitate decision-making processes in the organization. Filtering the Web to Feed Data Warehousesdiscusses areas such as: - how to use data warehouse for filtering Web content - how to retrieve relevant information from diverse sources on the Web - how to handle the time aspect - how to mechanically establish links among data warehouse structures and documents filtered from external sources - how to use collected information to increase corporate knowledge and gives a comprehensive example, illustrating the idea of supplying data warehouses with relevant information filtered from the Web.

Table of Contents

Introduction	1	(7)
Information Systems	1	(1)
Information Filtering Systems	2	(1)
Database Systems	2	(3)
Transactional Systems	3	(1)
Analytical Systems	4	(1)
Organization of this Book	5	(2)
Data Warehouse: Corporate Knowledge Repository	7	(34)
Introduction	7	(1)
Data Warehouse Definition and Features	7	(4)
Definition	7	(1)
Metadata	8	(1)
Characteristic Features of Data in the Data Warehouse	9	(2)
Data Warehouse System	11	(12)
Architecture of the Data Warehouse System	11	(5)
Metadata Structures	16	(6)
Data Warehouse Products	22	(1)
Deploying Data Warehouse in the Organization	23	(3)
Data Warehouse Life Cycle	23	(1)
Analysis and Research	24	(1)
Identifying Architecture and Demands	24	(1)
Design and Development	25	(1)
Implementation and On-going Administration	25	(1)
Knowledge Management in Data Warehouses	26	(3)
Knowledge Management	26	(1)
Knowledge in Terms of Data Warehousing	27	(1)
Knowledge Discovery in Data Warehouses	28	(1)
Significance of Business Metadata	29	(1)
Evolution of the Data Warehouse	29	(7)
Criticism of the Traditional Data Warehouse	29	(1)
Virtual Data Warehouse	30	(1)
Information Data Superstore	30	(1)
Exploration Warehouse	30	(2)
Internet/Intranet Data Warehouse	32	(1)
Web Farming	33	(2)
Enterprise Information Portals	35	(1)
Chapter Summary	36	(1)
References	37	(4)
Knowledge Representation Standards	41	(34)
Introduction	41	(4)
Basic Concepts	41	(1)
Metadata Representation	42	(1)
Metadata Interoperability	43	(1)
Theory of Metadata	43	(2)
Markup Languages	45	(3)
Background	45	(1)
XML Document	46	(1)
Document Presentation	47	(1)
Document Linking	47	(1)
Programming Interfaces	47	(1)
Dublin Core	48	(1)
Dublin Core Metadata Elements	48	(1)
Dublin Core in HTML	49	(1)
Warwick Framework	49	(2)
Meta Content Framework	51	(4)
Origins of MCF	51	(1)
Conceptual Building Blocks of MCF	51	(1)
XML Syntax	52	(2)
Directed Labelled Graph Formalism	54	(1)
Resource Description Framework	55	(10)
Background	55	(1)
Formal RDF Data Model	56	(2)
The RDF Syntax	58	(4)
RDF Schema	62	(3)
Common Warehouse Metamodel	65	(6)
History of OMG Projects	65	(1)
Objectives of the CWM	66	(1)
Metadata Architecture	66	(3)
CWM Elements	69	(1)
Conclusions for CWM	70	(1)
Chapter Summary	71	(1)
References	72	(3)
Information Filtering and Retrieval From Web Sources	75	(30)
Introduction	75	(4)
Document, Information, Knowledge	75	(1)
Indexing	76	(1)
Hypertext	77	(1)
Information on the Web	77	(1)
Constraints of this Book	78	(1)
Information Retrieval Systems	79	(9)
Definitions	79	(2)
Information Retrieval System Architectures and Models	81	(6)
Sample Information Retrieval Systems	87	(1)
Information Filtering Systems	88	(5)
Filtering Versus Retrieval	88	(1)
Information Filtering Models and Architectures	89	(3)
Sample Filtering Systems	92	(1)
Internet Sources of Business Information	93	(4)
Business View on Internet Information Sources	93	(1)
General Characteristics of Business Information Sources	94	(1)
Information Overflow	95	(2)
Filtering the Web to Feed Business Information Systems	97	(4)
Problems with Web Filtering and Retrieval	97	(1)
New Information Filtering System Model Proposal	98	(2)
Transparent Filtering and Retrieval	100	(1)
Chapter Summary	101	(1)
References	101	(4)
Enhanced Data Warehouse	105	(50)
Introduction	105	(1)
Justification of the Need for Integration	106	(4)
Value of Knowledge	106	(1)
Attention Economy	107	(1)
Content Management and Lifecycle of Content	108	(2)
Example of Integration: Metadata and Data	110	(1)
Preliminary Vision of the System	110	(3)
Analytical Point of View	111	(1)
Trends	111	(1)
Goals of the System	111	(1)
User Requirements Towards the Information Retrieval Systems	112	(1)
Software Agents	113	(8)
Introduction	113	(1)
Intelligent Agents or Just Agents?	113	(1)
Software Agents or Just Agents?	113	(1)
Possible Applications of Agents	114	(1)
Definitions of Software Agents	115	(2)
Agent Properties	117	(1)
Classifications of Software Agents	118	(2)
Agent-based Systems and Multi-agent Systems	120	(1)
Proposed Solution: enhanced Data Warehouse	121	(10)
Introduction	121	(1)
Overview of the eDW System	122	(2)
Assumptions for the eDW System	124	(2)
Components	126	(1)
Agent-based System Architecture	127	(1)
Loging Server	128	(1)
Profiling Server	128	(1)
Source Agent Server	129	(1)
Document Server	129	(1)
Properties of eDW Agents	130	(1)
Formal Model of eDW	131	(15)
CSL: The Extension of the Organizational Metamodel	131	(5)
Time Consistency among Documents and Warehouse Data	136	(3)
DWL: The Intranet Collection of Relevant Documents for the Data Warehouse	139	(2)
enhanced Data Warehouse Report: The Final Product of the eDW System	141	(3)
Formal Definitions of eDW Agents	144	(2)
System Implementation	146	(5)
Programming Environment	146	(1)
System Control Centre	147	(1)
Communication	148	(1)
Status	148	(1)
Configuration File	149	(1)
Logging Server	150	(1)
Chapter Summary	151	(1)
References	152	(3)
Profiling	155	(24)
Introduction	155	(1)
Personalization and Data Warehouse Profiles	155	(6)
Classification of Information	155	(1)
Personalization	156	(1)
Personalization in Data Warehouses and its Aspects	156	(1)
Overview of Profile Creation	157	(2)
Data Warehouse Profiles	159	(2)
Algorithms Specification	161	(5)
Algorithm for Creating Warehouse Profiles	161	(3)
Computational Complexity	164	(1)
Thesauri	165	(1)
Profiling Server	166	(9)
Basic Assumptions	166	(1)
Profiling Agent	166	(2)
User Interface in Profiling Application	168	(2)
Sample Results	170	(5)
Chapter Summary	175	(1)
References	175	(4)
Source Exploitation	179	(24)
Introduction	179	(1)
Sample Business Content Providers	179	(7)
Sample Business Gateways	179	(2)
Sample Business Search Engines	181	(1)
Sample Business Portals and Vortals	181	(3)
Sample Business Online Databases	184	(2)
Information Ants to Filter Information from Internet Sources	186	(7)
Introduction	186	(1)
Ant Colony Optimization	186	(1)
Environment for Information Ants	187	(2)
Information Ants to Filter Information from the Web	189	(1)
Experiment with Ant-like Navigation	190	(2)
Advantages and Drawbacks of the Proposed Solution	192	(1)
Indexing Parser	193	(5)
Parsing Web Documents	193	(4)
Indexing Web Documents	197	(1)
Transparent Filtering in the eDW System	198	(3)
Building Warehouse Profiles	198	(1)
Registering Sources	199	(1)
Source Exploration	200	(1)
Source Penetration	201	(1)
Chapter Summary	201	(1)
References	202	(1)
Building Data Warehouse Library	203	(28)
Introduction	203	(2)
Characteristics of WWW: A Dream of Non-volatile Internet	203	(1)
Digital Libraries	204	(1)
Time Indexing	205	(9)
Finite State Automaton	205	(2)
Time Indexer	207	(5)
Trapezoidal Time Indices	212	(1)
Simple Overlap Measure for Trapezoidal Time Indices	212	(2)
Experiment with Time Indexing	214	(11)
Experiment with Time Indexing Real-World Documents	214	(9)
Conclusions for the eDW System	223	(2)
Future Trends: Multimedia Indexing	225	(4)
Introduction	225	(1)
Filtering Web Documents	225	(1)
Neural Nets for Image Categorization	225	(1)
The Proposed Solution--Perceptron Categorization Tree	226	(2)
Advantages and Drawbacks	228	(1)
01>Application for eDW	228	(1)
Chapter Summary	229	(1)
References	229	(2)
Context Queries and Enhanced Reports	231	(30)
Introduction	231	(1)
Context Queries	231	(4)
Definition of Context	231	(1)
Justification of Transparent Retrieval	232	(1)
Elements of Context	232	(1)
Conceptual Similarity Measure	233	(1)
Simple Temporal Similarity Measure	233	(1)
Parameterized Temporal Similarity Measure	234	(1)
Pertinence	235	(1)
enhanced Report	235	(2)
User Interface in Accessing the Information	236	(1)
How enhanced Report is Created	236	(1)
Reporting Application	237	(15)
Basic Assumptions	237	(2)
Description of the Algorithms	239	(2)
Context Query Agent	241	(1)
Computational Complexity	242	(4)
User Interface in Reporting Application	246	(2)
Results	248	(4)
Histograms: The Helpful Tool for Analysis	252	(6)
Non-parameterized Histogram	253	(1)
Past-oriented Analysis	253	(1)
Future-oriented Analysis	254	(1)
General Documents	255	(1)
Detailed Documents	255	(1)
Compact and Dispersed Histograms	256	(2)
Chapter Summary	258	(1)
References	258	(3)
Conclusions	261	(4)
Concluding Remarks	261	(1)
Improvements	262	(1)
Open Issues and Future Work	262	(3)
Index	265