Exploration of Visual Data

by Zhou, Xiang Sean; Rui, Yong; Huang, Thomas S.

• ISBN13: 9781402075698
• ISBN10: 1402075693
• Format: Hardcover
• Copyright: 2003-08-01
• Publisher: Kluwer Academic Pub

Summary

Exploration of Visual Data presents latest research efforts in the area of content-based exploration of image and video data. The main objective is to bridge the semantic gap between high-level concepts in the human mind and low-level features extractable by the machines. The two key issues emphasized are "content-awareness" and "user-in-the-loop". The authors provide a comprehensive review on algorithms for visual feature extraction based on color, texture, shape, and structure, and techniques for incorporating such information to aid browsing, exploration, search, and streaming of image and video data. They also discuss issues related to the mixed use of textual and low-level visual features to facilitate more effective access of multimedia data. To bridge the semantic gap, significant recent research efforts have also been put on learning during user interactions, which is also known as "relevance feedback". The difficulty and challenge also come from the personalized information need of each user and a small amount of feedbacks the machine could obtain through real-time user interaction. The authors present and discuss several recently proposed classification and learning techniques that are specifically designed for this problem, with kernel- and boosting-based approaches for nonlinear extensions. Exploration of Visual Data provides state-of-the-art materials on the topics of content-based description of visual data, content-based low-bitrate video streaming, and latest asymmetric and nonlinear relevance feedback algorithms, which to date are unpublished. Exploration of Visual Data will be of interest to researchers, practitioners, and graduate-level students in the areas of multimedia information systems, multimedia databases, computer vision, machine learning.

Author Biography

Xiang Sean Zhou: Siemens Corporation Princeton, NJ, U.S.A. Yong Rui Microsoft Research Redmond, WA, U.S.A. Thomas S. Huang University of Illinois at Urbana-Champaign Urbana, IL, U.S.A.

Table of Contents

1. INTRODUCTION	1	(4)
1.1 Challenges	1	(1)
1.2 Research Scope	2	(1)
1.3 State-of-the-Art	2	(2)
1.3.1 Visual information extraction and representation	2	(1)
1.3.2 Learning from user interactions	3	(1)
1.3.3 Temporal analysis and segmentation of video	3	(1)
1.3.4 Content-sensitive low bit-rate video streaming	3	(1)
1.4 Outline of Book	4	(1)
2. OVERVIEW OF VISUAL INFORMATION REPRESENTATION	5	(10)
2.1 Color	6	(1)
2.2 Texture	7	(1)
2.3 Shape	8	(2)
2.4 Spatial Layout	10	(1)
2.5 Interest Points	11	(1)
2.6 Image Segmentation	12	(1)
2.7 Summary	13	(2)
3. EDGE-BASED STRUCTURAL FEATURES	15	(24)
3.1 Visual Feature Representation	15	(2)
3.1.1 The scope of our research	16	(1)
3.1.2 Background	16	(1)
3.1.3 The quest for structure	16	(1)
3.2 Edge-Based Structural Features	17	(7)
3.2.1 The water-filling algorithm	18	(2)
3.2.2 Edge feature extraction	20	(4)
3.2.3 Scale invariance and cross-scale matching	24	(1)
3.3 Experiments and Analysis	24	(15)
3.3.1 City/building and landscape images	25	(2)
3.3.2 Images with clear structure: Birds and airplanes in the sky	27	(2)
3.3.3 High-level concepts: Horses, tigers, and cars	29	(6)
3.3.4 Medical image retrieval	35	(1)
3.3.5 Cross-scale image matching	35	(1)
3.3.6 when will it fail?-Bad examples	36	(3)
4. PROBABILISTIC LOCAL STRUCTURE MODELS	39	(14)
4.1 Introduction	39	(1)
4.2 The Proposed Modeling Scheme	40	(4)
4.2.1 Classification by class-conditional density	41	(1)
4.2.2 Joint distribution for k-tuples	41	(1)
4.2.3 Histogram factorization based on ICA	42	(1)
4.2.4 Distance-sensitive histograming for modeling spatial dependencies	43	(1)
4.3 Implementation Issues	44	(1)
4.4 Experiments and Discussion	44	(7)
4.4.1 Object detection/localization	44	(5)
4.4.2 Image retrieval	49	(2)
4.5 Summary and Discussion	51	(2)
5. CONSTRUCTING TABLE-OF-CONTENT FOR VIDEOS	53	(22)
5.1 Introduction	53	(3)
5.2 Related Work	56	(2)
5.2.1 Shot and key frame based video ToC	56	(1)
5.2.2 Group based video ToC	57	(1)
5.2.3 Scene based video ToC	57	(1)
5.3 The Proposed Approach	58	(9)
5.3.1 Shot Boundary Detection and Key Frame Extraction	59	(1)
5.3.2 Spatio-Temporal Feature Extraction	59	(1)
5.3.3 Time-Adaptive Grouping	60	(2)
5.3.4 Scene Structure Construction	62	(5)
5.4 Determination of the Parameters	67	(3)
5.4.1 Gaussian Normalization	68	(1)
5.4.2 Determining W and WA	69	(1)
5.4.3 Determining groupThreshold and sceneThreshold	69	(1)
5.5 Experimental Results	70	(2)
5.6 Conclusions	72	(3)
6. NONLINEARLY SAMPLED VIDEO STREAMING	75	(22)
6.1 Introduction	76	(2)
6.2 Problem Statement	78	(1)
6.3 Frame Saliency Scoring	79	(1)
6.4 Scenario and Assumptions	80	(1)
6.5 Minimum Buffer Formulation	81	(4)
6.5.1 As an integer optimization problem	82	(1)
6.5.2 As a shortest path problem	82	(1)
6.5.3 Dynamic programming solution	83	(2)
6.6 Limited-Buffer Formulation	85	(2)
6.6.1 Channel and buffer modeling by Z-B diagram	85	(1)
6.6.2 The algorithm	86	(1)
6.6.3 Greedy strategy will fail	87	(1)
6.7 Extensions and Analysis	87	(5)
6.7.1 User-in-the-loop: Interactive frame selection	87	(2)
6.7.2 Variable file size	89	(1)
6.7.3 Time-varying bandwidth	90	(1)
6.7.4 Key-segments versus key-frames	91	(1)
6.7.5 Tolerance for small delays	91	(1)
6.7.6 Complexity analysis	92	(1)
6.8 Experimental Evaluation	92	(2)
6.9 Discussion	94	(3)
7. RELEVANCE FEEDBACK FOR VISUAL DATA RETRIEVAL	97	(52)
7.1 The Need for User-in-the-Loop	98	(1)
7.2 Problem Statement	99	(1)
7.3 Overview of Existing Techniques	100	(4)
7.3.1 Variants	100	(2)
7.3.2 From heuristic to optimal scheme	102	(2)
7.4 Learning from Positive Feedbacks	104	(15)
7.4.1 Notations	105	(2)
7.4.2 MARS and MindReader approaches	107	(2)
7.4.3 A hierarchical optimization approach	109	(4)
7.4.4 Experiments and Evaluations	113	(6)
7.4.5 Discussions	119	(1)
7.5 Adding Negative Feedbacks: Discriminant Analysis?	119	(3)
7.5.1 Two-class assumption	120	(1)
7.5.2 Multiclass assumption	121	(1)
7.5.3 Unsupervised clustering	121	(1)
7.5.4 Dimensionality reduction matrix	122	(1)
7.6 Biased Discriminant Analysis	122	(5)
7.6.1 (1+x)-class assumption	122	(1)
7.6.2 Biased discriminant analysis (BDA)	123	(1)
7.6.3 Generalized BDA	123	(1)
7.6.4 Regularization and discounting factors	124	(1)
7.6.5 Discriminating transform	125	(1)
7.6.6 Properties of the discriminating transform	125	(2)
7.7 Nonlinear Extensions Using Kernel and Boosting	127	(8)
7.7.1 Boosting biased discriminant analysis (BBDA)	127	(2)
7.7.2 Biased discriminant analysis using kernel (KBDA)	129	(6)
7.8 Comparisons and Analysis	135	(10)
7.8.1 Linear/quadratic case	136	(4)
7.8.2 Nonlinear case using kernel	140	(3)
7.8.3 Nonlinear case using boosting	143	(2)
7.9 Relevance Feedback on Image Tiles	145	(4)
8. TOWARD UNIFICATION OF KEYWORDS AND LOWLEVEL CONTENTS	149	(14)
8.1 Introduction	150	(3)
8.1.1 Automatic thesaurus construction in document analysis	150	(1)
8.1.2 Outline of proposed method	151	(1)
8.1.3 Background and assumptions	151	(2)
8.2 Joint Querying and Relevance Feedback	153	(3)
8.2.1 Soft vector representation of keywords	153	(1)
8.2.2 Joint global similarity search	154	(2)
8.3 Learning Semantic Relations between Keywords	156	(6)
8.3.1 WARF: Word Association via Relevance Feedback	157	(2)
8.3.2 Semantic grouping of keywords	159	(3)
8.4 Discussion	162	(1)
9. FUTURE RESEARCH DIRECTIONS	163	(4)
9.1 Low-level and intermediate-level visual descriptors	163	(1)
9.2 Learning from user interactions	164	(1)
9.3 Unsupervised detection of patterns/events	164	(1)
9.4 Domain-specific applications	164	(3)
REFERENCES	167	(18)
INDEX	185