Mastering Data Mining : The Art and Science of Customer Relationship Management

by Michael J. Berry; Gordon S. Linoff

• ISBN13: 9780471331230
• ISBN10: 0471331236
• Format: Paperback
• Copyright: 1999-12-01
• Publisher: Wiley

Summary

This book shows readers how to apply data mining techniques to real-world business cases. The objective: to teach the best methods for selecting a technique and applying it correctly to solve practical problems. Each chapter uses this basic approach: Explain the business problem and show how to pick the right data mining technique to solve the problem. How to collect the right set of data to mine and how to model that data. Perform the analysis and evaluate the results. Review lessons learned and their application to similar business cases across industries. Chapters provide solutions for marketing, and financial analysis, manufacturing, fraud detection, and other business applications; each successive chapter shows how to master a more complex set of business problems. Readers will find practical data mining examples for Improving production processes. Retaining customer loyalty. Targeting direct selling through catalogues. Identifying new markets for products and services. Identifying cross-selling opportunities. Each case is based on the authors' real experience in specific industries like banking, telecom, transportation, and retail-the authors explain how the lessons learned apply to similar cases across industries.

Author Biography

MICHAEL J. A. BERRY [e-mail] and GORDON S. LINOFF [e-mail] are the founders of Data Miners Inc., a respected data mining consultancy. When not actively engaged in data mining projects, they present classes and seminars that have been well received around the world.

Table of Contents

Acknowledgments	xv
Introduction	xvii
Part One Setting the Focus	1	(90)
Data Mining in Context	5	(16)
What Is Data Mining?	7	(1)
What Can Data Mining Do?	8	(3)
Classification	8	(1)
Estimation	9	(1)
Prediction	10	(1)
Affinity Grouping or Association Rules	10	(1)
Clustering	10	(1)
Description and Visualization	11	(1)
The Business Context for Data Mining	11	(3)
Data Mining as a Research Tool	12	(1)
Data Mining for Process Improvement	13	(1)
Data Mining for Marketing	13	(1)
Data Mining for Customer Relationship Management	14	(1)
The Technical Context for Data Mining	14	(5)
Data Mining and Machine Learning	15	(1)
Data Mining and Statistics	15	(1)
Data Mining and Decision Support	16	(3)
Data Mining and Computer Technology	19	(1)
The Societal Context for Data Mining	19	(2)
Why Master the Art?	21	(18)
Four Approaches to Data Mining	23	(13)
Purchasing Scores	24	(1)
Purchasing Software	24	(8)
Hiring Outside Experts	32	(3)
Developing In-House Expertise	35	(1)
Lessons Learned	36	(3)
Data Mining Methodology: The Virtuous Cycle Revisited	39	(26)
Two Styles of Data Mining	40	(3)
Directed Data Mining	40	(2)
Undirected Data Mining	42	(1)
The Virtuous Cycle of Data Mining	43	(1)
Identifying the Right Business Problem	44	(4)
Is the Data Mining Effort Necessary?	45	(1)
Is There a Particular Segment or Subgroup That Is Most Interesting?	46	(1)
What Are the Relevant Business Rules?	47	(1)
What about the Data?	47	(1)
Verifying the Opinion of Domain Experts	47	(1)
Transforming Data into Actionable Results	48	(9)
Identify and Obtain Data	49	(1)
Validate, Explore, and Clean the Data	50	(1)
Transpose the Data to the Right Granularity	51	(1)
Add Derived Variables	52	(1)
Prepare the Model Set	53	(1)
Choose the Modeling Technique and Train the Model	54	(1)
Check Performance of the Models	54	(3)
Acting on the Results	57	(1)
Measuring the Model's Effectiveness	58	(1)
What Makes Predictive Modeling Successful?	59	(5)
Time Frames of Predictive Modeling	59	(1)
Modeling Shelf-Life	60	(1)
The Past Is a Good Predictor of the Future	61	(1)
The Data is Available	62	(1)
The Data Contains What We Want to Predict	63	(1)
Lessons Learned	64	(1)
Customers and Their Lifecycles	65	(26)
Who Is the Customer?	66	(6)
Consumers	66	(1)
Business Customers	67	(3)
Customer Segments	70	(2)
The Customer Lifecycle	72	(7)
Stages of the Lifecycle	73	(2)
Major Lifecycle Events	75	(3)
Data Appears at Different Times in the Lifecycle	78	(1)
The Customer's Lifecycle	79	(1)
Targeting the Right Customers at the Right Time	80	(10)
Budget Optimization	80	(2)
Campaign Optimization	82	(4)
Customer Optimization	86	(4)
Lessons Learned	90	(1)
Part Two The Three Pillars of Data Mining	91	(162)
Data Mining Techniques and Algorithms	99	(32)
Different Goals Call for Different Techniques	100	(2)
Different Data Types Call for Different Techniques	102	(1)
Three Data Mining Techniques	102	(1)
Automatic Cluster Detection	103	(8)
How K-Means Cluster Detection Works	104	(3)
Consequences of Choosing Clustering	107	(4)
Decision Trees	111	(10)
How Decision Trees Work	112	(1)
How Decision Trees are Built	113	(6)
Consequences of Choosing Decision Trees	119	(2)
Neural Networks	121	(8)
Training Neural Networks	125	(2)
Consequences of Choosing Neural Networks	127	(2)
Lessons Learned	129	(2)
Data, Data Everywhere...	131	(52)
What Should Data Look Like?	132	(9)
The Rows	132	(2)
The Columns	134	(4)
Roles of Columns in Data Mining	138	(2)
Data for Data Mining	140	(1)
What Does Data Really Look Like?	141	(16)
Where Data Comes From	141	(9)
The Right Level of Granularity	150	(3)
Different Ways to Measure Data Values	153	(4)
How Much Data Is Enough?	157	(1)
Derived Variables	158	(11)
Issues in Working with Derived Variables	159	(1)
Handling Outliers	160	(2)
Combinations of Columns	162	(1)
Summarizations	163	(1)
Extracting Features from Single Columns	164	(3)
Time Series	167	(2)
Case Study: Defining Customer Behaviour	169	(8)
Dirty Data	177	(4)
Missing Data	177	(2)
Fuzzy Definitions	179	(1)
Incorrect Values	180	(1)
Lessons Learned	181	(2)
Building Effective Predictive Models	183	(44)
Building Good Predictive Models	184	(9)
A Process for Building Predictive Models	184	(2)
Lift as a Measure of Performance	186	(5)
Model Stability	191	(1)
The Challenge of Model Stability	192	(1)
Working with the Model Set	193	(20)
Divide and Conquer: Training, Test, and Evaluation Sets	193	(1)
How the Size of the Model Set Affects Results	194	(1)
How the Density of the Model Set Affects Results	195	(1)
Sampling	196	(1)
What is Oversampling?	197	(4)
Modeling Time-Dependent Data	201	(2)
Model Inputs and Model Outputs	203	(3)
Latency: Taking Model Deployment into Account	206	(3)
Time and Missing Data	209	(1)
Building Models that Easily Shift in Time	210	(2)
Naming Fields	212	(1)
Using Multiple Models	213	(8)
Multiple Model Voting	213	(5)
Segmenting the Input	218	(1)
Other Reasons to Combine Models	219	(2)
Experiment!	221	(3)
The Model Set	221	(2)
Different Types of Models and Model Parameters	223	(1)
Time Frame	223	(1)
Lessons Learned	224	(3)
Taking Control: Setting up a Data Mining Environment	227	(26)
Getting Started	228	(2)
What is a Data Mining Environment?	228	(1)
Four Case Studies	229	(1)
What Makes a Data Mining Environment Successful?	229	(1)
Building Up a Core Competency Internally	230	(5)
Data Mining in the Insurance Industry	231	(1)
Getting Started	231	(4)
Building a New Line of Business	235	(5)
Going Online	235	(1)
The Environment	236	(1)
The Prospect Data Warehouse	236	(1)
The Next Step	237	(3)
Building Data Mining Skills on Data Warehouse Efforts	240	(2)
A Special Kind of Data Warehouse	240	(1)
The Plan for Data Mining	240	(1)
Data Mining in IT	241	(1)
Data Mining Using Tessera RME	242	(9)
Requirements for an Advanced Data Mining Environment	242	(1)
What Is RME?	243	(1)
How RME Works	244	(2)
How RME Helps Prepare Data	246	(1)
How RME Supports Sampling	247	(1)
How RME Helps in Model Development	248	(1)
How RME Helps in Scoring and Managing Models	249	(2)
Lessons Learned	251	(2)
Part Three Case Studies	253	(232)
Who Needs Bag Balm and Pants Stretchers	261	(18)
The Vermont Country Store	262	(3)
How Vermont Country Store Got Where It Is Today	263	(2)
Predictive Modeling at Vermont Country Store	265	(1)
The Business Problem	265	(2)
The Data	267	(3)
The Technical Approach	270	(6)
Choice of Software Package	270	(1)
The Baseline---RFM and Segmentation	271	(2)
The Challengers---Neural Networks, Decision Trees, and Regression	273	(3)
Determining What Would have Happened	276	(1)
Calculating the Return on Investment	276	(1)
The Future	276	(1)
Expected Benefits	277	(1)
Lessons Learned	277	(2)
Who Gets What? Building a Best Next Offer Model for an Online Bank	279	(32)
Gaining Wallet-Share	279	(2)
The Business Problem	281	(1)
The Data	282	(8)
From Accounts to Customers	284	(3)
Defining the Products to Be Offered	287	(3)
Approach to the Problem	290	(4)
Comparable Scores	291	(1)
If it Walks Like a Duck,...	292	(1)
Pitfalls of This Approach	292	(2)
Building the Models	294	(14)
Building a Decision Tree Model for Brokerage	296	(10)
Building the Rest of the Models	306	(1)
Getting to a Cross-Sell Model	307	(1)
In a More Perfect World	308	(1)
Lessons Learned	308	(3)
Please Don't Go! Churn Modeling in Wireless communication	311	(46)
The Wireless Telephone Industry	312	(4)
A Rapidly Maturing Industry	313	(1)
Some Differences from other Industries	314	(2)
The Business Problem	316	(10)
Project Background	316	(1)
Specifics about this Market	317	(1)
What Is Churn?	318	(1)
Why is Churn Modeling useful?	319	(1)
Three Goals	320	(2)
Approach to Building the Churn Model	322	(3)
The Project Itself	325	(1)
Building a Churn Model: A Real-Life Application	326	(15)
The Choice of Tool	326	(1)
Segmenting the Model set	326	(1)
The Final Four (Models)	327	(4)
Choice of Modeling Algorithm	331	(7)
The Size and Density of the Model Set	338	(1)
The Effect of Latency (or Taking Deployment into Account)	339	(1)
Translating Models in Time	340	(1)
The Data	341	(8)
The Basic Customer Model	341	(2)
From Telephone Calls to Data	343	(1)
Historical Churn Rates	344	(1)
Data at the Customer and Account Level	345	(1)
Data at the Service Level	345	(1)
Data Billing History	346	(1)
Rejecting Some Variables	346	(1)
Derived Variables	347	(2)
Lessons about Building Churn Models	349	(6)
Finding the Most Significant Variables	349	(1)
Listening to the Business Users	349	(1)
Listening to the Data	350	(1)
Including Historical Churn Rates	351	(1)
Composing the Model Set	352	(2)
Building a Model for the Churn Management Application	354	(1)
Listening to the Data to Determine Model Parameters	354	(1)
Understanding the Algorithm and the Tool	355	(1)
Lessons Learned	355	(2)
Converging on the Customer: Understanding Customer Behavior in the Telecommunications Industry	357	(38)
Dataflows	358	(6)
What is a Dataflow?	359	(1)
Basic Operations	360	(1)
Dataflows in a Parallel Environment	361	(2)
Why are Dataflows Efficient?	363	(1)
The Business Problem	364	(2)
Project Background	364	(1)
Important Marketing Questions	365	(1)
The Data	366	(6)
Call Detail Data	366	(2)
Customer Data	368	(4)
Auxiliary Files	372	(1)
A Voyage of Discovery	372	(21)
What is in a Call Duration?	372	(2)
Calls by Time of Day	374	(4)
Calls by Market Segment	378	(4)
International Calling Patterns	382	(2)
When Are Customers at Home?	384	(3)
Internet Service Providers	387	(1)
Private Networks	388	(3)
Concurrent Calls	391	(2)
Lessons Learned	393	(2)
Who is Buying What? Getting to know Supermarket Shoppers	395	(40)
An Industry in Transition	396	(5)
Supermarkets as Information Brokers	396	(3)
Shifting the Focus from Products to Customers	399	(2)
Three Case Studies	401	(1)
Analyzing Ethnic Purchasing Patterns	402	(8)
Business Background	402	(1)
The Data	402	(3)
A Triumph for Visualization	405	(2)
A Failed Approach	407	(1)
Just the Facts	407	(3)
Who Buys Yogurt at the Supermarket?	410	(14)
Business Background	411	(1)
The Data	411	(5)
From Groceries to Customers	416	(3)
Finding Clusters of Customers	419	(2)
Putting the Clusters to Work	421	(3)
Who Buys Meat at the Health Food Store?	424	(8)
Association Rules for Market Basket Analysis	424	(5)
People are More Interesting Than Groceries	429	(3)
Lessons Learned	432	(3)
Waste Not, Want Not: Improving Manufacturing Processes	435	(30)
Data Mining to Reduce Cost at R. R Donneley	436	(9)
The Technical Problem	436	(1)
The Business Problem	437	(1)
The Data	438	(4)
Inducing Rules for Cylinder Bands	442	(2)
Change on the Shop Floor	444	(1)
Long-Term Impact	445	(1)
Reducing Paper Wastage at Time Inc.	445	(19)
The Business Problem	446	(3)
The Data	449	(3)
Approach to the Problem	452	(1)
Types of Waste	453	(2)
Addressable Waste	455	(1)
Inducing Rules for Addressable Waste	456	(1)
Data Transformation	456	(3)
Data Characterization and Profiling	459	(1)
Decision Trees	459	(3)
Association Rules	462	(1)
Putting it All Together	463	(1)
Lessons Learned	464	(1)
The Societal Context: Data Mining and Privacy	465	(20)
The Privacy Prism	466	(2)
Is Data Mining a Threat?	468	(2)
The Expectation of Privacy	470	(6)
The Importance of Privacy	471	(5)
Information in the Material World	476	(1)
Information in the Electronic World	477	(6)
Identifying the Customer	478	(2)
Putting it All Together	480	(3)
The Promise of Data Mining	483	(2)
Index	485