Judith Hurwitz is an expert in cloud computing, information management, and business strategy.

Alan Nugent has extensive experience in cloud-based big data solutions.

Dr. Fern Halper specializes in big data and analytics.

Marcia Kaufman specializes in cloud infrastructure, information management, and analytics.

Introduction 1

About This Book 2

Foolish Assumptions 2

How This Book Is Organized 3

Part I: Getting Started with Big Data 3

Part II: Technology Foundations for Big Data 3

Part III: Big Data Management 3

Part IV: Analytics and Big Data 4

Part V: Big Data Implementation 4

Part VI: Big Data Solutions in the Real World 4

Part VII: The Part of Tens 4

Glossary 4

Icons Used in This Book 5

Where to Go from Here 5

Part I: Getting Started with Big Data 7

Chapter 1: Grasping the Fundamentals of Big Data 9

The Evolution of Data Management 10

Understanding the Waves of Managing Data 11

Wave 1: Creating manageable data structures 11

Wave 2: Web and content management 13

Wave 3: Managing big data 14

Defining Big Data 15

Building a Successful Big Data Management Architecture 16

Beginning with capture, organize, integrate, analyze, and act 16

Setting the architectural foundation 17

Performance matters 20

Traditional and advanced analytics 22

The Big Data Journey 23

Chapter 2: Examining Big Data Types 25

Defining Structured Data 26

Exploring sources of big structured data 26

Understanding the role of relational databases in big data 27

Defining Unstructured Data 29

Exploring sources of unstructured data 29

Understanding the role of a CMS in big data management 31

Looking at Real-Time and Non-Real-Time Requirements 32

Putting Big Data Together 33

Managing different data types 33

Integrating data types into a big data environment 34

Chapter 3: Old Meets New: Distributed Computing 37

A Brief History of Distributed Computing 37

Giving thanks to DARPA 38

The value of a consistent model 39

Understanding the Basics of Distributed Computing 40

Why we need distributed computing for big data 40

The changing economics of computing 40

The problem with latency 41

Demand meets solutions 41

Getting Performance Right 42

Part II: Technology Foundations for Big Data 45

Chapter 4: Digging into Big Data Technology Components 47

Exploring the Big Data Stack 48

Layer 0: Redundant Physical Infrastructure 49

Physical redundant networks 51

Managing hardware: Storage and servers 51

Infrastructure operations 51

Layer 1: Security Infrastructure 52

Interfaces and Feeds to and from Applications and the Internet 53

Layer 2: Operational Databases 54

Layer 3: Organizing Data Services and Tools 56

Layer 4: Analytical Data Warehouses 56

Big Data Analytics 58

Big Data Applications 58

Chapter 5: Virtualization and How It Supports Distributed Computing 61

Understanding the Basics of Virtualization 61

The importance of virtualization to big data 63

Server virtualization 64

Application virtualization 65

Network virtualization 66

Processor and memory virtualization 66

Data and storage virtualization 67

Managing Virtualization with the Hypervisor 68

Abstraction and Virtualization 69

Implementing Virtualization to Work with Big Data 69

Chapter 6: Examining the Cloud and Big Data 71

Defining the Cloud in the Context of Big Data 71

Understanding Cloud Deployment and Delivery Models 72

Cloud deployment models 73

Cloud delivery models 74

The Cloud as an Imperative for Big Data 75

Making Use of the Cloud for Big Data 77

Providers in the Big Data Cloud Market 78

Amazon's Public Elastic Compute Cloud 78

Google big data services 79

Microsoft Azure 80

OpenStack 80

Where to be careful when using cloud services 81

Part III: Big Data Management 83

Chapter 7: Operational Databases 85

RDBMSs Are Important in a Big Data Environment 87

PostgreSQL relational database 87

Nonrelational Databases 88

Key-Value Pair Databases 89

Riak key-value database 90

Document Databases 91

MongoDB 92

CouchDB 93

Columnar Databases 94

HBase columnar database 94

Graph Databases 95

Neo4J graph database 96

Spatial Databases 97

PostGIS/OpenGEO Suite 98

Polyglot Persistence 99

Chapter 8: MapReduce Fundamentals 101

Tracing the Origins of MapReduce 101

Understanding the map Function 103

Adding the reduce Function 104

Putting map and reduce Together 105

Optimizing MapReduce Tasks 108

Hardware/network topology 108

Synchronization 108

File system 108

Chapter 9: Exploring the World of Hadoop 111

Explaining Hadoop 111

Understanding the Hadoop Distributed File System (HDFS) 112

NameNodes 113

Data nodes 114

Under the covers of HDFS 115

Hadoop MapReduce 116

Getting the data ready 117

Let the mapping begin 118

Reduce and combine 118

Chapter 10: The Hadoop Foundation and Ecosystem 121

Building a Big Data Foundation with the Hadoop Ecosystem 121

Managing Resources and Applications with Hadoop YARN 122

Storing Big Data with HBase 123

Mining Big Data with Hive 124

Interacting with the Hadoop Ecosystem 125

Pig and Pig Latin 125

Sqoop 126

Zookeeper 127

Chapter 11: Appliances and Big Data Warehouses 129

Integrating Big Data with the Traditional Data Warehouse 129

Optimizing the data warehouse 130

Differentiating big data structures from data warehouse data 130

Examining a hybrid process case study 131

Big Data Analysis and the Data Warehouse 133

The integration lynchpin 134

Rethinking extraction, transformation, and loading 134

Changing the Role of the Data Warehouse 135

Changing Deployment Models in the Big Data Era 136

The appliance model 136

The cloud model 137

Examining the Future of Data Warehouses 137

Part IV: Analytics and Big Data 139

Chapter 12: Defining Big Data Analytics 141

Using Big Data to Get Results 142

Basic analytics 142

Advanced analytics 143

Operationalized analytics 146

Monetizing analytics 146

Modifying Business Intelligence Products to Handle Big Data 147

Data 147

Analytical algorithms 148

Infrastructure support 148

Studying Big Data Analytics Examples 149

Orbitz 149

Nokia 150

NASA 150

Big Data Analytics Solutions 151

Chapter 13: Understanding Text Analytics and Big Data 153

Exploring Unstructured Data 154

Understanding Text Analytics 155

The difference between text analytics and search 156

Analysis and Extraction Techniques 157

Understanding the extracted information 159

Taxonomies 160

Putting Your Results Together with Structured Data 160

Putting Big Data to Use 161

Voice of the customer 161

Social media analytics 162

Text Analytics Tools for Big Data 164

Attensity 164

Clarabridge 165

IBM 165

OpenText 165

SAS 166

Chapter 14: Customized Approaches for Analysis of Big Data 167

Building New Models and Approaches to Support Big Data 168

Characteristics of big data analysis 168

Understanding Different Approaches to Big Data Analysis 170

Custom applications for big data analysis 171

Semi-custom applications for big data analysis 173

Characteristics of a Big Data Analysis Framework 174

Big to Small: A Big Data Paradox 177

Part V: Big Data Implementation 179

Chapter 15: Integrating Data Sources 181

Identifying the Data You Need 181

Exploratory stage 182

Codifying stage 184

Integration and incorporation stage 184

Understanding the Fundamentals of Big Data Integration 186

Defining Traditional ETL 187

Data transformation 188

Understanding ELT - Extract, Load, and Transform 189

Prioritizing Big Data Quality 189

Using Hadoop as ETL 191

Best Practices for Data Integration in a Big Data World 191

Chapter 16: Dealing with Real-Time Data Streams and Complex Event Processing 193

Explaining Streaming Data and Complex Event Processing 194

Using Streaming Data 194

Data streaming 195

The need for metadata in streams 196

Using Complex Event Processing 198

Differentiating CEP from Streams 199

Understanding the Impact of Streaming Data and CEP on Business 200

Chapter 17: Operationalizing Big Data 201

Making Big Data a Part of Your Operational Process 201

Integrating big data 202

Incorporating big data into the diagnosis of diseases 203

Understanding Big Data Workflows 205

Workload in context to the business problem 206

Ensuring the Validity, Veracity, and Volatility of Big Data 207

Data validity 207

Data volatility 208

Chapter 18: Applying Big Data within Your Organization 211

Figuring the Economics of Big Data 212

Identification of data types and sources 212

Business process modifications or new process creation 215

The technology impact of big data workflows 215

Finding the talent to support big data projects 216

Calculating the return on investment (ROI) from big data investments 216

Enterprise Data Management and Big Data 217

Defining Enterprise Data Management 217

Creating a Big Data Implementation Road Map 218