Data Mining: The Textbook
| 1.5. SCALABILITY ISSUES AND THE STREAMING SCENARIO
|
|
21 |
time series. For network data, the clustering problem discovers densely connected groups of nodes, and is also referred to as community detection.
1.4.5.3 Outlier Detection with Complex Data Types
Dependencies can be used to define expected values of data items. Deviations from these expected values are outliers. For example, a sudden jump in the value of a time series will result in a position outlier at the specific spot at which the jump occurs. The idea in these methods is to use prediction-based techniques to forecast the value at that position. Significant deviation from the prediction is reported as a position outlier. Such outliers can be defined in the context of time-series, spatial, and sequential data, where significant deviations from the corresponding neighborhoods can be detected using autoregressive, Markovian, or other models. In the context of graph data, outliers may correspond to unusual properties of nodes, edges, or entire subgraphs. Thus, the complex data types show significant richness in terms of how outliers may be defined.
1.4.5.4 Classification with Complex Data Types
The classification problem also shows a significant amount of variation in the different complex data types. For example, class labels can be attached to specific positions in a series, or they can be attached to the entire series. When the class labels are attached to a specific position in the series, this can be used to perform supervised event detection, where the first occurrence of an event-specific label (e.g., the breakdown of a machine as suggested by the underlying temperature and pressure sensor) of a particular series represents the occurrence of the event. For the case of network data, the labels may be attached to individual nodes in a very large network, or to entire graphs in a collection of multiple graphs. The former case corresponds to the classification of nodes in a social network, and is also referred to as collective classification. The latter case corresponds to the chemical compound classification problem, in which labels are attached to compounds on the basis of their chemical properties.
1.5 Scalability Issues and the Streaming Scenario
Scalability is an important concern in many data mining applications due to the increasing sizes of the data in modern-day applications. Broadly speaking, there are two important scenarios for scalability:
The data are stored on one or more machines, but it is too large to process efficiently. For example, it is easy to design efficient algorithms in cases where the entire data can be maintained in main memory. When the data are stored on disk, it is important to be design the algorithms in such a way that random access to the disk is minimized. For very large data sets, big data frameworks, such as MapReduce, may need to be used. This book will touch upon this kind of scalability at the level of disk-resident processing, where needed.
The data are generated continuously over time in high volume, and it is not practical to store it entirely. This scenario is that of data streams, in which the data need to be processed with the use of an online approach.
22 CHAPTER 1. AN INTRODUCTION TO DATA MINING
The latter scenario requires some further exposition. The streaming scenario has become increasingly popular because of advances in data collection technology that can collect large amounts of data over time. For example, simple transactions of everyday life such as using a credit card or the phone may lead to automated data collection. In such cases, the volume of the data is so large that it may be impractical to store directly. Rather, all algorithms must be executed in a single pass over the data. The major challenges that arise in the context of data stream processing are as follows:
Dostları ilə paylaş:
