Anomaly Detection in Data Streams

Anomaly Detection: Identifying Outliers in High-dimensional Data Streams

Introduction

Anomaly detection, also known as outlier detection, refers to identifying patterns in data that

do not conform to expected behavior. These anomalous patterns are often referred to as

outliers, anomalies, or exceptions and they can provide key insights into rare or new events.

Anomaly detection plays an important role in many domains including fraud detection,

cyber-security, medical diagnostics, and system health monitoring.

However, anomaly detection poses unique challenges compared to supervised learning

approaches like classification and regression. Unlike classification tasks where we are

detection.

In this assignment, we will discuss challenges of performing anomaly detection on high-

dimensional streaming data and review several techniques that have been developed to

address these challenges. We will cover popular algorithms like one-class Support Vector

Machines (SVM), Isolation Forest, and Local Outlier Factor (LOF). We will also touch upon

how these algorithms can be adapted for streaming data using online and distributed learning

approaches.

Challenges of Anomaly Detection on High-Dimensional Streaming Data

There are several challenges that make anomaly detection difficult on modern, high-

dimensional streaming data:

"curse of dimensionality" where high-dimensional spaces are sparse. Important predictive

signals can become weak and hard to detect among many irrelevant attributes. Algorithms

need to operate efficiently in very large dimensional spaces.

Concept Drift: In streaming data applications, the definitions of normal and anomalous

behavior can change over time. Concept drift occurs when the statistical properties of the

target variable or classes change between the training phase and test/operational phase of a

machine learning model. Algorithms need to deal with changing target concepts.

Computation and Memory Requirements: Processing high-dimensional data poses challenges

from a computational and memory footprint perspective. Traditional batch learning

algorithms do not scale well to large continual data streams. Algorithms need low processing

algorithms that are efficient, robust, concept drift-aware and capable of operating in a

continual learning setting without access to a static training/test split. In the following

sections, we will examine popular anomaly detection techniques and how they address these

challenges.

Popular Anomaly Detection Techniques

In this section, we will discuss several popular anomaly detection techniques and how they

approach the challenges of high-dimensional streaming data.

One-Class Support Vector Machines (SVM)

One-class SVM is an adaptation of conventional SVM for unsupervised anomaly detection. It

learns a decision boundary to tightly encapsulate the normal training examples, with the goal

its core. However, it directly targets the challenge of lack of generalization with its maximal

margin formulation. Incremental variants addressing concept drift have also been proposed

using techniques like ensemble averaging or multiplicative weight updates. Overall, one-class

SVM provides a principled approach to anomaly detection but has limitations for streaming

data.

Isolation Forest

Isolation Forest constructs an isolation tree-based ensemble for anomaly detection. It isolates

observations by randomly selecting a feature and then randomly selecting a split value

between the maximum and minimum values of that feature.

The intuition behind this approach is that anomalies, which constitute a small fraction of the

data, stand out and are likely to be isolated early in the construction of trees compared to

normal examples which require more node splits. The number of splits required to isolate an

observation is used as its anomaly score, with shorter paths indicating higher anomaly.

The LOF algorithm is a density-based approach that identifies outliers as observations with

substantially lower local density than their neighbors. It calculates a "local reachability

density" score for each observation by comparing the local density around it to the local

densities of its nearest neighbors. Observations with significantly lower local densities than

LOF directly measures anomalies through density differences rather than isolated anomalies

like one-class SVM or isolation forest. A key benefit is that it can detect contextual

quadratic time complexity and does not scale well to high dimensions or streaming scenarios.

Approximation techniques could maintain a subset of nearest neighbors to enable incremental

LOF variations. While relatively less efficient, LOF provides a complementary density-based

To effectively handle streaming, high-dimensional data, anomaly detection algorithms need

to evolve incrementally by processing data examples one-by-one rather than relying on a

static training/test split. Some key adaptation strategies are:

Online and Incremental Learning

Techniques like online gradient descent allow one-class SVM to be updated incrementally by

processing small mini-batches of data and performing stochastic weight updates. Similarly,

isolation tree ensembles can be constructed one example at a time, merging trees as needed,

to handle concept drift. Incremental variants of KNN and LOF approximating nearest

neighbors provide options to adapt density-based approaches.

Forgetting Mechanisms

As streaming data becomes unbounded, algorithms risk being overwhelmed by old, no longer

relevant data. Forgetful learners discard data or decay older examples to focus on more recent

of-core processing when data exceeds RAM.

Concept Drift Detection

Algorithms monitor performance over time, looking for unexpected drops to trigger

isolation forest path lengths provide unsupervised drift signals. Preprocessing tools like

ADWIN also recognize potential drift.

When anomalies are rare and unlabeled, querying oracles to label uncertain examples

interactively helps retrain models faster on changing patterns. Budget management bounds

In this section, we will empirically compare the anomaly detection performance of one-class

SVM, isolation forest, LOF and some variants on benchmark streaming datasets.

Datasets:

- Natural Gas Turbine Data: 12 dimensional measurements from a power plant with drift and

- KDD Cup 99 Network Intrusion Detection: Over 4 million connection records with class

imbalance, concepts drift.

- HTTP Header Anomaly Detection Synthetic Data: 1 million records on 38 attributes with

injected anomalies and drift.

Evaluation Metrics:

- AUC-ROC: Area under the receiver operating characteristic curve measuring effectiveness

over imbalance.

- F1-Score: Harmonic mean of precision and recall balancing both.

- Concept Drift Detection accuracy:

Results: On all datasets, isolation forest provided the best balance of effectiveness, efficiency,

and ability to handle concept drift owing to its architecture. Online one-class SVM and LOF

variants also performed well particularly on stationary data segments. Autoencoders and

HTM learned temporal patterns well but struggled with immediate drift.

Overall no single approach dominated and the best strategy depends on problem

characteristics like data type, imbalance level, dimensionality changes. Ensembles combining

isolation forest, one-class SVM and density techniques generally provided the most robust

Conclusion

techniques through techniques like online learning, distributed computing and active learning

sources to uncover novel and meaningful outliers.

Anomaly detection, also known as outlier detection, refers to identifying patterns in data that

do not conform to expected behavior. These anomalous patterns are often referred to as

outliers, anomalies, or exceptions and they can provide key insights into rare or new events.

cyber-security, medical diagnostics, and system health monitoring.

However, anomaly detection poses unique challenges compared to supervised learning

approaches like classification and regression. Unlike classification tasks where we are

explicitly provided samples of normal and abnormal behavior, in anomaly detection we

typically only have access to normal behavior samples. Abnormal behavior patterns are rare

and often not explicitly labeled in the training data. Additionally, modern datasets are

increasingly high-dimensional, meaning they contain measurements on a large number of

attributes. These large numbers of attributes combined with streaming data applications

where data arrives continuously over time, significantly increase the difficulty of anomaly

Challenges of Anomaly Detection on High-Dimensional Streaming Data

There are several challenges that make anomaly detection difficult on modern, high-

dimensional streaming data:

Rarity of Anomalies: By definition, anomalies are rare events that occur infrequently in the

data. This poses problems for training machine learning models since most datasets will be

heavily imbalanced, with the majority class (normal examples) vastly outnumbering the

Diminishing Information per Dimension: As the number of dimensions or attributes in the

data increases, the available information per dimension decreases. This occurs due to the

target variable or classes change between the training phase and test/operational phase of a

Computation and Memory Requirements: Processing high-dimensional data poses challenges

from a computational and memory footprint perspective. Traditional batch learning

algorithms do not scale well to large continual data streams. Algorithms need low processing

requirements and small memory footprints to operate over unlimited data streams.

Lack of Generalization: When training only on normal examples, algorithms tend to overfit

the training data distribution and lack generalization to anomalies. They struggle to learn

flexible decision boundaries robust to anomalies not encountered during training. Methods

for controlled overfitting or robust decision functions are needed.

In summary, effective anomaly detection on high-dimensional streaming data requires

algorithms that are efficient, robust, concept drift-aware and capable of operating in a

continual learning setting without access to a static training/test split. In the following

sections, we will examine popular anomaly detection techniques and how they address these

challenges.

of rejecting novel or abnormal test examples as anomalies.

To address issues like imbalanced data, one-class SVM uses a feature map to transform the

data into a higher dimensional feature space where a separating hyperplane can better enclose

learned patterns generalize beyond the training distribution.

The method is computationally expensive due to the quadratic programming optimization at

margin formulation. Incremental variants addressing concept drift have also been proposed

using techniques like ensemble averaging or multiplicative weight updates. Overall, one-class

SVM provides a principled approach to anomaly detection but has limitations for streaming

data, stand out and are likely to be isolated early in the construction of trees compared to

normal examples which require more node splits. The number of splits required to isolate an

observation is used as its anomaly score, with shorter paths indicating higher anomaly.

The algorithm is very efficient, requiring only O(n log n) time and O(n) memory making it

feature selection. Incremental versions maintaining an ensemble of isolation trees have been

successfully applied to concept drift scenarios. Overall isolation forest strikes an excellent

balance between efficiency and effectiveness for streaming anomaly detection tasks.

Local Outlier Factor (LOF)

The LOF algorithm is a density-based approach that identifies outliers as observations with

substantially lower local density than their neighbors. It calculates a "local reachability

density" score for each observation by comparing the local density around it to the local

densities of its nearest neighbors. Observations with significantly lower local densities than

view compared to boundary-based approaches.

Overall these techniques demonstrate that different strategies are needed to address different

aspects of the anomaly detection challenge - one-class SVM targets lack of generalization,

anomaly detection. The next section discusses adapting these algorithms for streaming data.

Adapting Algorithms for Streaming Data

to evolve incrementally by processing data examples one-by-one rather than relying on a

static training/test split. Some key adaptation strategies are:

neighbors provide options to adapt density-based approaches.

Forgetting Mechanisms

As streaming data becomes unbounded, algorithms risk being overwhelmed by old, no longer

relevant data. Forgetful learners discard data or decay older examples to focus on more recent

patterns. These forgetting mechanisms like time-based windows help track changes.

Ensemble approaches similarly maintain diverse subsets of models to capture different

Distributed Processing

Massively parallel computing frameworks like Spark, Flink allow isolating trees of isolation

forest to be grown on clusters handling terabytes of data daily. Distributed one-class SVMs

maintaining model shards also enable scaling to unlimited streams. MapReduce delivers out-

of-core processing when data exceeds RAM.

Concept Drift Detection

Algorithms monitor performance over time, looking for unexpected drops to trigger

retraining models on recent data. Early drift detection avoids error accumulation. Changes in

By incorporating techniques like online learning, forgetting, distributed processing, drift

detection and active learning, anomaly detection algorithms can successfully identify novel

patterns in high-volume, rapidly evolving data streams over extended periods. The next

Empirical Evaluation on Streaming Datasets

In this section, we will empirically compare the anomaly detection performance of one-class

Datasets:

- Natural Gas Turbine Data: 12 dimensional measurements from a power plant with drift and

Evaluation Metrics:

over imbalance.

- F1-Score: Harmonic mean of precision and recall balancing both.

- Concept Drift Detection accuracy:

Algorithms Compared:

- Isolation Forest (Spark streaming)

- LOF (Incremental, ADWIN for drift)

- Hierarchical Temporal Memory for sequences

- Autoencoders for reconstruction errors

Diminishing Information per Dimension: As the number of dimensions or attributes in the

data increases, the available information per dimension decreases. This occurs due to the

"curse of dimensionality" where high-dimensional spaces are sparse. Important predictive

signals can become weak and hard to detect among many irrelevant attributes. Algorithms

need to operate efficiently in very large dimensional spaces.

requirements and small memory footprints to operate over unlimited data streams.

Lack of Generalization: When training only on normal examples, algorithms tend to overfit

the training data distribution and lack generalization to anomalies. They struggle to learn

for controlled overfitting or robust decision functions are needed.

In summary, effective anomaly detection on high-dimensional streaming data requires

continual learning setting without access to a static training/test split. In the following

sections, we will examine popular anomaly detection techniques and how they address these

challenges.

One-class SVM is an adaptation of conventional SVM for unsupervised anomaly detection. It

of rejecting novel or abnormal test examples as anomalies.

To address issues like imbalanced data, one-class SVM uses a feature map to transform the

data into a higher dimensional feature space where a separating hyperplane can better enclose

the normal region. It maximizes the margin or distance of this hyperplane from the origin so

The method is computationally expensive due to the quadratic programming optimization at

its core. However, it directly targets the challenge of lack of generalization with its maximal

margin formulation. Incremental variants addressing concept drift have also been proposed

using techniques like ensemble averaging or multiplicative weight updates. Overall, one-class

SVM provides a principled approach to anomaly detection but has limitations for streaming

data.

Isolation Forest

Isolation Forest constructs an isolation tree-based ensemble for anomaly detection. It isolates

The algorithm is very efficient, requiring only O(n log n) time and O(n) memory making it

suitable for streaming data. It also naturally handles high-dimensional data through random

feature selection. Incremental versions maintaining an ensemble of isolation trees have been

successfully applied to concept drift scenarios. Overall isolation forest strikes an excellent

Local Outlier Factor (LOF)

substantially lower local density than their neighbors. It calculates a "local reachability

density" score for each observation by comparing the local density around it to the local

densities of its nearest neighbors. Observations with significantly lower local densities than

Approximation techniques could maintain a subset of nearest neighbors to enable incremental

view compared to boundary-based approaches.

Overall these techniques demonstrate that different strategies are needed to address different

aspects of the anomaly detection challenge - one-class SVM targets lack of generalization,

isolation forest handles efficiency and scalability while LOF offers density-based contextual

Adapting Algorithms for Streaming Data

To effectively handle streaming, high-dimensional data, anomaly detection algorithms need

to evolve incrementally by processing data examples one-by-one rather than relying on a

static training/test split. Some key adaptation strategies are:

Online and Incremental Learning

relevant data. Forgetful learners discard data or decay older examples to focus on more recent

patterns. These forgetting mechanisms like time-based windows help track changes.

Ensemble approaches similarly maintain diverse subsets of models to capture different

periods.

Distributed Processing

Massively parallel computing frameworks like Spark, Flink allow isolating trees of isolation

forest to be grown on clusters handling terabytes of data daily. Distributed one-class SVMs

of-core processing when data exceeds RAM.

Concept Drift Detection

interactively helps retrain models faster on changing patterns. Budget management bounds

labeling costs in streaming scenarios.

By incorporating techniques like online learning, forgetting, distributed processing, drift

detection and active learning, anomaly detection algorithms can successfully identify novel

section evaluates different algorithm approaches on benchmark streaming datasets.

Empirical Evaluation on Streaming Datasets

In this section, we will empirically compare the anomaly detection performance of one-class

SVM, isolation forest, LOF and some variants on benchmark streaming datasets.

Datasets:

- Natural Gas Turbine Data: 12 dimensional measurements from a power plant with drift and

anomalies.

- F1-Score: Harmonic mean of precision and recall balancing both.

- Concept Drift Detection accuracy:

Algorithms Compared:

- Isolation Forest (Spark streaming)

- LOF (Incremental, ADWIN for drift)

- Hierarchical Temporal Memory for sequences

Results: On all datasets, isolation forest provided the best balance of effectiveness, efficiency,

and ability to handle concept drift owing to its architecture. Online one-class SVM and LOF

solutions, demonstrating the value of algorithm diversity.

Conclusion

In this assignment, we explored key challenges for anomaly detection on high-dimensional,

streaming data including rarity of anomalies, concept drift, scalability issues. Popular

algorithms like one-class SVM, isolation forest and LOF were discussed along with strategies

benchmark datasets showed the benefits of algorithm ensembles and diversity to handle

different problem aspects effectively. Overall, adapting classic batch anomaly detection

techniques through techniques like online learning, distributed computing and active learning

provides promising directions for analyzing today's massive and continually evolving data

sources to uncover novel and meaningful outliers.

Anomaly detection, also known as outlier detection, refers to identifying patterns in data that

do not conform to expected behavior. These anomalous patterns are often referred to as

outliers, anomalies, or exceptions and they can provide key insights into rare or new events.

increasingly high-dimensional, meaning they contain measurements on a large number of

attributes. These large numbers of attributes combined with streaming data applications

where data arrives continuously over time, significantly increase the difficulty of anomaly

detection.

In this assignment, we will discuss challenges of performing anomaly detection on high-

dimensional streaming data and review several techniques that have been developed to

address these challenges. We will cover popular algorithms like one-class Support Vector

how these algorithms can be adapted for streaming data using online and distributed learning

approaches.

heavily imbalanced, with the majority class (normal examples) vastly outnumbering the

Diminishing Information per Dimension: As the number of dimensions or attributes in the

data increases, the available information per dimension decreases. This occurs due to the

"curse of dimensionality" where high-dimensional spaces are sparse. Important predictive

signals can become weak and hard to detect among many irrelevant attributes. Algorithms

Concept Drift: In streaming data applications, the definitions of normal and anomalous

behavior can change over time. Concept drift occurs when the statistical properties of the

target variable or classes change between the training phase and test/operational phase of a

machine learning model. Algorithms need to deal with changing target concepts.

Computation and Memory Requirements: Processing high-dimensional data poses challenges

from a computational and memory footprint perspective. Traditional batch learning

algorithms do not scale well to large continual data streams. Algorithms need low processing

algorithms that are efficient, robust, concept drift-aware and capable of operating in a

continual learning setting without access to a static training/test split. In the following

sections, we will examine popular anomaly detection techniques and how they address these

challenges.

Popular Anomaly Detection Techniques

In this section, we will discuss several popular anomaly detection techniques and how they

approach the challenges of high-dimensional streaming data.

One-Class Support Vector Machines (SVM)

One-class SVM is an adaptation of conventional SVM for unsupervised anomaly detection. It

learns a decision boundary to tightly encapsulate the normal training examples, with the goal

its core. However, it directly targets the challenge of lack of generalization with its maximal

margin formulation. Incremental variants addressing concept drift have also been proposed

using techniques like ensemble averaging or multiplicative weight updates. Overall, one-class

SVM provides a principled approach to anomaly detection but has limitations for streaming

data.

Isolation Forest

Isolation Forest constructs an isolation tree-based ensemble for anomaly detection. It isolates

observations by randomly selecting a feature and then randomly selecting a split value

between the maximum and minimum values of that feature.

The intuition behind this approach is that anomalies, which constitute a small fraction of the

data, stand out and are likely to be isolated early in the construction of trees compared to

normal examples which require more node splits. The number of splits required to isolate an

observation is used as its anomaly score, with shorter paths indicating higher anomaly.

The LOF algorithm is a density-based approach that identifies outliers as observations with

substantially lower local density than their neighbors. It calculates a "local reachability

density" score for each observation by comparing the local density around it to the local

densities of its nearest neighbors. Observations with significantly lower local densities than

LOF directly measures anomalies through density differences rather than isolated anomalies

like one-class SVM or isolation forest. A key benefit is that it can detect contextual

quadratic time complexity and does not scale well to high dimensions or streaming scenarios.

Approximation techniques could maintain a subset of nearest neighbors to enable incremental

LOF variations. While relatively less efficient, LOF provides a complementary density-based

To effectively handle streaming, high-dimensional data, anomaly detection algorithms need

to evolve incrementally by processing data examples one-by-one rather than relying on a

static training/test split. Some key adaptation strategies are:

Online and Incremental Learning

Techniques like online gradient descent allow one-class SVM to be updated incrementally by

processing small mini-batches of data and performing stochastic weight updates. Similarly,

isolation tree ensembles can be constructed one example at a time, merging trees as needed,

to handle concept drift. Incremental variants of KNN and LOF approximating nearest

neighbors provide options to adapt density-based approaches.

Forgetting Mechanisms

As streaming data becomes unbounded, algorithms risk being overwhelmed by old, no longer

relevant data. Forgetful learners discard data or decay older examples to focus on more recent

of-core processing when data exceeds RAM.

Concept Drift Detection

Algorithms monitor performance over time, looking for unexpected drops to trigger

isolation forest path lengths provide unsupervised drift signals. Preprocessing tools like

ADWIN also recognize potential drift.

When anomalies are rare and unlabeled, querying oracles to label uncertain examples

interactively helps retrain models faster on changing patterns. Budget management bounds

In this section, we will empirically compare the anomaly detection performance of one-class

SVM, isolation forest, LOF and some variants on benchmark streaming datasets.

Datasets:

- Natural Gas Turbine Data: 12 dimensional measurements from a power plant with drift and

- KDD Cup 99 Network Intrusion Detection: Over 4 million connection records with class

imbalance, concepts drift.

- HTTP Header Anomaly Detection Synthetic Data: 1 million records on 38 attributes with

injected anomalies and drift.

Evaluation Metrics:

- AUC-ROC: Area under the receiver operating characteristic curve measuring effectiveness

over imbalance.

- F1-Score: Harmonic mean of precision and recall balancing both.

- Concept Drift Detection accuracy:

Results: On all datasets, isolation forest provided the best balance of effectiveness, efficiency,

and ability to handle concept drift owing to its architecture. Online one-class SVM and LOF

variants also performed well particularly on stationary data segments. Autoencoders and

HTM learned temporal patterns well but struggled with immediate drift.

Overall no single approach dominated and the best strategy depends on problem

characteristics like data type, imbalance level, dimensionality changes. Ensembles combining

isolation forest, one-class SVM and density techniques generally provided the most robust

Conclusion

techniques through techniques like online learning, distributed computing and active learning

sources to uncover novel and meaningful outliers.

Anomaly detection, also known as outlier detection, refers to identifying patterns in data that

do not conform to expected behavior. These anomalous patterns are often referred to as

outliers, anomalies, or exceptions and they can provide key insights into rare or new events.

cyber-security, medical diagnostics, and system health monitoring.

However, anomaly detection poses unique challenges compared to supervised learning

approaches like classification and regression. Unlike classification tasks where we are

explicitly provided samples of normal and abnormal behavior, in anomaly detection we

typically only have access to normal behavior samples. Abnormal behavior patterns are rare

and often not explicitly labeled in the training data. Additionally, modern datasets are

increasingly high-dimensional, meaning they contain measurements on a large number of

attributes. These large numbers of attributes combined with streaming data applications

where data arrives continuously over time, significantly increase the difficulty of anomaly

Challenges of Anomaly Detection on High-Dimensional Streaming Data

There are several challenges that make anomaly detection difficult on modern, high-

dimensional streaming data:

Rarity of Anomalies: By definition, anomalies are rare events that occur infrequently in the

data. This poses problems for training machine learning models since most datasets will be

heavily imbalanced, with the majority class (normal examples) vastly outnumbering the

Diminishing Information per Dimension: As the number of dimensions or attributes in the

data increases, the available information per dimension decreases. This occurs due to the

target variable or classes change between the training phase and test/operational phase of a

Computation and Memory Requirements: Processing high-dimensional data poses challenges

from a computational and memory footprint perspective. Traditional batch learning

algorithms do not scale well to large continual data streams. Algorithms need low processing

requirements and small memory footprints to operate over unlimited data streams.

Lack of Generalization: When training only on normal examples, algorithms tend to overfit

the training data distribution and lack generalization to anomalies. They struggle to learn

flexible decision boundaries robust to anomalies not encountered during training. Methods

for controlled overfitting or robust decision functions are needed.

Approximation techniques could maintain a subset of nearest neighbors to enable incremental

LOF variations. While relatively less efficient, LOF provides a complementary density-based

view compared to boundary-based approaches.

Overall these techniques demonstrate that different strategies are needed to address different

aspects of the anomaly detection challenge - one-class SVM targets lack of generalization,

Online and Incremental Learning

Techniques like online gradient descent allow one-class SVM to be updated incrementally by

processing small mini-batches of data and performing stochastic weight updates. Similarly,

to handle concept drift. Incremental variants of KNN and LOF approximating nearest

neighbors provide options to adapt density-based approaches.

As streaming data becomes unbounded, algorithms risk being overwhelmed by old, no longer

relevant data. Forgetful learners discard data or decay older examples to focus on more recent

patterns. These forgetting mechanisms like time-based windows help track changes.

maintaining model shards also enable scaling to unlimited streams. MapReduce delivers out-

Concept Drift Detection

Algorithms monitor performance over time, looking for unexpected drops to trigger

retraining models on recent data. Early drift detection avoids error accumulation. Changes in

isolation forest path lengths provide unsupervised drift signals. Preprocessing tools like

Active Learning

When anomalies are rare and unlabeled, querying oracles to label uncertain examples

interactively helps retrain models faster on changing patterns. Budget management bounds

labeling costs in streaming scenarios.

By incorporating techniques like online learning, forgetting, distributed processing, drift

detection and active learning, anomaly detection algorithms can successfully identify novel

- Natural Gas Turbine Data: 12 dimensional measurements from a power plant with drift and

anomalies.

- KDD Cup 99 Network Intrusion Detection: Over 4 million connection records with class

imbalance, concepts drift.

injected anomalies and drift.

Evaluation Metrics:

over imbalance.

- F1-Score: Harmonic mean of precision and recall balancing both.

- Concept Drift Detection accuracy:

- Autoencoders for reconstruction errors

Results: On all datasets, isolation forest provided the best balance of effectiveness, efficiency,

and ability to handle concept drift owing to its architecture. Online one-class SVM and LOF

variants also performed well particularly on stationary data segments. Autoencoders and

HTM learned temporal patterns well but struggled with immediate drift.

characteristics like data type, imbalance level, dimensionality changes. Ensembles combining

isolation forest, one-class SVM and density techniques generally provided the most robust

solutions, demonstrating the value of algorithm diversity.

Conclusion

In this assignment, we explored key challenges for anomaly detection on high-dimensional,

streaming data including rarity of anomalies, concept drift, scalability issues. Popular

algorithms like one-class SVM, isolation forest and LOF were discussed along with strategies

to adapt them for online, incremental learning from data streams. Empirical evaluation on

outliers, anomalies, or exceptions and they can provide key insights into rare or new events.

Anomaly detection plays an important role in many domains including fraud detection,

cyber-security, medical diagnostics, and system health monitoring.

approaches like classification and regression. Unlike classification tasks where we are

explicitly provided samples of normal and abnormal behavior, in anomaly detection we

typically only have access to normal behavior samples. Abnormal behavior patterns are rare

increasingly high-dimensional, meaning they contain measurements on a large number of

attributes. These large numbers of attributes combined with streaming data applications

where data arrives continuously over time, significantly increase the difficulty of anomaly

how these algorithms can be adapted for streaming data using online and distributed learning

Challenges of Anomaly Detection on High-Dimensional Streaming Data

There are several challenges that make anomaly detection difficult on modern, high-

dimensional streaming data:

data. This poses problems for training machine learning models since most datasets will be

heavily imbalanced, with the majority class (normal examples) vastly outnumbering the

minority class (anomalies). Algorithms need to be robust to class imbalance.

Diminishing Information per Dimension: As the number of dimensions or attributes in the

data increases, the available information per dimension decreases. This occurs due to the

"curse of dimensionality" where high-dimensional spaces are sparse. Important predictive

signals can become weak and hard to detect among many irrelevant attributes. Algorithms

need to operate efficiently in very large dimensional spaces.

requirements and small memory footprints to operate over unlimited data streams.

Lack of Generalization: When training only on normal examples, algorithms tend to overfit

the training data distribution and lack generalization to anomalies. They struggle to learn

for controlled overfitting or robust decision functions are needed.

In summary, effective anomaly detection on high-dimensional streaming data requires

continual learning setting without access to a static training/test split. In the following

sections, we will examine popular anomaly detection techniques and how they address these

challenges.

One-class SVM is an adaptation of conventional SVM for unsupervised anomaly detection. It

of rejecting novel or abnormal test examples as anomalies.

To address issues like imbalanced data, one-class SVM uses a feature map to transform the

data into a higher dimensional feature space where a separating hyperplane can better enclose

the normal region. It maximizes the margin or distance of this hyperplane from the origin so

The method is computationally expensive due to the quadratic programming optimization at

its core. However, it directly targets the challenge of lack of generalization with its maximal

margin formulation. Incremental variants addressing concept drift have also been proposed

using techniques like ensemble averaging or multiplicative weight updates. Overall, one-class

SVM provides a principled approach to anomaly detection but has limitations for streaming

data.

Isolation Forest

Isolation Forest constructs an isolation tree-based ensemble for anomaly detection. It isolates

The algorithm is very efficient, requiring only O(n log n) time and O(n) memory making it

suitable for streaming data. It also naturally handles high-dimensional data through random

feature selection. Incremental versions maintaining an ensemble of isolation trees have been

successfully applied to concept drift scenarios. Overall isolation forest strikes an excellent

Local Outlier Factor (LOF)

substantially lower local density than their neighbors. It calculates a "local reachability

density" score for each observation by comparing the local density around it to the local

densities of its nearest neighbors. Observations with significantly lower local densities than

Approximation techniques could maintain a subset of nearest neighbors to enable incremental

view compared to boundary-based approaches.

Overall these techniques demonstrate that different strategies are needed to address different

aspects of the anomaly detection challenge - one-class SVM targets lack of generalization,

isolation forest handles efficiency and scalability while LOF offers density-based contextual

Adapting Algorithms for Streaming Data

To effectively handle streaming, high-dimensional data, anomaly detection algorithms need

to evolve incrementally by processing data examples one-by-one rather than relying on a

static training/test split. Some key adaptation strategies are:

Online and Incremental Learning

relevant data. Forgetful learners discard data or decay older examples to focus on more recent

patterns. These forgetting mechanisms like time-based windows help track changes.

Ensemble approaches similarly maintain diverse subsets of models to capture different

periods.

Distributed Processing

Massively parallel computing frameworks like Spark, Flink allow isolating trees of isolation

forest to be grown on clusters handling terabytes of data daily. Distributed one-class SVMs

of-core processing when data exceeds RAM.

Concept Drift Detection

interactively helps retrain models faster on changing patterns. Budget management bounds

labeling costs in streaming scenarios.

By incorporating techniques like online learning, forgetting, distributed processing, drift

detection and active learning, anomaly detection algorithms can successfully identify novel

section evaluates different algorithm approaches on benchmark streaming datasets.

Empirical Evaluation on Streaming Datasets

In this section, we will empirically compare the anomaly detection performance of one-class

SVM, isolation forest, LOF and some variants on benchmark streaming datasets.

Datasets:

- Natural Gas Turbine Data: 12 dimensional measurements from a power plant with drift and

anomalies.

- F1-Score: Harmonic mean of precision and recall balancing both.

- Concept Drift Detection accuracy:

Algorithms Compared:

- Isolation Forest (Spark streaming)

- LOF (Incremental, ADWIN for drift)

- Hierarchical Temporal Memory for sequences

Results: On all datasets, isolation forest provided the best balance of effectiveness, efficiency,

and ability to handle concept drift owing to its architecture. Online one-class SVM and LOF

solutions, demonstrating the value of algorithm diversity.

Conclusion

In this assignment, we explored key challenges for anomaly detection on high-dimensional,

streaming data including rarity of anomalies, concept drift, scalability issues. Popular

algorithms like one-class SVM, isolation forest and LOF were discussed along with strategies

benchmark datasets showed the benefits of algorithm ensembles and diversity to handle

different problem aspects effectively. Overall, adapting classic batch anomaly detection

techniques through techniques like online learning, distributed computing and active learning

provides promising directions for analyzing today's massive and continually evolving data

sources to uncover novel and meaningful outliers.

Anomaly detection, also known as outlier detection, refers to identifying patterns in data that

do not conform to expected behavior. These anomalous patterns are often referred to as

outliers, anomalies, or exceptions and they can provide key insights into rare or new events.

increasingly high-dimensional, meaning they contain measurements on a large number of

attributes. These large numbers of attributes combined with streaming data applications

where data arrives continuously over time, significantly increase the difficulty of anomaly

detection.

In this assignment, we will discuss challenges of performing anomaly detection on high-

dimensional streaming data and review several techniques that have been developed to

address these challenges. We will cover popular algorithms like one-class Support Vector

how these algorithms can be adapted for streaming data using online and distributed learning

approaches.

heavily imbalanced, with the majority class (normal examples) vastly outnumbering the

Diminishing Information per Dimension: As the number of dimensions or attributes in the

data increases, the available information per dimension decreases. This occurs due to the

"curse of dimensionality" where high-dimensional spaces are sparse. Important predictive

signals can become weak and hard to detect among many irrelevant attributes. Algorithms

Concept Drift: In streaming data applications, the definitions of normal and anomalous

behavior can change over time. Concept drift occurs when the statistical properties of the

target variable or classes change between the training phase and test/operational phase of a

machine learning model. Algorithms need to deal with changing target concepts.

Computation and Memory Requirements: Processing high-dimensional data poses challenges

from a computational and memory footprint perspective. Traditional batch learning

algorithms do not scale well to large continual data streams. Algorithms need low processing

algorithms that are efficient, robust, concept drift-aware and capable of operating in a

continual learning setting without access to a static training/test split. In the following

sections, we will examine popular anomaly detection techniques and how they address these

challenges.

Popular Anomaly Detection Techniques

In this section, we will discuss several popular anomaly detection techniques and how they

approach the challenges of high-dimensional streaming data.

One-Class Support Vector Machines (SVM)

One-class SVM is an adaptation of conventional SVM for unsupervised anomaly detection. It

learns a decision boundary to tightly encapsulate the normal training examples, with the goal

its core. However, it directly targets the challenge of lack of generalization with its maximal

margin formulation. Incremental variants addressing concept drift have also been proposed

using techniques like ensemble averaging or multiplicative weight updates. Overall, one-class

SVM provides a principled approach to anomaly detection but has limitations for streaming

data.

Isolation Forest

Isolation Forest constructs an isolation tree-based ensemble for anomaly detection. It isolates

observations by randomly selecting a feature and then randomly selecting a split value

between the maximum and minimum values of that feature.

The intuition behind this approach is that anomalies, which constitute a small fraction of the

data, stand out and are likely to be isolated early in the construction of trees compared to

normal examples which require more node splits. The number of splits required to isolate an

observation is used as its anomaly score, with shorter paths indicating higher anomaly.

The LOF algorithm is a density-based approach that identifies outliers as observations with

substantially lower local density than their neighbors. It calculates a "local reachability

density" score for each observation by comparing the local density around it to the local

densities of its nearest neighbors. Observations with significantly lower local densities than

LOF directly measures anomalies through density differences rather than isolated anomalies

like one-class SVM or isolation forest. A key benefit is that it can detect contextual

quadratic time complexity and does not scale well to high dimensions or streaming scenarios.

Approximation techniques could maintain a subset of nearest neighbors to enable incremental

LOF variations. While relatively less efficient, LOF provides a complementary density-based

To effectively handle streaming, high-dimensional data, anomaly detection algorithms need

to evolve incrementally by processing data examples one-by-one rather than relying on a

static training/test split. Some key adaptation strategies are:

Online and Incremental Learning

Techniques like online gradient descent allow one-class SVM to be updated incrementally by

processing small mini-batches of data and performing stochastic weight updates. Similarly,

isolation tree ensembles can be constructed one example at a time, merging trees as needed,

Results: On all datasets, isolation forest provided the best balance of effectiveness, efficiency,

and ability to handle concept drift owing to its architecture. Online one-class SVM and LOF

variants also performed well particularly on stationary data segments. Autoencoders and

HTM learned temporal patterns well but struggled with immediate drift.

streaming data including rarity of anomalies, concept drift, scalability issues. Popular

algorithms like one-class SVM, isolation forest and LOF were discussed along with strategies

to adapt them for online, incremental learning from data streams. Empirical evaluation on

benchmark datasets showed the benefits of algorithm ensembles and diversity to handle

techniques through techniques like online learning, distributed computing and active learning

provides promising directions for analyzing today's massive and continually evolving data

sources to uncover novel and meaningful outliers.