DROPS

Document

DOI: 10.4230/OASIcs.ICCSW.2018.8

Anomaly Detection for Big Data Technologies

Authors: Ahmad Alnafessah and Giuliano Casale

Published in: OASIcs, Volume 66, 2018 Imperial College Computing Student Workshop (ICCSW 2018)

Abstract

The main goal of this research is to contribute to automated performance anomaly detection for large-scale and complex distributed systems, especially for Big Data applications within cloud computing. The main points that we will investigate are: - Automated detection of anomalous performance behaviors by finding the relevant performance metrics with which to characterize behavior of systems. - Performance anomaly localization: To pinpoint the cause of a performance anomaly due to internal or external faults. - Investigation of the possibility of anomaly prediction. Failure prediction aims to determine the possible occurrences of catastrophic events in the near future and will enable system developers to utilize effective monitoring solutions to guarantee system availability. - Assessment for the potential of hybrid methods that combine machine learning with traditional methods used in performance for anomaly detection. The topic of this research proposal will offer me the opportunity to more deeply apply my interest in the field of performance anomaly detection and prediction by investigating and using novel optimization strategies. In addition, this research provides a very interesting case of utilizing the anomaly detection techniques in a large-scale Big Data and cloud computing environment. Among the various Big Data technologies, in-memory processing technology like Apache Spark has become widely adopted by industries as result of its speed, generality, ease of use, and compatibility with other Big Data systems. Although Spark is developing gradually, currently there are still shortages in comprehensive performance analyses that specifically build for Spark and are used to detect performance anomalies. Therefore, this raises my interest in addressing this challenge by investigating new hybrid learning techniques for anomaly detection in large-scale and complex systems, especially for in-memory processing Big Data platforms within cloud computing.

Cite as

Ahmad Alnafessah and Giuliano Casale. Anomaly Detection for Big Data Technologies. In 2018 Imperial College Computing Student Workshop (ICCSW 2018). Open Access Series in Informatics (OASIcs), Volume 66, p. 8:1, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2019)

Copy BibTex To Clipboard

@InProceedings{alnafessah_et_al:OASIcs.ICCSW.2018.8,
  author =	{Alnafessah, Ahmad and Casale, Giuliano},
  title =	{{Anomaly Detection for Big Data Technologies}},
  booktitle =	{2018 Imperial College Computing Student Workshop (ICCSW 2018)},
  pages =	{8:1--8:1},
  series =	{Open Access Series in Informatics (OASIcs)},
  ISBN =	{978-3-95977-097-2},
  ISSN =	{2190-6807},
  year =	{2019},
  volume =	{66},
  editor =	{Pirovano, Edoardo and Graversen, Eva},
  publisher =	{Schloss Dagstuhl -- Leibniz-Zentrum f{\"u}r Informatik},
  address =	{Dagstuhl, Germany},
  URL =		{https://drops.dagstuhl.de/entities/document/10.4230/OASIcs.ICCSW.2018.8},
  URN =		{urn:nbn:de:0030-drops-101899},
  doi =		{10.4230/OASIcs.ICCSW.2018.8},
  annote =	{Keywords: Performance anomalies, Apache Spark, Neural Network, Resilient Distributed Dataset (RDD)}
}

Document

DOI: 10.4230/DagSemProc.07461.8

Interarrival Times Characterization and Fitting for Markovian Traffic Analysis

Authors: Giuliano Casale, Eddy Z. Zhang, and Evgenia Smirni

Published in: Dagstuhl Seminar Proceedings, Volume 7461, Numerical Methods for Structured Markov Chains (2008)

Abstract

We propose a traffic fitting algorithm for Markovian Arrival Processes (MAPs) that can capture statistics of any order of interarrival times. By studying real traffic traces, we show that matching higher order properties, in addition to first and second order descriptors, results in increased queueing prediction accuracy with respect to other algorithms that only match the mean, coefficient of variation, and autocorrelations. The result promotes the idea of modeling traffic traces using the interarrival time process instead of the counting process that is more frequently employed in previous work, but for which higher order moments are difficult to manipulate. We proceed by first characterizing the general properties of MAPs using a spectral approach. Based on this characterization, we show how different MAP processes can be combined together using Kronecker products to define a larger MAP with predefined properties of interarrival times. We then devise an algorithm that is based on this Kronecker composition and can accurately fit traffic traces. The algorithm employs nonlinear optimization programs that can be customized to fit an arbitrary number of moments and to meet the desired cost-accuracy tradeoff. Numerical results of the fitting algorithm on real HTTP and TCP traffic data, such as the Bellcore Aug89 trace, indicate that the proposed fitting methods achieve increased prediction accuracy with respect to other state-of-the-art fitting methods.

Cite as

Giuliano Casale, Eddy Z. Zhang, and Evgenia Smirni. Interarrival Times Characterization and Fitting for Markovian Traffic Analysis. In Numerical Methods for Structured Markov Chains. Dagstuhl Seminar Proceedings, Volume 7461, pp. 1-8, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2008)

Copy BibTex To Clipboard

@InProceedings{casale_et_al:DagSemProc.07461.8,
  author =	{Casale, Giuliano and Zhang, Eddy Z. and Smirni, Evgenia},
  title =	{{Interarrival Times Characterization and Fitting for Markovian Traffic Analysis}},
  booktitle =	{Numerical Methods for Structured Markov Chains},
  pages =	{1--8},
  series =	{Dagstuhl Seminar Proceedings (DagSemProc)},
  ISSN =	{1862-4405},
  year =	{2008},
  volume =	{7461},
  editor =	{Dario Bini and Beatrice Meini and Vaidyanathan Ramaswami and Marie-Ange Remiche and Peter Taylor},
  publisher =	{Schloss Dagstuhl -- Leibniz-Zentrum f{\"u}r Informatik},
  address =	{Dagstuhl, Germany},
  URL =		{https://drops.dagstuhl.de/entities/document/10.4230/DagSemProc.07461.8},
  URN =		{urn:nbn:de:0030-drops-13908},
  doi =		{10.4230/DagSemProc.07461.8},
  annote =	{Keywords: MAP fitting, interarrival time process, higher-order moments}
}

Search Results

Documents authored by Casale, Giuliano

Anomaly Detection for Big Data Technologies

Abstract

Cite as

Interarrival Times Characterization and Fitting for Markovian Traffic Analysis

Abstract

Cite as

Thanks for your feedback!

Could not send message