Page-based anomaly detection in large scale web clusters using adaptive MapReduce

Junsup Lee, Sungdeok Cha

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

While anomaly detection systems typically work on single server, most commercial web sites operate cluster environments, and user queries trigger transactions scattered through multiple servers. For this reason, anomaly detectors in a same server farm should communicate with each other to integrate their partial profile. In this paper, we describe a real-time distributed anomaly detection system that can deal with over one billion transactions per day. In our system, base on Google MapReduce algorithm, an anomaly detector in each node shares profiles of user behaviors and propagates intruder information to reduce false alarms. We evaluated our system using web log data from www.microsoft.com. The web log data, about 250GB in size, contains over one billion transactions recorded in a day.

Original languageEnglish
Title of host publicationLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Pages404-405
Number of pages2
Volume5230 LNCS
DOIs
Publication statusPublished - 2008 Dec 1
EventRecent Advances in Intrusion Detection - 11th International Symposium, RAID 2008, Proceedings - Cambridge, MA, United States
Duration: 2008 Sep 152008 Sep 17

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume5230 LNCS
ISSN (Print)03029743
ISSN (Electronic)16113349

Other

OtherRecent Advances in Intrusion Detection - 11th International Symposium, RAID 2008, Proceedings
CountryUnited States
CityCambridge, MA
Period08/9/1508/9/17

Fingerprint

MapReduce
Anomaly Detection
Servers
Transactions
Datalog
Anomaly
Detectors
Server
Detector
Farms
Single Server
Websites
User Behavior
False Alarm
Trigger
Integrate
Query
Real-time
Partial
Vertex of a graph

ASJC Scopus subject areas

  • Computer Science(all)
  • Theoretical Computer Science

Cite this

Lee, J., & Cha, S. (2008). Page-based anomaly detection in large scale web clusters using adaptive MapReduce. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 5230 LNCS, pp. 404-405). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 5230 LNCS). https://doi.org/10.1007/978-3-540-87403-4_28

Page-based anomaly detection in large scale web clusters using adaptive MapReduce. / Lee, Junsup; Cha, Sungdeok.

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 5230 LNCS 2008. p. 404-405 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 5230 LNCS).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Lee, J & Cha, S 2008, Page-based anomaly detection in large scale web clusters using adaptive MapReduce. in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol. 5230 LNCS, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 5230 LNCS, pp. 404-405, Recent Advances in Intrusion Detection - 11th International Symposium, RAID 2008, Proceedings, Cambridge, MA, United States, 08/9/15. https://doi.org/10.1007/978-3-540-87403-4_28
Lee J, Cha S. Page-based anomaly detection in large scale web clusters using adaptive MapReduce. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 5230 LNCS. 2008. p. 404-405. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-540-87403-4_28
Lee, Junsup ; Cha, Sungdeok. / Page-based anomaly detection in large scale web clusters using adaptive MapReduce. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 5230 LNCS 2008. pp. 404-405 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{b5e8637d6ba84e4f8807516df286b6fa,
title = "Page-based anomaly detection in large scale web clusters using adaptive MapReduce",
abstract = "While anomaly detection systems typically work on single server, most commercial web sites operate cluster environments, and user queries trigger transactions scattered through multiple servers. For this reason, anomaly detectors in a same server farm should communicate with each other to integrate their partial profile. In this paper, we describe a real-time distributed anomaly detection system that can deal with over one billion transactions per day. In our system, base on Google MapReduce algorithm, an anomaly detector in each node shares profiles of user behaviors and propagates intruder information to reduce false alarms. We evaluated our system using web log data from www.microsoft.com. The web log data, about 250GB in size, contains over one billion transactions recorded in a day.",
author = "Junsup Lee and Sungdeok Cha",
year = "2008",
month = "12",
day = "1",
doi = "10.1007/978-3-540-87403-4_28",
language = "English",
isbn = "354087402X",
volume = "5230 LNCS",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
pages = "404--405",
booktitle = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

}

TY - GEN

T1 - Page-based anomaly detection in large scale web clusters using adaptive MapReduce

AU - Lee, Junsup

AU - Cha, Sungdeok

PY - 2008/12/1

Y1 - 2008/12/1

N2 - While anomaly detection systems typically work on single server, most commercial web sites operate cluster environments, and user queries trigger transactions scattered through multiple servers. For this reason, anomaly detectors in a same server farm should communicate with each other to integrate their partial profile. In this paper, we describe a real-time distributed anomaly detection system that can deal with over one billion transactions per day. In our system, base on Google MapReduce algorithm, an anomaly detector in each node shares profiles of user behaviors and propagates intruder information to reduce false alarms. We evaluated our system using web log data from www.microsoft.com. The web log data, about 250GB in size, contains over one billion transactions recorded in a day.

AB - While anomaly detection systems typically work on single server, most commercial web sites operate cluster environments, and user queries trigger transactions scattered through multiple servers. For this reason, anomaly detectors in a same server farm should communicate with each other to integrate their partial profile. In this paper, we describe a real-time distributed anomaly detection system that can deal with over one billion transactions per day. In our system, base on Google MapReduce algorithm, an anomaly detector in each node shares profiles of user behaviors and propagates intruder information to reduce false alarms. We evaluated our system using web log data from www.microsoft.com. The web log data, about 250GB in size, contains over one billion transactions recorded in a day.

UR - http://www.scopus.com/inward/record.url?scp=56749160523&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=56749160523&partnerID=8YFLogxK

U2 - 10.1007/978-3-540-87403-4_28

DO - 10.1007/978-3-540-87403-4_28

M3 - Conference contribution

AN - SCOPUS:56749160523

SN - 354087402X

SN - 9783540874027

VL - 5230 LNCS

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 404

EP - 405

BT - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

ER -