This website uses cookies to ensure you have the best experience. Learn more

Log Analysis

2121 words - 9 pages

Log is a file that records the events which happens while an operating system or software runs [1]. It may include any activity such as information about a simple keystroke, the complete record of communication between two machines, system errors, inter-process communication, update events, server activities, client sessions, browsing history, etc. Logs provide a good insight into various states of a system at any instant and their analytical and statistical study can manage systems and mine useful knowledge about a user on various aspects. Log data is voluminous, growing at a very fast rate, with varying structure across various applications, usages, servers, etc. It possesses the key ...view middle of the document...

Three approaches that had been worked upon are term level analysis, query level analysis and session level analysis. But it is need of time, to take a step ahead and introduce some intelligence to the analyzer and take advantage of complex models to view the data with various perspectives. Big Data analysis and data mining are the answer to such need. The upcoming sections will discuss how Big Data technology and data mining algorithms facilitate handling the voluminous and rapidly growing log data.
Big Data is an emerging area of interest for researchers, it being a powerful source for predicting unforeseen results and an able supporter of decision. Log data is voluminous, growing at a very fast rate, with varying structure across various applications, usages, servers, etc. Thus it possesses the key characteristics of the Big Data.
Heterogeneity of logs can be handled by NoSQL databases such as Hadoop, Dynamo, BigTables etc., because they provide a schema-less and scalable model [4]. They are primarily of four types: key-value pair oriented, column oriented, document oriented and graph databases.
1.1.5. Challenges in Log Analysis
a. Loss of semantic while writing log messages make it difficult to retrieve correct meaning.
b. Log quality and excessive logging can put adverse effect on analysis.
c. Sampling based logging can miss out on crucial and rare events.
d. Power of analysis is limited by the information in logs [5].
e. A mechanism to distinguish adversary from a legitimate user performing correct operation.
1.2. Introduction to Big Data
In 2010, Apache Hadoop defined it as, “Datasets which could not be captured, managed, and processed by general computers within an acceptable scope” [6]. This interpretation of Big Data varies to three levels: Big Data, Very Big Data and Massive Data. It is a conceptual idea, where other than size, rate of growth of incoming data, types of data, completeness of data and many other factors play an equally important role. It is a trending domain where lot of undiscovered potential lies and is expected to be explored in upcoming years. Figure-3 shows some important results presented in the webinar “ Big Data Opportunities in Vertical Industries” hosted by Gartner in July 2012 [7].
Data can vary from terabytes to petabytes or even more. So the requirements for the analysis depend highly on the available computing infrastructure and scalable algorithms. That defines many parameters for evaluating a data set as Big Data. Figure-4 shows the key parameters for such evaluation, which include volume, variety, velocity, veracity, viscosity, value.
1.2.1. Handling Big Data
a. Data Generation
The input data which may include day-to-day data generated from enterprises, sensor nodes or IoTs, Bio-medical analysis, and other fields etc.
b. Data Acquisition
The generated data is not worth if not acquired well in time, particularly in case of time sensitive data. Thus, collection and transmission of data to the data centres...

Find Another Essay On Log Analysis

A Professional Development Log Essay

601 words - 3 pages their personal performance. A Professional Development Log would be used to record the dates, actions, the opportunities, and responses from the staff member and their supervisor on effect of their development during the year. Recording examples: • Annotations of improved knowledge or abilities and how they were efficient. • Development towards goals. • Annotations where new knowledge / abilities could be applied in the future.

Mocking Jay reading log Essay

672 words - 3 pages In Mocking Jay by Suzanne Collins, Katniss takes on her new position as the rebel of district 13 which is a secret district that is rebelling against the capitol. Katniss and a few of her friends were lucky enough to escape from the second hunger games but, Peeta was not so lucky as he was captured by the capitol. Peeta warns district 13 about the capitol’s plans of bombing district 13, and Katniss and the rest of district 13 retreat underground

Learning Log Chats

954 words - 4 pages To begin, each Friday during group chat my learning log jumps into many directions. I am not sure if I would have thought about these topics by myself but the fact someone else is said something interesting about their findings and I would want to explore if it has any relevant to my topic must be good. This project is very unknowing because no set direction is wrong or right and this process of learning is so different from anything I did

Reading Response Log

1433 words - 6 pages 1) P.2 But on the last day an astonishing number of English heavies opened up on us with high explosive, drumming ceaselessly on our position, so that we suffered severely and came back only eighty strong. Reading Response: I wonder how aggressive a war could be. This tells me that a war can be ruthless, with each nation trying to destroy others. A war is only going to make things worse not better since there is always people dying everyday

The Learning Log

990 words - 4 pages The context of the room was enclosed and as a result we crowded chairs around the carer. This was not ideal, as the carer had never met us before, and being surrounded can be daunting and overwhelming. We decided to ensure we asked whether she felt comfortable and wanted water before we begun. This worked well, as it acted as an icebreaker, initiated our rapport building and expressed our consideration. We also attempted to move the chairs to a

ISU Reading Log 2

821 words - 4 pages Analysis 1. In my novel Matched the major conflict is mostly resolved. I think this because the main conflict in the story is that she can't choose who she wants to be with as well as hiding everything she knows that since the society has made its existence completely and utterly illegal. In the novel, Cassia does choose which boy she wants, which ends up being Ky, but closer to the end of the novel, Cassia does a sort job that gets Ky

The Odyssey Travel Log

2001 words - 9 pages Log 1 • Upon leaving Troy, the winds of destiny sweep Odysseus and his man to Cicones. Deciding it’s a good idea to plunder, the natives attack Odysseus killing 72 • A real world connection here is the rape of Nanking. In 1937 Japanese soldiers slaughter the men of Nanking, China, Rape their women, and demolishes the land. However, in The Odyssey, the attackers are the ones who truly suffer. The men learned from this to always size up your

The Media: Web Log Post

1960 words - 8 pages people in a child and t eens’ life. As parents learn and better grasp this concept of thinking, it will allow them to understand more of how to help their victimized teen. After all, in this time of despair there is still hope for all of us. Works Cited Arnold, Dr. Elizabeth. "Cyber Bullying." Web log post. Wake Forest Baptist Health. Wake Forest Baptist Health, 2011. Web. 18 Dec. 2013.

"Scarlett Letter" (Hawthorne) reading log

4170 words - 17 pages CHAPTER 1- The Prison-Door-I found this one page introductory chapter to be very dramatic as it explained the historyof the colony by telling the history of the prison-door. From the looks of this chapter, this storywill probably be dragged out as much as possible.At first I thought the way Nathaniel Hawthorne approached the latter half of the lastparagraph was annoying, but when I read the chapter the second time, it appealed to me

Reading Log: "Struggle of Acceptance"

1043 words - 5 pages Reading Log: “Struggle of Acceptance” The Lost City by Alan Ehrenhalt, “The Love Song of J. Alfred Prufrock” by T.S. Eliot, and “My Two Lives” by Jhumpa Lahiri, are all great stories that have a connecting theme of “struggle of acceptance”. The short excerpt from The Lost City is about the past of an African-American neighborhood, called Bronzeville, during the 1950’s. Alan Ehrenhalt explains that the people in the neighborhood were close and

Journal Log Hamlet Act I

663 words - 3 pages 1.3-6) In lines 59-80 Polonious gives some fatherly advice to Laertes before his departure to France. Instead of giving him some actual advice from the heart, Polonious just spits out a bunch of clichés for Laertes to remember. "To thine own self be true" and "Costly thy habit as thy purse can buy" are some very typical bits of advice that Polonious gives him. This meeting is a very stereotypical conversation between father and son. It

Similar Essays

Analysis Of Large Log Files

10573 words - 42 pages Analysis of large log files Kasper Laursen s093078 Kongens Lyngby 2012 IMM-B.Sc.-2012-37 Technical University of Denmark Informatics and Mathematical Modelling Building 321, DK-2800 Kongens Lyngby, Denmark Phone +45 4525 3351, Fax +45 4588 2673 reception@imm.dtu.dk www.imm.dtu.dk IMM-B.Sc.-2012-37 Summary This thesis covers pattern recognition of large log files using clustering analysis in form of mini-batch K-means clustering and data

Learning Log Essay

2283 words - 10 pages design narrative environment like playwrights using design to wright in spaces. By narrative environment's design made stories can be continuous in a variety of spatial and temporal. creative process(pictures) 1. research>collect data pieces 2. analysis>select pieces and put them together 3. experimentation>try different combinations of shapes 4. collaboration>combined into a stage 5. subject knowledge>planning moving lines 6. narrative

Resarch Log Essay

838 words - 4 pages Assessment and Intervention Paper: The Smith Family Being that client(s) are unique and have diverse needs, therapeutic structures are not linear and areas that are being explored in therapy sessions would depend on the needs of the client(s) (Harper & Spellman, p. 102). However, for the purposes of this assignment I have decided to follow the structural pattern as offered by Morgan’s (2000) introduction to narrative therapy as a guide for how

Food Log Essay

966 words - 4 pages injuries, but consuming too much isn’t healthy for the body. Carbohydrates can be found in breads, rice, and pasta, and is one of the sources that provide us with energy. During the five days of food log, the worst nutritionally food I consumed was 2 slices of cheese pizza. The pizza contains the most amounts of fat, carbohydrate, sodium, cholesterol, calcium, and iron. Fat in pizza is saturated, meaning it can raise the level of cholesterol in