Logstract: A Decision Support Tool to Extract and Correlate Events from Application-based Log Files

Currently, modern companies and organizations are often limited in their capabilities to deliver high quality services without employing highly sophisticated IT infrastructures to support their final businesses. Sophisticated IT infrastructures, in turn, are usually accompanied by complex management challenges that often lead to increasing maintenance costs. A rational management of IT infrastructures becomes a critical issue for any organization that aims at keeping it at a financially stable state. In order to provide a systematic IT infrastructure management – and thus reduce management costs – the widely well-known Information Technology Infrastructure Library (ITIL) [1] presents a set of best practices and processes that helps organizations to properly maintain their IT infrastructures.

However, even by employing such best practices as ITIL, it remains very hard to maintain such complex IT infrastructures in an organized manner. In a real IT environment, mainly incomplete documentation, lack of data (such as about changes or monitoring of resources), and a limited understanding of IT infrastructure components’ dependencies [2] are eminent due to the dynamic nature of large data centers. Therefore, systems with the ability to identify relevant performance or misbehavioral issues (e.g., throughput degradations, critical changes, downtime), to understand the operation of this IT infrastructure, and to provide respective decision support to IT or operations managers become a key element to deliver high quality services.

In the context of large IT environments, Vito Tauriello performed a VA which he implemented a Decision Support Tool for Large IT Infrastructures to analyze traces related to an application (from a Swiss financial Institution). The implementation focused on taking these traces and identify changes or abnormalities. Traces used in the VA scope contains the following set of information:

• HTTP access-logs from entry-points servers (i.e., load balancers) of the IT infrastructure (requests coming from Swiss employees’ browser). These requests are only originated from the company’s intranet of Switzerland (i.e., the employees are not out of the company’s intranet domain).
• HTTP access-logs from the backend-system that delivers stock information (e.g., graphs, share prices, research) including payload and timestamps
• Statistic-logs from the backend-system (including information such as service-request, response-times for different components in the backend system, DB-times, requested service-names with some parameters)

In this BA, the work should focus on refining and evaluating what was developed in the context of the VA. Therefore, the BA is divided into four distinct parts:

• Identify possible enhancements for the data set analysis algorithm. These enhancements should be focused on performance (e.g., time to complete the algorithm) and results accuracy.
• Implement changes to enhance the data set analysis algorithm
• Evaluate the changes to quantify how the enhancements impacted in the overall results -- how accurate is the algorithm?
• Define generally applicable rules and guidelines on how to interpret results achieved

The traces data set are based on what was collected (from a Swiss financial Institution) from April 10, 2012 to April 19, 2012, with a duration of 10 days. During this period, a major infrastructure change happened: old servers have been replaced by new ones following the x86 architecture. Actually, this IT infrastructure change exceeded the period that the data set was collected, since it was run continuously from March 20, 2012 until April 20, 2012. However, it was documented that on April 19, 2012 it was the first day in which all incoming request traffic was fully directed to the new x86-based infrastructure.

References:

[1] IT Infrastructure Library: ITIL Service Transition, v. 3. London: The Stationery Office, 2007, 270 p.
[2] Distributed Management Task Force: Common Information Model. Available at: http://www.dmtf.org/standards/cim. Last visited on: Oct. 2013.