Digital Edition

SYS-CON.TV
The Rise of the Machine: Intelligent Log Analysis Is Here
Developers use log data to troubleshoot and investigate what affects or causes a problem with their applications

Application logs contain a massive repository of events and come in many different formats. They can have valuable information, but gaining useful insight can be difficult without the assistance of machine learning to help reveal critical problems.

Transaction logs can contain gigabytes of data and come in proprietary formats. Some applications even have separate consoles, and captured events differ by organization depending upon compliance requirements and other considerations. Centralized log management has made it easier to troubleshoot applications and investigate security incidents from one location, but the data still must be interpreted. That often involves complex mapping of key and value structures.

Log management used to be a dirty word in the enterprise. Just four years ago, a Verizon study determined that nearly 70 percent of security breach victims were sitting on logs teeming with sufficient evidence of active exploits. That was primarily because analysis was delayed and failed to provide effective insights. It can be a burdensome undertaking without the right tools.

Developers use log data to troubleshoot and investigate what affects or causes a problem with their applications, both during testing and production. That means processing a huge volume of data and search events to find a needle in a haystack. Logs might have information about where a problem occurred, which component crashed, or which system events had an effect on the application. Previously, much effort went into managing and analyzing searchable logs.

Security Information and Event Management (SIEM) solutions then evolved to make it easier to correlate log information and identify some types of notable events with simple search and visualization solutions. There are still many options available in this category; some are free and others are commercial solutions. Log analysis remains a very time consuming and exacting process, because the onus is on the developer or information security analyst to know exactly what they are looking for. A search query in this generation of SIEM tool often returns a flat list of results without prioritizing what's important to application or network. Just imagine using Google without page rank - results would be lost.

The Rise of the Machine
The latest generation of SIEM tools has more built-in intelligence to expedite the most time-consuming work. Semantic search automates the troubleshooting process by using advanced algorithms to uncover errors, risk factors, and other signs of problems. That is accomplished through a combination of text and semantic processing, statistical models and machine learning technologies.

A pre-tuned information model, which is derived from user searches and decision- making during analysis, can be created for SIEM for each scenario - from operations to compliance and testing. User searches are augmented by machine learning analytics to find meaningful events and insight on the log data, saving time.

That's because augmented search helps to profile and gain instant insight and intelligence from the data, giving the developers a bead on where to start and what happened. While augmented search can deliver useful info out-of-the-box, it keeps getting better with more user searches. The most advanced SIEM solutions will even work with any home grown or third-party application logs without any mapping.

Expect to see new entrants, because there's now an unfolding semantic revolution. Gartner's 2013 Magic Quadrant report for SIEM concluded, "We continue to see large companies that are re-evaluating SIEM vendors to replace SIEM technology associated with partial, marginal or failed deployments." Gartner recognized that intelligence matters, and suggested that analytics should uncover both known and unknown problems.

SIEM is evolving alongside semantics so that organizations can obtain value from the first event analyzed. It can take hours to find errors in log data manually, but automated search tools can pinpoint critical events within seconds, in context and with high accuracy.

About Haim Koshchitzky
Haim Koshchitzky is the Founder and CEO of XpoLog and has over 20 years of experience in complex technology development and software architecture. Prior to XpoLog, he spent several years as the tech lead for Mercury Interactive (acquired by HP) and other startups. He has a passion for data analytics and technology, and is also an avid marathon runner and Judo black belt.

In order to post a comment you need to be registered and logged in.

Register | Sign-in

Reader Feedback: Page 1 of 1



ADS BY GOOGLE
Subscribe to the World's Most Powerful Newsletters

ADS BY GOOGLE

CloudEXPO New York 2018, colocated with DXWorldEXPO New York 2018 will be held November 11-13, 2018,...
@DevOpsSummit at Cloud Expo, taking place November 12-13 in New York City, NY, is co-located with 22...
The best way to leverage your Cloud Expo presence as a sponsor and exhibitor is to plan your news an...
Digital Transformation: Preparing Cloud & IoT Security for the Age of Artificial Intelligence. As au...
Bill Schmarzo, author of "Big Data: Understanding How Data Powers Big Business" and "Big Data MBA: D...
The now mainstream platform changes stemming from the first Internet boom brought many changes but d...
"We began as LinuxAcademy.com about five years ago as a very small outfit. Since then we've transiti...
DevOpsSummit New York 2018, colocated with CloudEXPO | DXWorldEXPO New York 2018 will be held Novemb...
With 10 simultaneous tracks, keynotes, general sessions and targeted breakout classes, @CloudEXPO an...
Machine Learning helps make complex systems more efficient. By applying advanced Machine Learning te...
The Internet of Things will challenge the status quo of how IT and development organizations operate...
More and more companies are looking to microservices as an architectural pattern for breaking apart ...
With the proliferation of both SQL and NoSQL databases, organizations can now target specific fit-fo...
Andi Mann, Chief Technology Advocate at Splunk, is an accomplished digital business executive with e...
In his session at 21st Cloud Expo, Raju Shreewastava, founder of Big Data Trunk, provided a fun and ...
Adding public cloud resources to an existing application can be a daunting process. The tools that y...
Organizations planning enterprise data center consolidation and modernization projects are faced wit...
Serveless Architectures brings the ability to independently scale, deploy and heal based on workload...
Let’s face it, embracing new storage technologies, capabilities and upgrading to new hardware often ...
CI/CD is conceptually straightforward, yet often technically intricate to implement since it require...