ELK - Elasticsearch Logstash Kibana

[DilipMami]

Logstash is a very light weight component to ship the logs from one server to centralized servers. In the centralized server we might have run the logstash to apply the pattern and get the required info extracted and then send it to elastic search server.

We can configure logstash in any machine, now lets see how can we install and configure in windows

• Logstash: The server component of Logstash that processes incoming logs


• Elasticsearch: Stores all of the logs


• Kibana: Web interface for searching and visualizing logs

Elastic has created an end-to-end stack called ELK and elastic stack makes searching and analyzing of data easier than ever before

Screenshot below explains capabilities of each component :



 



 

Elasticsearch and Logstash require Java 7 so it has to be installed

Configuration of Logstash is shown below:



 



 

Example in Windows OS ,

Grok is the bread and butter of Logstash filters and is used ubiquitously to derive structure out of unstructured data , it is used as a filter to Parse arbitrary text and structure it

For additional knowledge on creating patterns one can go through this :

Grok Constructor

logstash-patterns-core/patterns at master · logstash-plugins/logstash-patterns-core · GitHub

 



 



 

If one expects to see the output on the console just un-comment stdout , one can validate against the output to see if it is as expected



 

## Install Java JRE

Using /s will do a silent installation without asking you any questions. It should be save. Haven't had any additional browser toolbars installed afterwards

jre-windows-x64.exe /s INSTALLDIR=c:\java\jre

 

## Install NSSM Just extract the ZIP file to c:\nssm

## Logstash   ### Prepare the directory structure

REM Base install dir

md c:\logstash

REM Extract Logstash to this directory

md c:\logstash\install

REM NSSM will save Logstash's stdout/stderr here

md c:\logstash\nssm

REM Let's keep Logstash's config outside the install dir for easier updates

md c:\logstash\conf.d

 

Component Versions used

 

Elasticsearch 2.1.0 and 2.3.3

Logstash 1.5.4 and 2.3.2

Kibana-4.3.1-windows( has Sense which was useful in querying) and 4.5.1-windows(doesn't have Sense)

 

### Install Logstash as a Windows Service  

cd c:\nssm\win64

nssm install logstash C:\logstash\install\bin\logstash.bat

nssm set logstash AppParameters agent --config c:\logstash\conf.d

nssm set logstash AppDirectory C:\logstash\install

nssm set logstash AppEnvironmentExtra "JAVA_HOME=C:\java\jre"

nssm set logstash AppStdout c:\logstash\nssm\stdout.log

nssm set logstash AppStderr c:\logstash\nssm\stderr.log

REM Replace stdout and stderr files

nssm set logstash AppStdoutCreationDisposition 2

nssm set logstash AppStderrCreationDisposition 2

REM Disable WM_CLOSE, WM_QUIT in the Shutdown options.

Without it, NSSM can't stop Logstash properly

nssm set logstash AppStopMethodSkip 6

REM Let's start Logstash.

 

On completion of the above steps the configuration is in place to start with below steps:

 

net start logstash

### Remove Logstash's Windows service

net stop logstash

cd c:\nssm\win64

nssm remove logstash

## Troubleshooting   ### Have a look at Logstash's stderr/stdout data first

type c:\logstash\nssm\stderr.log

type  c:\logstash\nssm\stdout.log

### Is Java (64bit) installed correctly? c:\java\jre\bin\java -version java version "1.7.0_60" Java(TM) SE Runtime Environment (build 1.7.0_60-b19) Java HotSpot(TM) 64-Bit Server VM (build 24.60-b09, mixed mode)

### Test reading Security event logs on Windows   input { eventlog { type  => 'Win32-EventLog' logfile  => 'Security' } } output { stdout {} }

 



==============================================================================

==============================================================================

Now lets see how can we do it in Monsoon (SAP Cloud Server)

In addition to ELK (versions can vary), logstash forwarder needs to be installed


 

• Logstash Forwarder: Installed on servers that will send their logs to Logstash, Logstash Forwarder serves as a log forwarding agent that utilizes the lumberjack networking protocol to communicate with Logstash

The Logstash Forwarder will be installed on all of the servers that we want to gather logs for, which we will refer to collectively as our Servers








My Area of work was on Chef cookbooks to develop a Performance service in HCP for HTTP and LJS logs using ELK and Ruby

Example in Monsoon



 

Screenshot of a working filter





 



 

 

Here is a quick demo of the steps involved on Windows (also available on Youtube)

 



ELK - Elasticsearch Logstash Kibana - Introduction on Windows - YouTube

 

Remember that you can send pretty much any type of log to Logstash, but the data becomes even more useful if it is parsed and structured with GROK

 

Thanks,

Dilip


SAP BLR