The ELK Stack is a collection of three open-source products — Elasticsearch, Logstash, and Kibana. They are all developed, managed, and maintained by the company Elastic.
Kibana is an open-source visualisation and reporting user interface developed in 2013 by Rashid Khan of the Elastic Company.
Kibana allows users to produce visualisations from a variety of data inputs & can create pie charts, heat maps, line graphs, scatter plots and so much more to reflect data in an easily digestible format.
A few of the main benefits of using Kibana are as follows:
The ELK Stack can be used for a wide variety of use cases including but not limited to;
When Kibana has been installed using a .tar.gz package on Linux, it can be started from the command line using the following command: ./bin/kibana For additional installation types & operating systems consult the following guide.
In Kibana you can easily search the current index pattern you are viewing by entering your search criteria in the Query bar visible near the top left-hand side of your screen. Conveniently Kibana also shows its users a filtering dialogue which provides somewhat of a cheat sheet for easier filtering to be conducted.
Once you have Kibana loaded you will want to open the main menu and select Dashboard, from here you can select Create Dashboard. Once this step has been completed you will need to add panels of data to your dashboard for further visualisations and chart type selection to be applied.
Kibana dashboards are stored in Elasticsearch under the default index kibana-int which can be edited within the config.js file if you wish to store your dashboards in a different location.
If you find that your Kibana instance is loading slowly it is often mentioned in the support forums that the reason this happens is due to the Apps bundle or apps themselves loading in the background.
A line chart (also known as a line graph) is a type of data visualisation that displays data as a series of points that reflects changes over a designated time period.
Components of ELK stack:
Kibana Docker image: Kibana Docker images are of two different flavors called X-pack flavor and Oss Flavour. X-pack is the docker image that is pre-installed one, hence called as default. Concerning the Oss flavor, it doesn't have any link with the X-pack, but it is only an open-source one.
Kibana Port and Kibana.yml file: The configuration of the default setting is done on the localhost 5601 to run Kibana. The port number can be changed, or the connection can be established in an Elasticsearch that is installed on another machine, the kibana.yml file has to be updated. The Kibana server will then read the properties of the kibana.yml file.
Kibana provides the flexibility to host the Elasticsearch and navigate to perform the searching and modifications efficiently. The major components of the Kibana are listed below:
Kibana Visual interface: Kibana visual interface is a platform designed to make the changes or modifications to the customs based on the requirements. This includes bars, pie charts, and tables related to data.
Logstash is the product in ELK that is called as the data-pipeline tool. It is specifically designed to perform operations like collecting, storing, and parsing the logs for future use. It is an open-source data collection engine capable of unifying the data from multiple resources and normalizing it. Logstash also helps in diverse downstream analytics for business enhancements.
Filebeat is used to represent the shipping of the log files or log data. Filebeat is designed to play the logging agent's role, which is installed on the machine generating the log files, tailing them, and forwarding the data to either Logstash for advanced processing or directly into Elasticsearch for indexing.
The filebeat is dependent on the Logstash forwarder source code and replaces the Logstash forwarder. This method is specifically used to make use of the tailing log file and forward the same accordingly to the Logstash.
The Java version required to install the Logstash is Java8, while it does not support Java9. Some of the inputs used in Logstash are:
ELK Stack is designed to allow users to take to data from any source, in any format, and to search, analyze, and visualize that data in real-time.
Here, are reasons for using ELK stack:
Here, are important operation performed on documents:
X-Pack commands which will aid you in configuring security:
The document is fundamentally the same as a row in relational databases. Each report in the index has a diverse structure however has a similar data type for particular fields.
MySQL => Databases => Tables => Columns/Rows
ElasticSearch => Indices => Types => Documents with Properties
While ordering information in ElasticSearch, information is moved inside by the Analyzer characterized for the index, and afterwards listed. An analyzer is a collection of filters and tokenizers. Analyzers accessible in ElasticSearch 1.10 are given below:
The use cases of ELK log analytics are given below:
Elasticsearch provides a query DSL(Domain Specific Language) on the basis of JSON for defining the queries. Query DSL contains two kinds of clauses:
1) Leaf Query Clauses
Leaf Query Clauses search for a specific value in a specific field, like the term, range, or match queries.
2) Compound Query Clauses
Compound Query Clauses enclose other compound or leaf queries, and we use them for logically combining queries.
A Tokenizer breakdown fields values of a document into a stream, and inverted indexes are created and updated using these values, and these stream of values are stored in the document.
Each shard in ElasticSearch has 2 copies of the shard. These copies are called replicas. They serve the purpose of high availability and fault tolerance.
The process of storing data in an index is called indexing in ElasticSearch. Data in ElasticSearch can be divided into write-once and read-many segments. Whenever an update is attempted, a new version of the document is written to the index.
Each instance of ElasticSearch is called a node. Multiple nodes can work in harmony to form an ElasticSearch Cluster.
Due to resource limitations like RAM, CPU, etc, for scale-out, applications need to employ multiple instances of ElasticSearch on separate machines. Data in an index can be divided into multiple partitions, each handled by a separate node (instance) of ElasticSearch. Each such partition is called a shard. By default, an ElasticSearch index has 5 shards.
Syntax:
POST /_<index_name>/_type/_idGET API fetches the specified JSON document from an index.
Syntax:
GET <index_name>/_doc/<_id>The latest JDK or Java version 1.8.0 is a prerequisite to install Elasticsearch.
Command using with cat API are:
Some important configuration management tool supported by Elasticsearch is as follows:
We have different ways of searching in Elasticsearch:
Some important configuration management tool supported by Elasticsearch is as follows:
Grok is a filter plugin for Logstash that is used to parse unstructured data. It is often used for transforming Apache, Syslog and other webserver logs into a structured and queryable format for easier data analysis to be performed.
Fuzzy search allows Elasticsearch to return positive matches for non-exact matches of the term you are searching for. This is especially beneficial for eCommerce retailers where site visitors may often have typos in their spelling when trying to locate a product they wish to purchase. Fuzzy match results mean that these visitors are not served with a blank page which would often lead to a user being less likely to convert.
If you find that your Kibana instance is loading slowly it is often mentioned in the support forums that the reason this happens is due to the Apps bundle or apps themselves loading in the background.
A line chart (also known as a line graph) is a type of data visualisation that displays data as a series of points that reflects changes over a designated time period.
The curl command in Elasticsearch allows you to take many actions including but not limited to: deleting indexes, list all indexes, list all documents within an index, query using URL parameters, add data and list index mappings.
By using the DELETE /index name. Command.
To find individual download and install commands specific to the Logstash plugin you are wishing to install, RubyGems provides a wealth of resources to help you. Once you've selected the plugin you wish to use you can add it to your Logstash installation using the following command: bin/logstash-plugin install logstash-input-github
Logstash settings can be found located at the {extract.path}/config directory. Logstash settings can be configured from the following files found here: logstash.yml, pipelines.yml, jvm.options, log4j2.properties and within Linux there is an additional configuration file called startup.options.
The Logstash GeoIP filter is used to add supplementary information on the geographic location of IP addresses based on data available within the MaxMind GeoLite2 database.
You can use the node stats API to retrieve runtime statistics from Logstash.
A Logstash pipeline consists of these elements as listed in the following order: Input (the source of data), filter (processing, parsing & enrichment of data) & output (write the data for use within Elasticsearch & Kibana).