Konstantin Ryadov JVM engineer turned AI engineer. Things I have built.

ELK 5.X

With this post I am starting a series of notes on using Elasticsearch 5.X for aggregation and search. The site is https://www.elastic.co

Here is my own definition of the ELK stack (Elasticsearch, Logstash, Kibana + Beats):

  • Elasticsearch: a non-relational database (it stores so-called documents) optimised for search queries, both document lookups and aggregations. Platform: Java. Search engine: Apache Lucene. Scales horizontally out of the box. Has a REST API, among others.
  • Logstash: a data collector built around input => filter => output. The input can be file, jdbc, http, exec, kafka, log4j, s3 and so on (more than 30 types). The output can be file, elasticsearch, kafka, tcp and even jira (again, more than 30 types). Platform: JRuby.
  • Beats: a lightweight, simplified, native take on Logstash (an agent, more or less) that can ship data either to Logstash itself (for filtering) or straight to Elasticsearch.
  • Kibana: a visualiser for what is in Elasticsearch. Platform: Node.js.

What it looks like:

ELK stack diagram

What you can use ELK for:

  • Aggregating infrastructure logs, and searching and analysing them
  • BI: pulling and analysing large volumes of data from an arbitrary source
  • Infrastructure monitoring.

Originally published in Russian on LiveJournal.

All posts