🔍 Elastic Stack

These are notes from Learning Elastic Stack 7.0, by Pranav Shukla and Sharath Kumar MN.

Using the Kibana Console

Some simple APIs to warm up.

  • GET / <- Prints version information
  • GET <index-name>/_mappings <- schema/mappings of this index
  • GET <index-name>/_doc/<id_of_document> <- Content of this document

Fundamentals

An Index is loosely analogous to a table, and a document to a record. One Index can have only one Type.

Types are logical groupings same/similar documents in an Index. e.g. Employees could be one Type and Orders could be another, even if both were json documents and both had several common fields.

Documents: basic unit of information. Contains multiple fields like date, logMessage, processName, etc. Internal fields that Elastic itself maintains: _id (unique identifier), _type (Document's type, e.g. _doc), _index (Index name where it is stored)

Nodes form together to form a cluster.

Shards and Replicas

One can shard an index so that it is split into multiple segments, which will then reside on 1 or more nodes. By default 5 shards are made for every index. But if a node were to go down, those shards would be lost. So you can also create one replica for each shard, which will again be distributed in a slightly different order on the same nodes. Execution of queries is transparently distribute to either the primary or the replica shards.

Core DataTypes

  • String datatypes:
    • text - general lengthy text, elastic can do full-text search on this
    • keyword - let's you run some analytics on string fields, i.e. something you want to sort, filter, aggregate
  • Numeric datatypes:
    • byte/short/integer/long
    • float/double
    • half_float
    • scaled_float
  • date datatype
  • boolean datatype
  • binary datatype - arbitrary binary content, base64-encoded
  • Range datatypes: integer_range, float_range, long_range, double_range, date_range

Complex DataTypes

  • array - no mixing, list of same types
  • object - allows inner objects within json documents
  • nested - arrays of inner objects, where each inner object needs to be independently queriable

Other DataTypes

  • geo-point datatype - stores geo-points as lat and long
  • geo-shape datatype - store geometric shapes like polygons, maps, etc. Allows queries that search within a shape
  • ip datatype - ipv4/ipv6

Indexes

Check GET <index-name>/_mappings in the dev console to see the fields and their types in this index.

You will see stuff like this:

        "file" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },

What this means is, file is a field of type text, but it's also mapped as a keyword so you can also do analytics on it.

An Inverted Index is built from all fields.

CRUD APIs

An indexing operation is basically the addition of a document to the index. Elastic parses all the fields and builds the inverted index.

Use the PUT API to do this, with or without an id.

Get it with this:

GET <index-name>/_doc/<id of document>

You can call an UPDATE with just a specific field (say, price) to update that field in the document. Elastic will version and maintain both copies. The _version field will be incremented.

You can set the field doc_as_upsert to true and call a POST to <index>/_update/<id> to update if it exists or insert otherwise.

You can even do some scripting when you call the POST, using Elastic's 'painless' scripting language. e.g. to increment current value by 2.

DELETE: Call it on <index>/_doc/<id> as expected.

Updating a mapping

In this example, the 'code' field is converted to a keyword type:

PUT /catalog/_mapping
{
  "properties": {
    "code": {
      "type": "keyword"
    }
  }
}

REST API overview

Main categories:

  • Document APIs
  • Search APIs
  • Aggregation APIs
  • Indexes APIs
  • Cluster APIs
  • cat APIs

For pretty printing while using curl, suffix ?pretty=true. In the Console UI, it's turned on by default.

Searching

Use the _search API: 

GET /_search

This prints ALL docs in ALL indexes, first 10 results only though.

To search within an index:

GET /<index-name>/_search
GET /<index-name>/_doc/_search

In earlier versions of elastic, an index could have more than one type. In the above example, _doc is the type.

In Elastic 7.0, only one type is supported. So the second GET is deprecated.

To search across more than one index:

GET /catalog,my_index/_search

Analytics and Visualizing Data

Elastic has Analyzers that break down values of a field into terms, to make it searchable. This happens both during indexing and during searching. Final goal is for the searchable index to be created.

Analyzers comprise of Character Filters, a Tokenizer, and Token Filters.

Character filters map strings to something else, e.g. :) maps to smile. They are run at the beginning of the processing chain in an analyzer.

Token Filters are used for use cases like, removing stop words (a/an/the), replacing everything to lowercase, etc.

Apart from the (default) Standard Analyzer, there are many others.

To understand how the tokenization happens, here's an example:

GET /_analyze
{
  "text" : "test analysis",
  "analyzer": "standard"
}

Output:

{
  "tokens" : [
    {
      "token" : "test",
      "start_offset" : 0,
      "end_offset" : 4,
      "type" : "<ALPHANUM>",
      "position" : 0
    },
    {
      "token" : "analysis",
      "start_offset" : 5,
      "end_offset" : 13,
      "type" : "<ALPHANUM>",
      "position" : 1
    }
  ]
}

With different analyzers and filters, the final tokens would be different.

Term queries

You would use these in a search query to bypass the analysis stage and directly lookup the inverted index. Other more complex queries use these as a base.

Types of term queries:

  • range query - e.g. to show all Products where the Price attibute is >10 and <=20.
    • You can boost the weight of the results by suppplying a boost multipler.
    • You can query date ranges e.g. from now-7d to now.
  • exists query - Just tell if the field exists or not.
  • Term query - e.g do an exact match for a certain manufacturer in a Product index. Use the keyword type for this since keywords are not indexed.
    • You can get the keyword by querying <fieldname>.raw
  • Terms query - Same as above, but you can give multiple terms to search for.

And a few others, see the full list of term-levelqueries here.

match queries do the actual full-text searching. However if you search in a keyword (like datacenter.raw which is a keyword field), it skips all that and does an exact match.

You can set params for fuzziness in your search and it will return results accordingly e.g. victer will match with victor.

Bucket aggregations

Like a GROUP BY basically. Example:

GET <index_name>/_search
{
    "aggs" {
        "byCategory": {
            "terms": {
                "field": "category"
            }
        }
    },
    "size": 0
}
  • The size is set to 0 so we don't get raw results, but only the aggregated ones.

You can also bucketize by numerical ranges, e.g. show me everything between 1 and 100, 100 and 1000, etc.

Metric aggregations

Like doing a COUNT or AVG etc on numeric data. It's all json instead of SQL. Example:

GET <index_name>/_search
{
    "aggregations": {
        "download_max" : {
            "max": {
                "field": "downloadTotal"
            }
        }
    },
    "size": 0
}

A Stats aggregation is similar but it basically does the sum, average, mix, max and count in a single shot.

Buckets based on Geospatial data

  • Geodistance aggregation - based on a lat/long, query hits within a certain radius.
  • GeoHash grid aggregation - Divides the map into grids and searches within a wide imprecise grid or narrower, more precise grids.

Logstash

I already know this quite well. Input/Filter/Output sections etc.

Input Plugins

  • file is the most obvious, to read from a file.
  • beats tells logstash to pull from a beats daemon. Just takes a port setting and nothing else.
  • jdbc: imports from a database. Each row becomes an event, each column becomes a field. You can specify your sql statement and how often to query.
  • imap: read mails!

Output Plugins

  • elasticsearch, and kafka obviously.
  • csv
  • pagerduty to send to PD. e.g. your input plugin could match all 5xx errors and output could directly page someone.

Filter Plugins

grok is the one I've used most but there are others.

  • csv - Tell it to autodetect_columns otherwise set yours explicitly, and it will extract csv data.
  • mutate - You can convert fields here (Age to integer), rename them (FName to FirstName), strip them, uppercase them, etc. Looks quite powerful
  • grok - most poweful. match a line against an expression. Use %{PATTERN:FIELDNAME:type} to match a pattern with a field and set its type. Some in-built patterns are TIMERSTAMP_ISO8601, USERNAME, GREEDYDATA. A nice list is here.
  • date - You can set a pattern like dd/MMM/YYY:HH:mm:ss Z as your case may be. Overrides the @timestamp field by default.
  • geoip - converts an ip to a geoip json (timezone, lat/long, continent code, country name and code etc.)
  • useragent - converts a UA string based on Browserscope data, to OS, browser, version fields.

Codec Plugins

There are also 'codec' plugins to encode/decode events: these are hit just before the input stage, or just before it leaves the output stage. Examples:

  • json: treats data as json, otherwise falls back to plain text and adds a _jsonparsefailure tag.
  • rubydebug for ruby stuff
  • multiline for merging multiple lines into a single event, think a long backtrace. You can specify a regex e.g. any line that starts with a space "^\s ", and logstash will merge it with the previous event.

Elastic Pipelines

Newer elastic versions have an 'ingest node', if you use this you can potentially skip all the filtering in logstash. These nodes can do the preprocessing before the indexing happens.

You would define a pipeline, with a series of processors. Each processor transforms the document in some way.

Some processors: gsub, grok, convert, remove, rename, etc. Full list of processor directives.

e.g. I've seen dissect used to do basically what grok does.

You would use the _ingest API to play with pipelines.

Beats

Lightweight shippers. A library called libbeat is used. Go is used so a single fat binary is all you need. Looks like this does the input and output part of logstash, and pipelines do the filter part.

  • filebeat - takes files and sends them to elastic, kafka, logstash, etc.
    • You can use an out of the box module, consisting of path to look for logs, elastic Ingest pipeline to send to, elastic templates contianing field definitions, and sample kibana dashboards.
  • metricbeat - like collectd.
  • packetbeat - real time packet analyzer, understands Application layer like HTTP, MySQL, Redis etc.
  • heartbeat - check if service is up and reachable. Supports icmp, tcp, http probes.
  • winlogbeat - Reads event logs using windows APIs.
  • auditbeat - Skips auditd and directly communicates with underling audit framework apparently.
  • journalbeat - For journald.
  • functionbeat - For serverless.

3rd party stuff: spring, nginx, mysql, mongo, apache, docker, kafka, redis, kafka, amazon*. Full list here.

Kibana Notes

Initial Setup

You must first create an index-pattern that aggregates your indexes. Then you would see all its fields, and can make each of them searchable, aggregatable, etc.

Queries

Recollect the Term Queries section above. You can search for all those exact matches with field:value, e.g. datacenter:sjc.

OOh you can also do wildcard searches, like this: host:nginx* will match all host fields with the value nginx01, nginx02, etc.

MUST NOT is like this: ~response:200

Ranges are like this: response:[301 to 500]

KQL

KQL Guide

Example: response:200 or geoip.city_name:Diedorf

Visualizations

Kibana supports these 2 aggregations: - Bucket: like a GROUP BY. - Metric: you can plot Count, Average, Sum, Min, Max, Standard Deviation, etc.

X-Pack

You'd see stuff like this on the sidebar: Maps, Machine Learning, Infrastructure, Logs, APM, Uptime, Dev Tools, Stack Monitoring.