☰

Aggregations in ElasticSearch:

Aggregations in Elasticsearch: Types, Syntax and Examples

You have a million documents in an index, and a manager asks a simple question: how many cars did we list for each make, and what is the average price of each? You do not need to read a million documents to answer. Elasticsearch can group them and do the maths inside the cluster, and return a few lines of numbers. That is what aggregations are for.

An aggregation summarises your data. It can count documents in groups, calculate an average or a total, show how prices spread over ranges, or work on the results of other aggregations. If you have used GROUP BY in SQL, the idea will feel familiar. Aggregations are also the engine behind the charts and tables in Kibana.

In this guide we explain the aggs request, and the three types of aggregations. We then build a small index of cars, and run the three examples from the original page, this time with the full request text, so you can copy them. After that we add the things that most beginners miss: size 0, the keyword field rule, the default of 10 buckets, query scope, range and histogram, and a pipeline example.

What is the aggs Query in Elasticsearch?

Every aggregation lives inside the aggs parameter of a search request. You give each aggregation a name of your choice, and the name appears again in the response. The simplest form looks like this, with a placeholder name and field:

json
{
  "aggs": {
    "my_agg_name": {
      "terms": { "field": "my_field" }
    }
  }
}

The word aggs is a short form. You can also write aggregations in full, and both work. Elastic also documents that you can set size to 0 when you want only the aggregation results, and no search hits. We use that in every example below.

Types of Aggregations in Elasticsearch

Elasticsearch groups aggregations into three families.

The three families of aggregations in Elasticsearch
Family What it does Examples
Bucket Groups documents into buckets, based on field values, ranges or other rules. Each bucket has a document count terms, range, histogram, date_histogram
Metric Calculates a value from the field values, such as a sum or an average avg, min, max, sum, value_count, stats, cardinality
Pipeline Takes its input from other aggregations, and not from documents or fields avg_bucket, sum_bucket, derivative

The old page said that a metric aggregation computes a single statistic. That is true for avg, min, max and sum, which return one value. But stats returns five values, which are count, min, max, avg and sum. So it is better to say that a metric aggregation calculates numbers from field values, and that some return one value while others return several.

Set Up Sample Data

The old page used a cars example, and showed the queries only as pictures. To make the examples copyable, here is a small index of 10 cars. Run these two requests in Kibana Dev Tools. The first creates the index with the right field types. The second adds the documents. The refresh=true setting makes them searchable at once.

kibana dev tools
PUT cars
{
  "mappings": {
    "properties": {
      "make":  { "type": "keyword" },
      "model": { "type": "keyword" },
      "color": { "type": "keyword" },
      "year":  { "type": "integer" },
      "price": { "type": "integer" }
    }
  }
}
kibana dev tools
POST cars/_bulk?refresh=true
{"index": {"_id": "1"}}
{"make": "Toyota", "model": "Corolla", "year": 2022, "price": 21000, "color": "white"}
{"index": {"_id": "2"}}
{"make": "Toyota", "model": "Camry", "year": 2023, "price": 28000, "color": "black"}
{"index": {"_id": "3"}}
{"make": "Toyota", "model": "Fortuner", "year": 2021, "price": 41000, "color": "white"}
{"index": {"_id": "4"}}
{"make": "Toyota", "model": "Yaris", "year": 2020, "price": 14000, "color": "silver"}
{"index": {"_id": "5"}}
{"make": "Honda", "model": "City", "year": 2022, "price": 19000, "color": "silver"}
{"index": {"_id": "6"}}
{"make": "Honda", "model": "Civic", "year": 2023, "price": 26000, "color": "black"}
{"index": {"_id": "7"}}
{"make": "Honda", "model": "Amaze", "year": 2021, "price": 12000, "color": "white"}
{"index": {"_id": "8"}}
{"make": "Ford", "model": "Figo", "year": 2020, "price": 11000, "color": "white"}
{"index": {"_id": "9"}}
{"make": "Ford", "model": "Endeavour", "year": 2022, "price": 44000, "color": "grey"}
{"index": {"_id": "10"}}
{"make": "BMW", "model": "X1", "year": 2023, "price": 52000, "color": "black"}

Notice that make, model and color are keyword fields, and year and price are integers. This matters. The terms aggregation works on keyword fields, and we explain why below.

The sample index of 10 cars
Make Model Year Price Color
Toyota Corolla 2022 21000 white
Toyota Camry 2023 28000 black
Toyota Fortuner 2021 41000 white
Toyota Yaris 2020 14000 silver
Honda City 2022 19000 silver
Honda Civic 2023 26000 black
Honda Amaze 2021 12000 white
Ford Figo 2020 11000 white
Ford Endeavour 2022 44000 grey
BMW X1 2023 52000 black

The responses below are shown for this data, and trimmed to the aggregations part. A real response also has fields such as took and hits.

Example 1: Group Documents With a Bucket Aggregation

The first example from the old page groups the cars by make, in an aggregation called popular_cars. The terms aggregation makes one bucket for each different value in the field, and counts the documents in each.

Screenshot of the old page example, a terms aggregation named popular_cars that groups cars by make
kibana dev tools
GET cars/_search
{
  "size": 0,
  "aggs": {
    "popular_cars": {
      "terms": { "field": "make" }
    }
  }
}
output
{
  "aggregations": {
    "popular_cars": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "Toyota",
          "doc_count": 4
        },
        {
          "key": "Honda",
          "doc_count": 3
        },
        {
          "key": "Ford",
          "doc_count": 2
        },
        {
          "key": "BMW",
          "doc_count": 1
        }
      ]
    }
  }
}

Read the response like this. There are 4 Toyota cars, 3 Honda, 2 Ford and 1 BMW. The buckets are sorted with the biggest count first, which is the default. The field sum_other_doc_count is 0, which means no documents were left out. The field doc_count_error_upper_bound is also 0, which means the counts are exact here. We explain both below.

Why size 0?

A search request returns the top 10 matching documents by default, as well as the aggregations. If you only want the numbers, setting size to 0 skips the documents and keeps the response small. Elastic also notes that using size 0 together with the same preference string for each search can let you use the shard request cache for repeat queries.

To see how it looks when you do want both, here is a request that returns the two most expensive cars, and the make counts, in one response:

kibana dev tools
GET cars/_search
{
  "size": 2,
  "sort": [ { "price": "desc" } ],
  "_source": [ "make", "model", "price" ],
  "aggs": {
    "popular_cars": { "terms": { "field": "make" } }
  }
}

Example 2: Run Multiple Aggregations Together

You can put several aggregations in one request, and Elasticsearch runs them all on the same set of documents. The old example ran popular_cars with avg_price, max_price and min_price.

Screenshot of the old page example that runs popular_cars with avg_price, max_price and min_price in one request
kibana dev tools
GET cars/_search
{
  "size": 0,
  "aggs": {
    "popular_cars": { "terms": { "field": "make" } },
    "avg_price":    { "avg": { "field": "price" } },
    "max_price":    { "max": { "field": "price" } },
    "min_price":    { "min": { "field": "price" } }
  }
}
output
{
  "aggregations": {
    "popular_cars": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "Toyota",
          "doc_count": 4
        },
        {
          "key": "Honda",
          "doc_count": 3
        },
        {
          "key": "Ford",
          "doc_count": 2
        },
        {
          "key": "BMW",
          "doc_count": 1
        }
      ]
    },
    "avg_price": {
      "value": 26800.0
    },
    "max_price": {
      "value": 52000.0
    },
    "min_price": {
      "value": 11000.0
    }
  }
}

The three metric aggregations work on all 10 cars. The average is 26800, the highest price is 52000 and the lowest is 11000. Note that metric values come back as decimals, such as 26800.0, even for an integer field. Also note that these three numbers are not per make. They cover the whole set, because the metric aggregations sit next to popular_cars and not inside it. To get them per make, you need nesting, which is Example 3.

Example 3: Nest an Aggregation Inside Another

A bucket aggregation can hold other aggregations inside it, and they run once for every bucket. These are called sub aggregations. Here the stats aggregation runs inside each make bucket. The Elastic documentation says that there is no limit on how deep you can nest, but each level adds work, so keep it as shallow as you can.

kibana dev tools
GET cars/_search
{
  "size": 0,
  "aggs": {
    "popular_cars": {
      "terms": { "field": "make" },
      "aggs": {
        "stats_price": { "stats": { "field": "price" } }
      }
    }
  }
}
output
{
  "aggregations": {
    "popular_cars": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "Toyota",
          "doc_count": 4,
          "stats_price": {
            "count": 4,
            "min": 14000.0,
            "max": 41000.0,
            "avg": 26000.0,
            "sum": 104000.0
          }
        },
        {
          "key": "Honda",
          "doc_count": 3,
          "stats_price": {
            "count": 3,
            "min": 12000.0,
            "max": 26000.0,
            "avg": 19000.0,
            "sum": 57000.0
          }
        },
        {
          "key": "Ford",
          "doc_count": 2,
          "stats_price": {
            "count": 2,
            "min": 11000.0,
            "max": 44000.0,
            "avg": 27500.0,
            "sum": 55000.0
          }
        },
        {
          "key": "BMW",
          "doc_count": 1,
          "stats_price": {
            "count": 1,
            "min": 52000.0,
            "max": 52000.0,
            "avg": 52000.0,
            "sum": 52000.0
          }
        }
      ]
    }
  }
}

Now every make has its own count, minimum, maximum, average and sum. For example, the 4 Toyotas have an average price of 26000, and the single BMW has an average of 52000. The old page said that stats gives the average, minimum, maximum and sum. As you can see, it also gives the count.

Sort Buckets by a Sub Aggregation

By default the buckets are sorted by document count. You can sort them by the result of a sub aggregation instead. Here the makes are ordered by average price, with the most expensive first.

kibana dev tools
GET cars/_search
{
  "size": 0,
  "aggs": {
    "popular_cars": {
      "terms": { "field": "make", "order": { "avg_price": "desc" } },
      "aggs": {
        "avg_price": { "avg": { "field": "price" } }
      }
    }
  }
}
output
{
  "aggregations": {
    "popular_cars": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "BMW",
          "doc_count": 1,
          "avg_price": {
            "value": 52000.0
          }
        },
        {
          "key": "Ford",
          "doc_count": 2,
          "avg_price": {
            "value": 27500.0
          }
        },
        {
          "key": "Toyota",
          "doc_count": 4,
          "avg_price": {
            "value": 26000.0
          }
        },
        {
          "key": "Honda",
          "doc_count": 3,
          "avg_price": {
            "value": 19000.0
          }
        }
      ]
    }
  }
}

The Elastic documentation advises care with ordering by a sub aggregation, and warns that ordering by ascending count gives inaccurate results. For rare values, it points to the rare_terms aggregation.

How Many Buckets Do You Get? The Size Setting

A terms aggregation returns the top 10 terms by default. Use size to get more or fewer, up to the limit set by the search.max_buckets setting of the cluster. Here is a request for only the top 2 makes:

kibana dev tools
GET cars/_search
{
  "size": 0,
  "aggs": {
    "popular_cars": { "terms": { "field": "make", "size": 2 } }
  }
}
output
{
  "aggregations": {
    "popular_cars": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 3,
      "buckets": [
        {
          "key": "Toyota",
          "doc_count": 4
        },
        {
          "key": "Honda",
          "doc_count": 3
        }
      ]
    }
  }
}

Now sum_other_doc_count is 3. This is the number of documents in the terms that did not make it into the top 2. Ford has 2 and BMW has 1, so 3 documents were left out. If this number is above 0 in your own data, some buckets were dropped, and you should raise size.

Limit What You Aggregate With a Query

Aggregations run on the documents that match the query. If you add a query, the aggregation sees only those documents. Here we count the colours of cars that cost at least 20000:

kibana dev tools
GET cars/_search
{
  "size": 0,
  "query": { "range": { "price": { "gte": 20000 } } },
  "aggs": {
    "colors": { "terms": { "field": "color" } }
  }
}
output
{
  "aggregations": {
    "colors": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "black",
          "doc_count": 3
        },
        {
          "key": "white",
          "doc_count": 2
        },
        {
          "key": "grey",
          "doc_count": 1
        }
      ]
    }
  }
}

Four cars cost less than 20000, so they are left out of the counts. Without the query, white would have 4.

More Metric Aggregations

Besides avg, min, max and stats, three more are used all the time.

  • sum adds up the values of a field.
  • value_count counts how many values the field has.
  • cardinality counts how many different values the field has.
kibana dev tools
GET cars/_search
{
  "size": 0,
  "aggs": {
    "total_value":  { "sum": { "field": "price" } },
    "priced_cars":  { "value_count": { "field": "price" } },
    "different_makes": { "cardinality": { "field": "make" } }
  }
}
output
{
  "aggregations": {
    "total_value": {
      "value": 268000.0
    },
    "priced_cars": {
      "value": 10
    },
    "different_makes": {
      "value": 4
    }
  }
}

The cardinality aggregation gives an approximate count for large data. For a tiny set like ours it is exact. The result of 4 matches the four makes we saw earlier.

A whole set of numbers in one go is also available with stats, as in Example 3, but on the whole index:

kibana dev tools
GET cars/_search
{
  "size": 0,
  "aggs": {
    "stats_price": { "stats": { "field": "price" } }
  }
}
output
{
  "aggregations": {
    "stats_price": {
      "count": 10,
      "min": 11000.0,
      "max": 52000.0,
      "avg": 26800.0,
      "sum": 268000.0
    }
  }
}

Group Numbers With Range and Histogram

The terms aggregation works well for labels. For numbers, you often want bands. The range aggregation makes buckets that you define. Here three price bands are named budget, mid and premium. The from value is included, and the to value is not.

kibana dev tools
GET cars/_search
{
  "size": 0,
  "aggs": {
    "price_bands": {
      "range": {
        "field": "price",
        "ranges": [
          { "key": "budget",  "to": 20000 },
          { "key": "mid",     "from": 20000, "to": 40000 },
          { "key": "premium", "from": 40000 }
        ]
      }
    }
  }
}
output
{
  "aggregations": {
    "price_bands": {
      "buckets": [
        {
          "key": "budget",
          "to": 20000.0,
          "doc_count": 4
        },
        {
          "key": "mid",
          "from": 20000.0,
          "to": 40000.0,
          "doc_count": 3
        },
        {
          "key": "premium",
          "from": 40000.0,
          "doc_count": 3
        }
      ]
    }
  }
}

The histogram aggregation makes equal sized steps. Here the step is 10000. Empty steps in the middle are shown with a count of 0.

kibana dev tools
GET cars/_search
{
  "size": 0,
  "aggs": {
    "price_steps": { "histogram": { "field": "price", "interval": 10000 } }
  }
}
output
{
  "aggregations": {
    "price_steps": {
      "buckets": [
        {
          "key": 10000.0,
          "doc_count": 4
        },
        {
          "key": 20000.0,
          "doc_count": 3
        },
        {
          "key": 30000.0,
          "doc_count": 0
        },
        {
          "key": 40000.0,
          "doc_count": 2
        },
        {
          "key": 50000.0,
          "doc_count": 1
        }
      ]
    }
  }
}

There is also the date_histogram aggregation, which groups documents by a time step such as a day or a month. It is the one behind most time charts in Kibana. Our sample has no date field, so I have not run it here. Please add a date field to your own data to try it.

Pipeline Aggregations: An Example

A pipeline aggregation takes its input from other aggregations. Here avg_bucket finds the average of the average prices of all the makes. The path to the input uses the greater than sign to go from the bucket aggregation to the metric inside it.

kibana dev tools
GET cars/_search
{
  "size": 0,
  "aggs": {
    "popular_cars": {
      "terms": { "field": "make" },
      "aggs": {
        "avg_price": { "avg": { "field": "price" } }
      }
    },
    "avg_of_make_averages": {
      "avg_bucket": { "buckets_path": "popular_cars>avg_price" }
    }
  }
}
output
{
  "aggregations": {
    "popular_cars": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "Toyota",
          "doc_count": 4,
          "avg_price": {
            "value": 26000.0
          }
        },
        {
          "key": "Honda",
          "doc_count": 3,
          "avg_price": {
            "value": 19000.0
          }
        },
        {
          "key": "Ford",
          "doc_count": 2,
          "avg_price": {
            "value": 27500.0
          }
        },
        {
          "key": "BMW",
          "doc_count": 1,
          "avg_price": {
            "value": 52000.0
          }
        }
      ]
    },
    "avg_of_make_averages": {
      "value": 31125.0
    }
  }
}

The average of the four make averages is 31125. This is different from the overall average of 26800 in Example 2. The reason is that the overall average counts each car once, so Toyota, with 4 cars, has more weight. The pipeline average counts each make once. Neither is wrong, but they answer different questions, and you should pick the one that fits.

A Common Error: Aggregating on a Text Field

The most common beginner problem is running a terms aggregation on a text field. A text field is split into words for search, and is not meant for grouping. The Elastic documentation says that you cannot run a terms aggregation on a text field by default. There are two ways round it.

  • Best: use a keyword field, or a keyword sub field of the text field.
  • Possible, but costly: turn on fielddata for the text field. This can use a lot of memory.

If you let Elasticsearch map a new text field on its own, it usually makes a text field with a keyword sub field named keyword. You can also do this yourself:

kibana dev tools
PUT cars_fixed
{
  "mappings": {
    "properties": {
      "make": {
        "type": "text",
        "fields": { "keyword": { "type": "keyword" } }
      }
    }
  }
}

Then you aggregate on the sub field, make.keyword:

kibana dev tools
GET cars_fixed/_search
{
  "size": 0,
  "aggs": {
    "popular_cars": { "terms": { "field": "make.keyword" } }
  }
}

I could not run this error here, so I have not quoted the exact error text. Check it on your own cluster. It is an illegal argument error, and it tells you to use a keyword field.

Accuracy and Limits to Know

  • Terms counts can be approximate. On large data, each shard returns only its top terms, so a term can be missed or its count can be too low. The field doc_count_error_upper_bound tells you the largest possible error. The Elastic documentation says to set show_term_doc_count_error to true if you want this value per bucket. The shard_size setting, which defaults to size times 1.5 plus 10, controls how many terms each shard returns.
  • Large whole numbers. Elasticsearch uses double values inside aggregations. The documentation says that aggregations on long numbers above 2 to the power of 53 are approximate.
  • Bucket limit. The search.max_buckets setting limits how many buckets one request may create.
  • Runtime fields. If a field does not fit the aggregation you need, a runtime field can help, but it adds some overhead.
  • meta and typed_keys. The meta option adds your own note to an aggregation. The typed_keys option adds the aggregation type to the name in the response, such as terms#popular_cars.

Aggregations Compared With SQL

If you know SQL, this table links the ideas. This is an idea map, and I am not claiming that one is faster than the other. That depends on your data, your hardware and your setup.

SQL ideas and the matching Elasticsearch aggregations
SQL Elasticsearch Where to find it
GROUP BY make terms on make Example 1
COUNT, AVG, MIN, MAX value_count, avg, min, max Example 2
GROUP BY with AVG per group terms with an avg sub aggregation Example 3
WHERE price >= 20000 a range query next to the aggs Query scope section
CASE WHEN price bands range aggregation Range section
COUNT DISTINCT make cardinality Metrics section

The SQL for Example 3 would look like this:

sql
SELECT make, COUNT(*) AS cars, AVG(price) AS avg_price
FROM cars
GROUP BY make
ORDER BY cars DESC;

One difference matters. A SQL query runs on tables that you design for joins and exact answers. Elasticsearch is built for search first, and for fast summaries on large amounts of data. It does not do joins in the same way, so check that it fits your task before you move work from a database.

Why Do Aggregations Matter for Big Data Analytics?

Screenshot of the output of the old page aggregation example

The old page said that one query can replace several SQL queries and a lot of application code, and that Elasticsearch is faster than an RDBMS. It gave no example or test. Here is what you can show. In a single request, Example 3 gave a count and five statistics for every make. Doing the same without aggregations would mean fetching documents and calculating in your own code.

That is the real value. The summary is built where the data lives, so only a few lines travel over the network. This is why aggregations are used for dashboards, log analysis, product filters on shopping sites, and reports on search data.

How fast it runs depends on the size of your data, the number of shards, the field types and how many buckets you ask for. If you plan to use it for analytics at work, test with your own data.

Common Aggregation Mistakes and How to Fix Them

Common aggregation mistakes, their usual cause and the fix
Problem Usual cause Fix
An error on a terms aggregation The field is a text field Use a keyword field or a keyword sub field
The response is full of hits size was not set to 0 Add size 0
Only 10 buckets come back The default size of a terms aggregation is 10 Set size, and check sum_other_doc_count
The metric numbers are for all documents, not per group The metric is next to the bucket, not inside it Put the metric in the aggs of the bucket aggregation
Counts look slightly wrong on a large index Terms counts are approximate across shards Raise shard_size, and check doc_count_error_upper_bound
The average has a long decimal Metric values are doubles Round the number in your app
The aggregation name is not found in the response Names are your own, and are case sensitive Use the exact name from your request

Practice Exercises

  1. Create the cars index and load the 10 documents. Run Example 1 and check that you get 4, 3, 2 and 1.
  2. Group the cars by color, and add the average price per color.
  3. Use a query to keep only cars from 2022 or later, and count them by make.
  4. Make a range aggregation with your own price bands, and give each band a clear name.
  5. Add a sub aggregation inside a range bucket, such as the average year for each band.

Aggregations turn Elasticsearch from a search box into an analysis tool. Start with one terms aggregation on a keyword field, add one metric inside it, and add a query to narrow the data. Once that is comfortable, move on to range, date_histogram and pipeline aggregations. A good next step is to build a Kibana chart from the same index, because Kibana runs these same aggregations behind every chart.

Popular Tags: