Learnitweb

How Many Primary Shards Should You Create in Elasticsearch?

One of the first questions that almost everyone asks, sooner or later, is this:

“How many primary shards should I create for my index?”

If you have asked this question before, you are absolutely not alone. In fact, this is one of the most common questions in the Elasticsearch community. Unfortunately, it is also a question that has an answer most people do not like:

It depends.

There is no single number that is correct for every application, every dataset, or every use case. The right answer depends on how much data you will store, how fast it will grow, how you query it, and how your cluster is sized. Still, even though there is no universal answer, we can build a very solid mental model and some practical guidelines that will help you make a good decision.

First, Let Us Understand What a Shard Can Actually Handle

A shard in Elasticsearch is not just a logical concept. Under the hood, it is a Lucene index, and Lucene has some very real technical limits.

From a purely theoretical point of view:

  • A single shard can hold up to about 2.1 billion documents. That is an extremely large number, and for most applications, they will never hit this limit.

However, this does not mean:

  • That a shard can efficiently store hundreds of terabytes of data.
  • Or that performance will remain good as the shard keeps growing indefinitely.

In practice, what matters much more than the number of documents is the size of the shard on disk and in memory.

Based on real-world experience and years of production usage:

A single shard usually performs best when its size is somewhere in the range of about 25 GB to 50 GB.

Some systems work fine with slightly smaller or slightly larger shards, but this range is a very good practical guideline. In fact, even 25 GB is not considered a very large shard in many production environments.

Thinking in Terms of Data Volume, Not Just Shard Count

Instead of asking:

“How many shards should I create?”

A much better question is:

“How much data do I expect this index to hold, both now and in the future?”

For example, imagine that:

  • Today your application stores 100 GB of data.
  • In the next few years, you expect it to grow to around 500 GB.

If we use the 25–50 GB per shard guideline, then:

  • 500 GB divided by 25 GB per shard gives you about 20 shards.
  • 500 GB divided by 50 GB per shard gives you about 10 shards.

So in this case, choosing something like 10 to 20 primary shards would be a very reasonable and well-justified decision. It gives you enough parallelism, keeps shard sizes in a healthy range, and leaves room for growth.

This way of thinking is far more reliable than choosing a random number like 3, 5, or 50 shards without any connection to your actual data volume.

The Very Common Mistake: “Let Me Create 100 or 1000 Shards Just in Case”

Many people think like this:

“I do not know how big my data will become in the future, so let me create 100 or even 1000 shards now. That way, I will never have to worry about scaling later.”

This sounds reasonable at first, but it is actually a very dangerous approach.

The reason is simple:

Nothing in Elasticsearch is free, and shards are not free either.

Every shard:

  • Consumes memory.
  • Consumes file handles.
  • Has metadata that the master node must track and manage.
  • Increases the complexity of cluster state updates, shard allocation, and recovery.

If you create hundreds or thousands of shards unnecessarily, then:

  • The master node will struggle to manage the cluster state.
  • Cluster operations such as index creation, node joins, shard rebalancing, and recovery will become slower and heavier.
  • Overall cluster stability and performance can suffer, even if your actual data volume is not that big.

So while over-sharding may look like “future-proofing”, in reality it often becomes self-inflicted technical debt.

The Honest Answer: It Depends, But With Good Guidelines

So let us repeat the honest answer again:

There is no single correct number. It always depends on your application, your data, and your growth expectations.

However, you now have some very useful practical guidelines:

  • A single shard can technically hold billions of documents, but in practice it usually performs best when its size is roughly 25 GB to 50 GB.
  • You should estimate your future data size, not just your current data size, and choose the number of shards so that each shard stays in that healthy size range.
  • Creating a huge number of shards “just in case” is almost always a bad idea, because shards have real costs and put pressure on the master and the cluster.