Learnitweb

Can We Change the Number of Shards Later? And How Many Replicas Should We Use?

Two of the most common questions people ask after they start using Elasticsearch seriously are these:

“I created an index with five shards. Now I realize I need eight. How can I change it?”
“How many replica shards should I configure?”

Both questions look simple on the surface, but both are deeply connected to how Elasticsearch actually distributes documents across shards and how it guarantees availability. In this chapter, we will build a clear mental model for both problems and understand not only what is possible, but also why certain things are not allowed.

Why You Cannot Change the Number of Primary Shards for an Existing Index

Let us start with the first question. Suppose you created an index with four primary shards and then started indexing data into it. Later, you realize that you actually want five or eight shards instead. The natural question is: why not just add more shards?

The answer is: Elasticsearch does not allow this, and there is a very good technical reason for it.

To understand this, we must first understand how Elasticsearch decides which shard a document goes to.

When you index a document, Elasticsearch takes the document ID, runs it through a hash function, and then uses this formula internally:

shard_number = hash(document_id) % number_of_primary_shards

Now imagine this situation.

  • You have an index with 4 primary shards, numbered 0, 1, 2, and 3.
  • You index a document with ID ABCD.
  • Let us assume the hash function returns 44.
  • Elasticsearch calculates: 44 % 4 = 0.
  • So the document ABCD is stored in shard 0.

This same calculation is used every time you want to read that document as well. When you do a GET request for ABCD, Elasticsearch again computes the hash and again decides that it must look into shard 0.

Now imagine that you somehow add one more shard and make the index have 5 primary shards.

What happens now?

  • The hash of ABCD is still 44.
  • But now Elasticsearch computes: 44 % 5 = 4.
  • So Elasticsearch will now try to look for the document in shard 4.

But the document is not there. It is still sitting in shard 0, where it was originally stored.

This immediately shows the core problem:

If you change the number of primary shards, the mapping between document IDs and shards changes, and all existing documents are suddenly in the “wrong” shards.

To fix this, Elasticsearch would have to:

  • Read every existing document.
  • Recompute its shard using the new shard count.
  • Move it to the correct shard.
  • Essentially reindex the entire index.

This is a very expensive and very disruptive operation, and that is why Elasticsearch simply does not allow you to change the number of primary shards of an existing index.

What If the Index Is Empty?

There is one simple case where the problem disappears.

If:

  • You have just created the index, and
  • You have not indexed any documents yet,

Then the solution is straightforward:

  • Delete the index.
  • Recreate it with the correct number of shards.

Since there is no data yet, there is nothing to migrate and nothing to break.

What If the Index Already Has Data?

This is the more realistic and more common situation.

Once an index already contains data, you cannot change its primary shard count in place. The only correct and safe approach is:

  • Create a new index with the desired number of primary shards.
  • Reindex or migrate all documents from the old index into the new index.
  • Start using the new index going forward.

Elasticsearch provides APIs to help with this kind of migration, and we will see them in later chapters. Conceptually, however, what is happening is very simple:

You are not “changing” the shard count. You are creating a new index with a new layout and reindexing the data into it.

This is why shard count is something you must plan upfront, based on your expected data size and growth.

The Practical Rule: Plan Primary Shards Carefully at the Beginning

So the practical conclusion is:

You cannot change the number of primary shards of an existing index. You must plan this value carefully before you start indexing large amounts of data.

If you ever absolutely must change it, the only safe path is to create a new index and reindex the data.

Now the Second Question: How Many Replica Shards Should You Create?

The number of replica shards is a very different story from primary shards.

Replicas exist for two main reasons:

  • They provide high availability if a node fails.
  • They improve read performance, because search requests can be served from replicas as well.

In general, for any production environment, having at least one replica is strongly recommended. This means that every primary shard has at least one copy on another node.

Let us look at a concrete scenario to understand why this matters.

Thinking About Real Maintenance and Failures

Imagine a cluster where:

  • You have 4 primary shards.
  • You have 1 replica for each primary shard.
  • So for every piece of data, you have two copies in the cluster.

Now imagine that your operations team performs scheduled maintenance, such as:

  • Applying security patches.
  • Doing OS upgrades.
  • Rebooting machines.
  • Or upgrading Elasticsearch versions.

They might take two nodes down at the same time.

Now think about what could happen.

  • It is entirely possible that:
    • One of those nodes was holding a primary shard.
    • The other node was holding the replica of that same shard.
  • In that case, both copies of that shard are temporarily gone.
  • That part of your data becomes unavailable, and your application can no longer serve certain requests.

So even though you had one replica, your system was not highly available enough for your operational reality.

A More Realistic Way to Think About Replica Count

A much better way to think about replicas is this:

If you expect N nodes to be able to go down at the same time, you should have at least N replicas.

This ensures that:

  • Even if N nodes are unavailable,
  • There is still at least one copy of every shard left in the cluster,
  • And your system can continue to serve requests.

So replica count is not just a “performance” or “nice to have” setting. It is a direct expression of how much failure you want to tolerate.