Learnitweb

Understanding Master Election in an Elasticsearch Cluster

In this tutorial, we are going to observe and understand the master election process in Elasticsearch by performing a few controlled experiments on our already running three-node cluster.

Until now, we have been using the cluster normally, but we have not really tested what happens when nodes go down and come back up. This is exactly the kind of situation Elasticsearch is designed to handle in production, and the master election process is at the heart of this reliability.

Before proceeding further, please make sure:

Kibana is accessible and you can use the Dev Tools → Console.

Your three-node cluster (es01, es02, es03) is already up and running using Docker Compose.

Step 1: Check the Current Cluster State

First, let us see which node is currently acting as the master.

In Kibana Dev Tools, run:

GET _cat/nodes?v

You should see three nodes:

  • es01
  • es02
  • es03

One of them will be marked with a * in the master column.

In our example, es01 is the master. In your case, it might be es02 or es03, and that is perfectly fine. The exact node does not matter. What matters is that exactly one node is the master.

Step 2: Stop the Current Master Node

Now we are going to intentionally kill the master and observe what happens.

Open a new terminal and go to the folder where your docker-compose.yml file exists.

Since in this example es01 is the master, run:

docker compose stop es01

Docker will stop the container.

Now go back to Kibana and run again:

GET _cat/nodes?v

You will notice two important things:

  • es01 is gone from the list.
  • One of the remaining nodes, for example es02, has now become the new master.

This proves something extremely important:

Elasticsearch automatically elects a new master when the current master disappears.

Step 3: Start the Old Master Again

Now let us bring es01 back.

In the terminal, run:

docker compose start es01

Do not expect it to appear instantly in the cluster.

Elasticsearch needs some time to:

  • Start the JVM
  • Initialize internal services
  • Discover the cluster
  • Join the cluster

After a short wait, run again:

GET _cat/nodes?v

You will see:

  • es01 has rejoined the cluster.
  • es02 is still the master.

This is very important behavior:

Just because a node comes back does not mean it will automatically take back the master role.

Once a new master is elected and the cluster is stable, Elasticsearch does not keep re-electing masters unnecessarily.

Step 4: Stop the Current Master Again

Now let us kill the new master.

If es02 is the current master, run:

docker compose stop es02

Now go back to Kibana and run:

GET _cat/nodes?v

You will now see:

  • es03 has become the new master.

So far, everything looks very logical and reassuring:

As long as there are at least two nodes alive, Elasticsearch can always elect a master.

Step 5: Stop All Remaining Nodes One by One

Now let us push the cluster into a more extreme situation.

Stop es03 as well:

docker compose stop es03

At this point:

  • es01 is running
  • es02 is stopped
  • es03 is stopped

Now try in Kibana:

GET _cat/nodes?v

You will either get:

  • No response
  • Or a connection error
  • Or a timeout

You might think:

“But es01 is still running. Why does it not become the master?”

This is the most important learning moment of this chapter.

Why a Single Node Cannot Elect Itself as Master

In a multi-node cluster, master election is a voting process.

Conceptually:

  • A node cannot vote for itself and declare itself master.
  • It needs at least one other node to agree.

In practical terms:

A multi-node Elasticsearch cluster requires a majority (quorum) of master-eligible nodes to be alive in order to function.

In our three-node cluster:

  • To form a working cluster, at least two nodes must be alive.
  • With only one node alive, no majority exists.
  • Therefore, the cluster cannot exist at all, even though one Elasticsearch process is technically still running.

This is why:

es01 alone cannot form a cluster and cannot declare itself master.

If you truly want a single-node cluster, you must explicitly configure Elasticsearch with:

discovery.type=single-node

But that is not what we are doing here. We are running a real multi-node cluster.