Learnitweb

Why do we need Elasticsearch?

Introduction: Why Search Feels So Magical

Have you ever stopped to think about what actually happens when you type something into Google and press Enter?

Within a fraction of a second, you receive thousands—or even millions—of results. Not only that, but Google often starts predicting what you are about to type before you finish typing. This feels almost magical, yet it is the result of extremely advanced engineering and intelligent search systems working behind the scenes.

This experience is not limited to Google. Platforms such as Amazon, eBay, Walmart, and many others rely heavily on search to power their entire user experience. When users search for a product, they often do so with incomplete, vague, or even incorrect information—yet the system still delivers relevant results.

This tutorial is about understanding how such powerful search systems are built, why traditional databases are not sufficient for this task, and how Elasticsearch solves these challenges at scale.

Why Traditional Databases Are Not Enough for Search

Let us start with a simple thought experiment.

Imagine you have a massive table containing millions of products:

  • Product name
  • Description
  • Category
  • Price
  • Brand

At first glance, it may seem reasonable to store this data in a relational database and run SQL queries like:

SELECT * FROM products WHERE name LIKE '%iphone%';

But this approach breaks down very quickly when we deal with real-world behavior.

Problems with Traditional Databases for Search

  • They rely on exact or partial string matching, which means they struggle with typos, variations, and natural language input.
    For example, searching for iphnoe would fail unless explicitly handled.
  • Word order matters too much, even though humans do not think that way.
    Searching for “iPhone 15 Pro Max” and “iPhone Pro 15 Max” should return the same results, but traditional queries struggle with this.
  • Synonyms are not naturally understood.
    A database does not know that “phone cover” and “phone case” refer to the same thing unless explicitly programmed.
  • Scaling becomes extremely difficult when millions of records and high query traffic are involved.

This is why traditional relational databases are excellent for structured data and transactions—but not for intelligent search.

Understanding the Real Goal of Search

Search is not just about matching words.

It is about understanding user intent.

Consider this example:

  • A 20-year-old and a 70-year-old both walk into a store looking for a phone.
  • Their expectations, budget, and preferences are completely different.
  • A good salesperson will ask the right questions and guide them accordingly.

In the digital world, there is no human salesperson.

So the search engine must act as the salesperson by:

  • Understanding what the user really wants.
  • Interpreting vague or incomplete queries.
  • Suggesting relevant results.
  • Ranking results intelligently.

This is what modern search systems aim to achieve.

Key Challenges in Building a Search System

Let us break down the major challenges that any real-world search system must handle.

1. Understanding User Intent

Users often type incomplete or unclear queries. A good search engine must infer intent instead of relying only on literal matching.

2. Handling Synonyms

Different users may use different words to mean the same thing:

  • Cheap = Affordable
  • AWS = Amazon Web Services
  • Phone cover = Phone case

A search engine must recognize these as equivalent.

3. Handling Misspellings

Humans make mistakes. A good search system should gracefully handle typos like:

  • iphnoe instead of iphone
  • samsng instead of samsung

4. Providing Search Suggestions

Often users do not know exactly what they want.
Autocomplete suggestions help guide users toward meaningful queries.

5. Ranking Results Intelligently

When millions of results exist, users expect the most relevant results first, not just any matching results.

6. Filtering and Faceted Search

Especially in e-commerce, users expect filters such as:

  • Price range
  • Brand
  • Category
  • Ratings

These filters help narrow down results efficiently.

Why Elasticsearch Exists

All these challenges cannot be solved efficiently using traditional databases alone.

This is where Elasticsearch comes in.

Elasticsearch is:

  • Designed specifically for fast search.
  • Built for horizontal scalability.
  • Optimized for full-text search, filtering, and analytics.

It allows applications to perform complex searches across millions of documents in milliseconds.

Who Uses Elasticsearch?

Elasticsearch is not a niche technology. It is widely used by some of the biggest companies in the world:

  • Udemy uses it to power course searches.
  • LinkedIn uses it for professional and job searches.
  • Yelp uses it for business discovery.
  • Stack Overflow uses it to search questions and answers.
  • Tinder uses it for matching and recommendations.
  • NASA uses it for data exploration and analysis.

Even companies like Google and Amazon, despite having their own proprietary search engines, still use Elasticsearch internally for various use cases.