How ExecutorService manage threads?

1. Overview: What `ExecutorService` Really Is

At a high level, ExecutorService is an abstraction built on top of thread pools.
When you submit a task (a Runnable or Callable), the service does not create a new thread every time.
Instead, it reuses a fixed set of worker threads that continuously fetch and execute tasks from a queue.

So the architecture looks like this:

+---------------------------------------------------------+
|                    ExecutorService                      |
+---------------------------------------------------------+
| Thread Pool (Worker Threads) | Task Queue (WorkQueue)   |
+---------------------------------------------------------+

Let’s break it into components.

2. Key Internal Components

Internally, all major ExecutorService implementations are built on ThreadPoolExecutor, which is the real workhorse class in the java.util.concurrent package.

Components of ThreadPoolExecutor:

Core Pool Size (corePoolSize)
Minimum number of threads to keep alive in the pool, even if idle.
Maximum Pool Size (maximumPoolSize)
Maximum number of threads allowed. When the queue is full, new threads are created up to this number.
Work Queue (BlockingQueue<Runnable>)
Holds tasks that are waiting to be executed when all core threads are busy.
Common implementations:
- LinkedBlockingQueue (unbounded queue)
- SynchronousQueue (no storage; directly hands off tasks)
- ArrayBlockingQueue (bounded queue)
Thread Factory (ThreadFactory)
Used to create new threads with custom names, priorities, etc.
RejectedExecutionHandler (handler)
Handles tasks that cannot be executed when the pool is full and shut down.
Common strategies:
- AbortPolicy (default) → throws RejectedExecutionException
- CallerRunsPolicy → runs task in the calling thread
- DiscardPolicy / DiscardOldestPolicy → silently drop tasks
Keep Alive Time (keepAliveTime)
Time an idle thread waits before terminating when the number of threads is greater than corePoolSize.

3. Internal Workflow: Step-by-Step Execution

Let’s trace what happens when you call:

executor.submit(task);

Step 1: Task submission

When you submit a task:

It gets passed to ThreadPoolExecutor.execute(Runnable command).

Step 2: Check core threads

Inside execute():

If the current thread count < corePoolSize
→ a new worker thread is created immediately to execute the task.
Else, the task is added to the work queue.

if (workerCount < corePoolSize)
    addWorker(command, true);
else if (isRunning && workQueue.offer(command))
    // Queue accepts the task, wait for a free thread

Step 3: If the queue is full

If the queue is full:

The pool checks if the current thread count < maximumPoolSize.
- If yes, it creates a new thread to handle the task.
- If not, the task is rejected and handled by the RejectedExecutionHandler.

Step 4: Worker threads fetching tasks

Each worker thread runs this loop internally:

while (task != null || (task = getTask()) != null) {
    try {
        task.run();
    } finally {
        task = null;
    }
}

The getTask() method takes the next task from the work queue.
If the queue is empty, the thread waits (blocking) until a new task arrives or the executor is shut down.

Step 5: Task execution

The worker thread executes the Runnable or Callable.
If you used submit(), it wraps it in a FutureTask, which stores the result or exception for later retrieval via Future.get().

Step 6: Thread reuse

Once the task completes:

The worker thread doesn’t die.
It loops back to fetch the next task from the queue.

That’s how threads are reused efficiently.

Step 7: Shutdown handling

When you call:

executor.shutdown();

The pool stops accepting new tasks.
Existing tasks in the queue are executed.
Once the queue is empty and all tasks are complete, worker threads are terminated.

If you call:

executor.shutdownNow();

It tries to interrupt running threads and clears the queue immediately.

4. Visualization of Execution Flow

Client Threads
    |
    | submit(task)
    v
+-----------------------+
|   ThreadPoolExecutor  |
+-----------------------+
|   1. Check thread count   |
|   2. If < corePoolSize → create thread  |
|   3. Else enqueue task    |
|   4. If queue full → new thread or reject |
+-----------------------+
    |
    v
Worker Threads
    |
    | getTask() from queue
    v
execute(task.run())
    |
    +--> loop until shutdown

5. Example: Fixed Thread Pool

When you call:

ExecutorService executor = Executors.newFixedThreadPool(3);

Internally, it does this:

return new ThreadPoolExecutor(
    3,                // corePoolSize
    3,                // maximumPoolSize
    0L,               // keepAliveTime
    TimeUnit.MILLISECONDS,
    new LinkedBlockingQueue<Runnable>()  // unbounded queue
);

Only 3 threads are ever created.
Extra tasks are stored in the queue until a thread becomes free.
Threads are never terminated because keepAliveTime = 0 and queue is unbounded.

6. Example: Cached Thread Pool

ExecutorService executor = Executors.newCachedThreadPool();

Internally:

return new ThreadPoolExecutor(
    0,                      // corePoolSize
    Integer.MAX_VALUE,      // maximumPoolSize
    60L,                    // keepAliveTime
    TimeUnit.SECONDS,
    new SynchronousQueue<Runnable>()
);

No core threads; threads are created as needed.
If a thread stays idle for 60 seconds, it’s removed.
Suitable for many short-lived tasks.

7. How `Future` Works Internally

When you use:

Future<Integer> result = executor.submit(() -> 10);

The callable is wrapped in a FutureTask object.
FutureTask implements both Runnable and Future.
It runs in the thread pool as a normal task.
Once complete, it stores the return value or exception internally.
When you call result.get(), it either:
- Returns the computed value.
- Waits (blocks) until computation finishes.
- Throws an exception if the task failed or was cancelled.

8. Internal Classes You Should Know

ThreadPoolExecutor.Worker: Represents a worker thread that executes tasks.
FutureTask: Bridges Runnable and Callable and manages result retrieval.
BlockingQueue: Controls how tasks are queued.
RejectedExecutionHandler: Handles rejected tasks.

9. Advantages of This Internal Design

Thread reuse reduces creation overhead.
Queue-based scheduling ensures fair execution.
Flexible configuration for concurrency control.
Graceful shutdown and proper error handling.
Decoupling of task submission and execution.
Futures enable async computation and result retrieval.