Second Brain

Understanding FetchFailedException Message

2 min read

Debugging complex Spark Applications requires the ability to understand what the thrown exception message actually means.

Here’s what FetchFailedException looks like and how we understand what it’s trying to tell:

Lost task 71.0 in stage 436.0 (TID 16079) (ip-192-168-247-152.emretl.prod01-crdrp.net executor 34): Fetch Failed(BlockManagerId(12, ip-192-168-205-146.emretl.prod01-crdrp.net, 7337, None), shuffleId=80, mapIndex=27, mapId=15506, reduceId=70, message=org.apache.spark.shuffle.FetchFailedException
......)

It’s pretty clear that this is a reducer stage, i.e., Stage reading data processed by the previous stage.

Breakdown

This exception message has 3 sections:

Reducer Stage Details

  • Current Stage ID: 436.0 (0 represents the very first run)
  • Task Index: 71.0 (0 represents the First Task Attempt). The Task Index is the position of a task within a stage.
  • TID: Unique identifier for a Task in a job.

Remote Executor Details

The entire block in BlockManagerId(...) provides the details from where task 71 of stage 436 is trying to fetch shuffle data from.

  • Executor ID: 12
  • Executor Node IP: ip-192-168-205-146.emretl.prod01-crdrp.net
  • Port on Node: 7337 (This is External Shuffle Service Port)

Data Block Details

  • shuffleId: Unique identifier for the shuffle operation across the entire Spark application. Each shuffle stage gets a unique ID.
  • mapIndex: The partition index within the map stage that produced the data being fetched
  • mapId=15504: Unique identifier for the specific map task that generated the shuffle data.
  • reduceId: The partitionID of the reduce task that was trying to fetch the data.

How it’s read?

This is FetchFailedException, indicating that:

  • Task 71.0 in Stage 436.0 failed while trying to fetch shuffle data
  • The reduce task (partition 69) couldn’t retrieve data generated by a remote executor with id 12 ran on node ip-192-168-205-146.emretl.prod01-crdrp.net
  • The data was originally produced by map task 16079 (partition 27) in shuffle 80
  • The failure occurred when executor 34 tried to fetch from BlockManager on executor 12 or External Shuffle Service (based on port 7337) running on node ip-192-168-205-146.emretl.prod01-crdrp.net

Graph View

Backlinks