This post explain what you need to do when you see container “is running beyond physical memory limits” error in the hive application log. Symptoms: Hive query failed with the following error in the application log:
2015-07-06 09:53:30,004 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: 
Diagnostics report from attempt_1435155934213_16721_m_002655_0: 
Container [pid=46028,containerID=container_e54_1435155934213_16721_01_003666] is running beyond physical memory limits. 
Current usage: 1.0 GB of 1 GB physical memory used; 2.7 GB of 2.1 GB virtual memory used. Killing container.
Cause: There is not enough memory allocated to each container that will trigger the map tasks Solution: Find out the current memory for container and map task
SET mapreduce.map.memory.mb;
SET mapreduce.map.java.opts;
Then increase the heap size for both of them in Hive, the heap size for map task should slightly smaller than the container memory
SET mapreduce.map.memory.mb=2048;
SET mapreduce.map.java.opts=-Xmx1800M;
Then run the query again.

    Leave a Reply

    Your email address will not be published. Required fields are marked *