This article explains how to fix the following error when running a hive query:

MAP capability required is more than the supported max container capability in the cluster. 
Killing the Job. mapResourceRequest: <memory:1638, vCores:1> maxContainerCapability:<memory:1200, vCores:2>

rtaImage

This error might not be obvious, however, this is caused by the following config not setup properly in YARN:

mapreduce.map.memory.mb = 1638
yarn.scheduler.maximum-allocation-mb = 1200
yarn.nodemanager.resource.memory-mb = 1300

The solution here is to change the above mentioned property to have the following values:

mapreduce.map.memory.mb < yarn.nodemanager.resource.memory-mb < yarn.scheduler.maximum-allocation-mb mapreduce.map.memory.mb is the amount of heap can be allocated to map task, which is inside a container. yarn.nodemanager.resource.memory-mb is the amount of heap to be allocated to the container that will hold either mappers or reducers. yarn.scheduler.maximum-allocation-mb is the amount of maximum memory can be allocated to container, which need to be higher than yarn.nodemanager.resource.memory-mb. Then the problem will be solved. Hope this helps.

Leave a Reply

Your email address will not be published. Required fields are marked *