功能说明

适用模块

hivesql任务mr执行时报错oom，但日志打印时调用ranger的报错。

具体说明

16-12-2021 09:54:31 CST Serializing ReduceWork using kryo
Exception in thread "Thread-18" java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:3181)
    at java.util.ArrayList.grow(ArrayList.java:261)
    at java.util.ArrayList.ensureExplicitCapacity(ArrayList.java:235)
    at java.util.ArrayList.ensureCapacityInternal(ArrayList.java:227)
    at java.util.ArrayList.add(ArrayList.java:458)
    at org.apache.ranger.plugin.util.TrieNode.addPolicy(RangerResourceTrie.java:280)
    at org.apache.ranger.plugin.util.TrieNode.addPolicies(RangerResourceTrie.java:287)
    at org.apache.ranger.plugin.util.TrieNode.postSetup(RangerResourceTrie.java:311)
    at org.apache.ranger.plugin.util.TrieNode.postSetup(RangerResourceTrie.java:327)
    at org.apache.ranger.plugin.util.TrieNode.postSetup(RangerResourceTrie.java:327)
    at org.apache.ranger.plugin.util.TrieNode.postSetup(RangerResourceTrie.java:327)
    at org.apache.ranger.plugin.util.TrieNode.postSetup(RangerResourceTrie.java:327)
    at org.apache.ranger.plugin.util.TrieNode.postSetup(RangerResourceTrie.java:327)
    at org.apache.ranger.plugin.util.TrieNode.postSetup(RangerResourceTrie.java:327)
    at org.apache.ranger.plugin.util.TrieNode.postSetup(RangerResourceTrie.java:327)
    at org.apache.ranger.plugin.util.TrieNode.postSetup(RangerResourceTrie.java:327)
    at org.apache.ranger.plugin.util.TrieNode.postSetup(RangerResourceTrie.java:327)
    at org.apache.ranger.plugin.util.TrieNode.postSetup(RangerResourceTrie.java:327)
    at org.apache.ranger.plugin.util.TrieNode.postSetup(RangerResourceTrie.java:327)
    at org.apache.ranger.plugin.util.TrieNode.postSetup(RangerResourceTrie.java:327)
    at org.apache.ranger.plugin.util.TrieNode.postSetup(RangerResourceTrie.java:327)
    at org.apache.ranger.plugin.util.TrieNode.postSetup(RangerResourceTrie.java:327)
    at org.apache.ranger.plugin.util.TrieNode.postSetup(RangerResourceTrie.java:327)
    at org.apache.ranger.plugin.util.TrieNode.postSetup(RangerResourceTrie.java:327)
    at org.apache.ranger.plugin.util.RangerResourceTrie.<init>(RangerResourceTrie.java:90)
    at org.apache.ranger.plugin.policyengine.RangerPolicyRepository.initResourceTries(RangerPolicyRepository.java:264)
    at org.apache.ranger.plugin.policyengine.RangerPolicyRepository.<init>(RangerPolicyRepository.java:133)
    at org.apache.ranger.plugin.policyengine.RangerPolicyEngineImpl.<init>(RangerPolicyEngineImpl.java:102)
    at org.apache.ranger.plugin.service.RangerBasePlugin.setPolicies(RangerBasePlugin.java:163)
    at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:203)
    at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:162)

问题原因

该问题并非常见的mr端的oom可以在sql中设置参数。通过日志信息可以判断出报错原因是ranger拉策略文件内存溢出了，策略文件比较大。此时把hiveclient的提交方式切换为hive jdbc。问题即可解决。

作者：刘思伟