任务运行时异常：Buffer pool is destroyed.

问题描述/异常栈

java.lang.RuntimeException: Buffer pool is destroyed.
    at org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:110) ~[flink-streaming-java_2.11-1.8.1.jar:1.8.1]
    at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:89) ~[flink-streaming-java_2.11-1.8.1.jar:1.8.1]
    at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:45) ~[flink-streaming-java_2.11-1.8.1.jar:1.8.1]
    at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:718) ~[flink-streaming-java_2.11-1.8.1.jar:1.8.1]
    at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:696) ~[flink-streaming-java_2.11-1.8.1.jar:1.8.1]
    at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41) ~[flink-streaming-java_2.11-1.8.1.jar:1.8.1]
    at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:579) ~[flink-streaming-java_2.11-1.8.1.jar:1.8.1]
    at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:554) ~[flink-streaming-java_2.11-1.8.1.jar:1.8.1]
    at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:534) ~[flink-streaming-java_2.11-1.8.1.jar:1.8.1]
    at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:718) ~[flink-streaming-java_2.11-1.8.1.jar:1.8.1]
    at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:696) ~[flink-streaming-java_2.11-1.8.1.jar:1.8.1]
    at org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collect(StreamSourceContexts.java:104) ~[flink-streaming-java_2.11-1.8.1.jar:1.8.1]
    at com.ucarinc.framework.flink.connectors.flexq.FlexQSource.run(FlexQSource.java:204) ~[flink-connector-flexq-1.8.500-20191206.054312-28.jar:1.8.500-SNAPSHOT]
    at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:93) ~[flink-streaming-java_2.11-1.8.1.jar:1.8.1]
    at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:57) ~[flink-streaming-java_2.11-1.8.1.jar:1.8.1]
    at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:97) ~[flink-streaming-java_2.11-1.8.1.jar:1.8.1]
    at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:300) ~[flink-streaming-java_2.11-1.8.1.jar:1.8.1]
    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711) [flink-runtime_2.11-1.8.1.jar:1.8.1]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_31]
Caused by: java.lang.IllegalStateException: Buffer pool is destroyed.
    at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.internalRequestMemorySegment(LocalBufferPool.java:264) ~[flink-runtime_2.11-1.8.1.jar:1.8.1]
    at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestMemorySegment(LocalBufferPool.java:240) ~[flink-runtime_2.11-1.8.1.jar:1.8.1]
    at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBufferBuilderBlocking(LocalBufferPool.java:218) ~[flink-runtime_2.11-1.8.1.jar:1.8.1]
    at org.apache.flink.runtime.io.network.api.writer.RecordWriter.requestNewBufferBuilder(RecordWriter.java:264) ~[flink-runtime_2.11-1.8.1.jar:1.8.1]
    at org.apache.flink.runtime.io.network.api.writer.RecordWriter.getBufferBuilder(RecordWriter.java:257) ~[flink-runtime_2.11-1.8.1.jar:1.8.1]
    at org.apache.flink.runtime.io.network.api.writer.RecordWriter.copyFromSerializerToTargetChannel(RecordWriter.java:177) ~[flink-runtime_2.11-1.8.1.jar:1.8.1]
    at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:162) ~[flink-runtime_2.11-1.8.1.jar:1.8.1]
    at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:128) ~[flink-runtime_2.11-1.8.1.jar:1.8.1]
    at org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:107) ~[flink-streaming-java_2.11-1.8.1.jar:1.8.1]
    ... 18 more

解决方案

高级参数种添加：taskmanager.memory.network.fraction 0.2 （默认值为0.1，可根据实际情况适当调整）

问题原因

一般为为任务network buffer不足。可以调整下任务的network buffer的大小。

作者：邓崃翔