INFO-mysql2kafka，写入kafka数据太大超过请求的大小限制

适用模块

数据传输

具体说明

2025-08-05 20:43:17,314 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: MySQL Source, dataSourceId 5037 -> (ddl, tableKeyInjector -> split -> project -> Sink: ods_pol_ndh_t_rel_material_content) (1/3) (a9a5865d4fdb102562b16b0cf2f00d09) switched from RUNNING to FAILED on container_e12_1742969014254_120488_01_000002 @ ed08-s03b1508.dev.qizhidao.com (dataPort=34629).
org.apache.flink.util.FlinkRuntimeException: Failed to send data to Kafka null with FlinkKafkaInternalProducer{transactionalId='null', inTransaction=false, closed=false} 
    at org.apache.flink.connector.kafka.sink.KafkaWriter$WriterCallback.throwException(KafkaWriter.java:405) ~[plugin_ndi_ne-flink-1.14.0-1.0.13_scala2.12-release-3.28.0-2.21.0.jar:?]
    at org.apache.flink.connector.kafka.sink.KafkaWriter$WriterCallback.lambda$onCompletion$0(KafkaWriter.java:391) ~[plugin_ndi_ne-flink-1.14.0-1.0.13_scala2.12-release-3.28.0-2.21.0.jar:?]
    at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) ~[flink-dist_2.12-ne-flink-1.14.0-1.0.18.3.jar:ne-flink-1.14.0-1.0.18.3]
    at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90) ~[flink-dist_2.12-ne-flink-1.14.0-1.0.18.3.jar:ne-flink-1.14.0-1.0.18.3]
    at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsNonBlocking(MailboxProcessor.java:353) ~[flink-dist_2.12-ne-flink-1.14.0-1.0.18.3.jar:ne-flink-1.14.0-1.0.18.3]
    at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:317) ~[flink-dist_2.12-ne-flink-1.14.0-1.0.18.3.jar:ne-flink-1.14.0-1.0.18.3]
    at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:201) ~[flink-dist_2.12-ne-flink-1.14.0-1.0.18.3.jar:ne-flink-1.14.0-1.0.18.3]
    at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:809) ~[flink-dist_2.12-ne-flink-1.14.0-1.0.18.3.jar:ne-flink-1.14.0-1.0.18.3]
    at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:761) ~[flink-dist_2.12-ne-flink-1.14.0-1.0.18.3.jar:ne-flink-1.14.0-1.0.18.3]
    at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958) ~[plugin_ndi_ne-flink-1.14.0-1.0.13_scala2.12-release-3.28.0-2.21.0.jar:?]
    at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937) ~[plugin_ndi_ne-flink-1.14.0-1.0.13_scala2.12-release-3.28.0-2.21.0.jar:?]
    at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766) ~[plugin_ndi_ne-flink-1.14.0-1.0.13_scala2.12-release-3.28.0-2.21.0.jar:?]
    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575) ~[plugin_ndi_ne-flink-1.14.0-1.0.13_scala2.12-release-3.28.0-2.21.0.jar:?]
    at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_152]
Caused by: org.apache.kafka.common.errors.RecordTooLargeException: The message is 1196954 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration. 我应该修改flink的配置还是去向kafka

使用示例

添加任务参数：target.sink.options.properties.max.request.size

作者：潘晓宇