flinksql写入arctic表，报错is exceeded: limit=1040576

问题描述/异常栈

Caused by: org.apache.hadoop.ipc.RemoteException: The directory item limit of /user/ods/hive_db/ods.db/ods_mes_gaolan_task_process_1688_arctic_rt/change/data is exceeded: limit=1048576 items=1048576
    at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyMaxDirItems(FSDirectory.java:1144)
    at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addLastINode(FSDirectory.java:1201)
    at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addINode(FSDirectory.java:1052)
    at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.addFile(FSDirWriteFileOp.java:490)
    at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.startFile(FSDirWriteFileOp.java:368)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2383)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2301)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:733)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:413)
    at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:501)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
    at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:931)
    at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:870)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2712)

解决方案

用spark 执行，将表的历史数据保留时间缩短
alter table hive_catalog.ods.ods_mes_gaolan_task_process_1688_arctic_rt set tblproperties (
    'clean-orphan-file.enabled' = 'true');
alter table hive_catalog.ods.ods_mes_gaolan_task_process_1688_arctic_rt set tblproperties (
    'change.data.ttl.minutes' = '2880');
alter table hive_catalog.ods.ods_mes_gaolan_task_process_1688_arctic_rt set tblproperties (
    'snapshot.change.keep.minutes' = '2880');

问题原因

原因checkpoint时间短，历史数据保持时间长，小文件多。导致hdfs默认文件的items打满，调整arctic表的历史数据保留时间。

作者：邓崃翔