万象数据库ETCD存储耗尽
更新时间: 2025-11-03 20:32:43
阅读 134
FAQ-万象数据库-database space exceeded
问题描述/异常栈
万象数据库出现服务异常告警,agent-common 出现database space exceeded 关键字[251102.115831][MainThread] E Error when update agent config with etcd. <class 'grpc._channel._InactiveRpcError'>/<_InactiveRpcError of RPC that terminated with:
status = StatusCode.RESOURCE_EXHAUSTED
details = "etcdserver: mvcc: database space exceeded"
debug_error_string = "{"created":"@1762055911.949140652","description":"Error received from peer ipv4:111.124.204.139:2379","file":"src/core/lib/surface/call.cc","file_line":1066,"grpc_message":"etcdserver: mvcc: database space exceeded","grpc_status":8}"发现版本
万象数据库-3.3.7.3解决方案
临时方案:压缩并清理etcd存储数据 长期方案:升级ETCD服务,开启自动压缩清理# 临时方案操作步骤
## 登录etcd容器
docker exec -it mysql-paas-etcd /bin/bash
## 执行压缩
rev=`etcdctl --endpoints=$ETCD_LISTEN_CLIENT_URLS endpoint status --write-out="json" | egrep -o '"revision":[0-9]*' | egrep -o '[0-9].*'`
result=`etcdctl --endpoints=$ETCD_LISTEN_CLIENT_URLS compact "${rev}"`
## 此命令可能出现超时,等待10分钟左右重试一般可执行成功
result=`etcdctl --endpoints=$ETCD_LISTEN_CLIENT_URLS defrag`
size=`etcdctl --endpoints=$ETCD_LISTEN_CLIENT_URLS endpoint status|awk -F ',' '{print $4}'`
echo "压缩后的etcd数据大小如下"
echo $size
# 长期方案参考灵犀文档:https://docs.popo.netease.com/lingxi/382da0713ab84b579fad3bf188ec85c5
问题原因
etcd未开启自动压缩清理功能,导致数据持续增长直到超出存储空间配额。作者:Simon
文档反馈
以上内容对您是否有帮助?