组件功能测试

1 前置步骤
2 组件服务功能测试

1 前置步骤

1.1 LDAP 增加用户

1. 查看LDAP配置信息进入EasyOps点击左侧服务LDAP，进入开发者调试模式，并点击右侧LDAP服务实例名，查看api/v1/service_instance_detail接口返回结果获取以下配置，如下图所示

bind_dn: "cn=admin,dc=bdms,dc=com"
ldap_base_dc: "dc=bdms,dc=com"
password: "JUc7FE4RkTYvnCCEHxB5"
port: "1389"

2. 新建ndh_test.ldif文件登录LDAP客户端部署节点，新建ndh_test.ldif，内容如下

# 替换文本中dc=bdms,dc=com 为步骤1.1中ldap_base_dc配置
dn: uid=ndh_test_user,ou=People,dc=bdms,dc=com
objectClass: top
objectClass: account
objectClass: posixAccount
objectClass: shadowAccount
cn: ndh_test_user
uid: ndh_test_user
uidNumber: 30001
gidNumber: 80001
homeDirectory: /home/ndh_test_user
loginShell: /bin/bash
gecos: Demo [Demo user (at) bdms]
userPassword: {crypt}x
shadowLastChange: 17058
shadowMin: 0
shadowMax: 99999
shadowWarning: 7

dn: cn=ops,ou=Group,dc=bdms,dc=com
objectClass: posixGroup
objectClass: top
cn: ops
gidNumber: 80001

3. 增加用户ndh_test_user

# 替换-D中 'cn=admin,dc=bdms,dc=com'为步骤1.1中获取的bind_dn配置
# 替换-H中 'ldap://xnode3.local:1389'为LDAPserver部署机器及1.1中获取的port
ldapadd -x -W -D 'cn=admin,dc=bdms,dc=com' -f ndh_test.ldif -H 'ldap://xnode3.local:1389'

4. 确认用户是否创建成功登录任一LDAP client部署机器，执行id ndh_test_user

[root@xnode1 ~]# id ndh_test_user
uid=30001(ndh_test_user) gid=80001 groups=80001

1.2 Kerbores 增加Principal

1. 登录kerberos master服务部署机器执行kadmin.local，创建principal: ndh_test_user

# 密码可随意设置
[root@xnode2 ~]# kadmin.local
Authenticating as principal hdfs/admin@BDMS.COM with password.
kadmin.local:
kadmin.local:  add_principal ndh_test_user
WARNING: no policy specified for ndh_test_user@BDMS.COM; defaulting to no policy
Enter password for principal "ndh_test_user@BDMS.COM":
Re-enter password for principal "ndh_test_user@BDMS.COM":
Principal "ndh_test_user@BDMS.COM" created.

2. 生成对应的keytab文件

执行xst -k /etc/security/keytabs/ndh_test_user.keytab -norandkey ndh_test_user@BDMS.COM

[root@xnode2 ~]# kadmin.local
Authenticating as principal hdfs/admin@BDMS.COM with password.
kadmin.local:
kadmin.local:  xst -k /etc/security/keytabs/ndh_test_user.keytab -norandkey ndh_test_user@BDMS.COM
Entry for principal ndh_test_user@BDMS.COM with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:ndh_test_user.keytab.
Entry for principal ndh_test_user@BDMS.COM with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:ndh_test_user.keytab.
Entry for principal ndh_test_user@BDMS.COM with kvno 1, encryption type des3-cbc-sha1 added to keytab WRFILE:ndh_test_user.keytab.
Entry for principal ndh_test_user@BDMS.COM with kvno 1, encryption type arcfour-hmac added to keytab WRFILE:ndh_test_user.keytab.
Entry for principal ndh_test_user@BDMS.COM with kvno 1, encryption type camellia256-cts-cmac added to keytab WRFILE:ndh_test_user.keytab.
Entry for principal ndh_test_user@BDMS.COM with kvno 1, encryption type camellia128-cts-cmac added to keytab WRFILE:ndh_test_user.keytab.
Entry for principal ndh_test_user@BDMS.COM with kvno 1, encryption type des-hmac-sha1 added to keytab WRFILE:ndh_test_user.keytab.
Entry for principal ndh_test_user@BDMS.COM with kvno 1, encryption type des-cbc-md5 added to keytab WRFILE:ndh_test_user.keytab.

3. kerbores认证

[root@xnode2 ~]# kinit -kt  /etc/security/keytabs/ndh_test_user.keytab ndh_test_user@BDMS.COM
[root@xnode2 ~]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: ndh_test_user@BDMS.COM

Valid starting       Expires              Service principal
10/25/2022 23:20:37  10/26/2022 09:20:37  krbtgt/BDMS.COM@BDMS.COM
        renew until 11/01/2022 23:20:37

4. 拷贝keytab文件至测试客户端所在机器

1.3 YARN 资源池配置

1. 登录EasyOps进入YARN 服务

2. 创建label，并添加资源 3. 新建资源池注意：

• 新增、修改队列，要确保目前已有队列的容量总和为100%
• 最大AM比例此需要填写为0-1
• 新增、修改队列，多队列共用同一label时，需要确保多队列共用的label资源和为100%，否则会导致同步配置失败

4. 保存并同步配置 5. 进入YARN UI查看队列是否生效

1.4 Hive 测试库预置

1. 登录Hive Client部署机器 2. 执行kinit认证操作：通过hive用户认证

# 执行klist查看Principal
klist -kt /etc/security/keytabs/hive/hive.service.keytab
# 获取Principal后执行kinit认证
kinit -kt /etc/security/keytabs/hive/hive.service.keytab hive/xnode2.local@BDMS.COM

3. 通过hive命令创建测试库、表

create database if not exists ndh_test_db;
create table if not exists ndh_test_db.ndh_base_tbl(id int, name string);

1.5 HDFS 目录预置

1. 登录HDFS Client部署机器

2. 执行kinit认证操作：通过hdfs用户认证

# 执行klist查看Principal
klist -kt /etc/security/keytabs/hdfs/hdfs.keytab
# 获取Principal后执行kinit认证
kinit -kt /etc/security/keytabs/hdfs/hdfs.keytab hdfs/xnode2.local@BDMS.COM

3. 创建新用户对应HDFS目录并授权

hdfs dfs -mkdir /user/ndh_test_user
hdfs dfs -chown ndh_test_user:hadoop /user/ndh_test_user
hdfs dfs -chmod 755 /user/ndh_test_user

1.6 EasyRanger 权限预置

1. 通过EasyRanger quickUrls进入RangerAdmin管理页面，用户名密码默认为admin/Ranger@163 2. 创建用户：ndh_test_user

3. 创建hive 权限规则，将库ndh_test_db权限授权ndh_test_user 下拉至最下方，保存即可。

4. 创建HDFS权限规则，将库路径/user/warehouse/ndh_test_db.db 权限授权ndh_test_user用户

2 组件服务功能测试

2.1 HDFS 服务功能测试

2.1.1 前置操作

1. 登录HDFS Client部署服务器

2. 执行kinit认证操作

# 执行klist查看Principal
klist -kt /etc/security/keytabs/ndh_test_user.keytab
# 获取Principal后执行kinit认证
kinit -kt /etc/security/keytabs/ndh_test_user.keytab  ndh_test_user@BDMS.COM

2.1.2 验证步骤

1. 查看hdfs文件目录

hdfs dfs -ls /

2. 在hdfs上创建文件并写入数据

hdfs dfs -touchz  /user/ndh_test_user/test_file.txt
echo "<Text to append, ndh test: word count case>" | hdfs dfs -appendToFile - /user/ndh_test_user/test_file.txt

3. 从hdfs上下载文件

hdfs dfs -get /user/ndh_test_user/test_file.txt

2.1.3 预期结果

1. 命令正常执行

2. 创建文件后可在hdfs文件系统查看到 3. 成功下载到上一步的文件，查看其内容为上一步写入的内容

2.2 Yarn 服务功能测试

2.2.1 前置操作

1. 登录YARN Client部署服务器

2. 执行kinit认证操作

# 执行klist查看Principal
klist -kt /etc/security/keytabs/ndh_test_user.keytab
# 获取Principal后执行kinit认证
kinit -kt /etc/security/keytabs/ndh_test_user.keytab  ndh_test_user@BDMS.COM

3. 查看YarnClient部署目录

查看EasyOps服务部署根目录（BASE_DIR），如/home/bdms 进入BASE_DIR,并逐级找到Yarn Client部署目录（其中/20221018113442217309170ee为组件实例名，不同机器上不同），如下所示的/home/bdms/yarn/default_yarn/client//20221018113442217309170ee

# 替换/home/bdms为上述获取的EasyOps服务部署的根目录BASE_DIR
[root@xnode3 ~]# cd /home/bdms/
[root@xnode3 bdms]# ls
hadoop  hdfs  impala  java  knox  kyuubi  logs  monitor  nginx_ha  spark2  yarn  zookeeper
[root@xnode3 bdms]# cd yarn
[root@xnode3 yarn]# ls
default_yarn   package_shared
[root@xnode3 yarn]# cd dafault_yarn/
[root@xnode3 default_yarn]# ls
client  historyserver  nodemanager  resourcemanager
[root@xnode3 default_yarn]# cd client/
[root@xnode3 client]# ls
/20221018113442217309170ee
# 此处若组件安装多次，则选择目录创建时间最近的
[root@xnode3 client]# cd /20221018113442217309170ee/
[root@xnode3 /20221018113442217309170ee]# ls
config  current  data  keytab  logs pid
[root@xnode3 /20221018113442217309170ee]# pwd
/home/bdms/yarn/default_yarn/client/20221018113442217309170ee

2.2.2 验证步骤

1. 执行MR example wordcount任务

# 若已执行过，则需要删除输出目录
hdfs dfs -rmr -skipTrash /tmp/input
hdfs dfs -rmr -skipTrash /tmp/output 

hdfs dfs -mkdir -p /tmp/input
hdfs dfs -touchz /tmp/input/ndh_test_file.txt
echo "<Text to append, ndh test: word count case>" | hdfs dfs -appendToFile - /tmp/input/ndh_test_file.txt

# 指定EXAMPLE_JAR_DIR为上述查看的YARN Client部署目录
EXAMPLE_JAR_DIR=/home/bdms/yarn/default_yarn/client/20221018113442217309170ee
source ${EXAMPLE_JAR_DIR}/config/hadoop-env.sh
# 执行队列为前置步骤中创建的队列名
hadoop jar ${EXAMPLE_JAR_DIR}/current/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.0.jar wordcount -Dmapreduce.job.queuename=ndh_test /tmp/input /tmp/output

2. 执行过程中，打开YARN UI，查看正在运行中名为word count的MR任务

2.2.3 预期结果

1. 任务正常执行完成

2. 通过MR HistoryServer正常查看该任务

2.3 Hive 服务功能测试

2.3.1 前置操作

1. 登录Hive Client 部署服务器 2. 执行kinit认证操作：通过ndh_test_user keytab认证

# 执行klist查看Principal
klist -kt /etc/security/keytabs/ndh_test_user.keytab
# 获取Principal后执行kinit认证
kinit -kt /etc/security/keytabs/ndh_test_user.keytab ndh_test_user@BDMS.COM

如下图所示 3. 通过EasyOps查看HiveServer2服务连接串，如下所示：

4. hive_link:"jdbc:hive2://xnode3.local:2182,xnode4.local:2182,xnode2.local:2182/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;principal=hive/_HOST@BDMS.COM"

2.3.2 验证步骤

1. Hive基本功能验证

通过beeline连接HiveServer2，并执行以下SQL

# 通过beeline连接
beeline -u "jdbc:hive2://xnode3.local:2182,xnode4.local:2182,xnode2.local:2182/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;principal=hive/_HOST@BDMS.COM?mapreduce.job.queuename=ndh_test"
# 连接成功后，分别执行以下SQL
use ndh_test_db;  
create table hive_tbl_01(id int,name string,address string);
insert into table hive_tbl_01 select 1,"joey","zj";
select * from hive_tbl_01;
select count(1) from hive_tbl_01;
drop table hive_tbl_01;

执行记录如下（已过滤部分日志）：

[root@xnode3 ~]# beeline -u "jdbc:hive2://xnode3.local:2182,xnode4.local:2182,xnode2.local:2182/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;principal=hive/_HOST@BDMS.COM?mapreduce.job.queuename=ndh_test"
Connecting to jdbc:hive2://xnode3.local:2182,xnode4.local:2182,xnode2.local:2182/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;principal=hive/_HOST@BDMS.COM?mapreduce.job.queuename=ndh_test
log4j:WARN No appenders could be found for logger (org.apache.hive.jdbc.Utils).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Connected to: Apache Hive (version 2.3.8-1.1.0)
Driver: Hive JDBC (version 2.3.9)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 2.3.9 by Apache Hive
0: jdbc:hive2://xnode3.local:2182,xnode4.loca> use ndh_test_db;
INFO  : Compiling command(queryId=hive_20221025222450_9e73cb27-151a-4ebc-b206-c95d76836ff2);USER_NAME:ndh_test_user;IP:172.30.2.213;QUERY_STRING: use ndh_test_db
INFO  : Semantic Analysis Completed
INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20221025222450_9e73cb27-151a-4ebc-b206-c95d76836ff2);USER_NAME:ndh_test_user;IP:172.30.2.213;Time taken: 0.072 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing command(queryId=hive_20221025222450_9e73cb27-151a-4ebc-b206-c95d76836ff2); USER_NAME=ndh_test_user;IP=172.30.2.213;QUERY_STRING:use ndh_test_db
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20221025222450_9e73cb27-151a-4ebc-b206-c95d76836ff2);USER_NAME=ndh_test_user;IP=172.30.2.213;Time taken: 0.04 seconds
INFO  : OK
No rows affected (0.197 seconds)
0: jdbc:hive2://xnode3.local:2182,xnode4.loca>
0: jdbc:hive2://xnode3.local:2182,xnode4.loca> create table hive_tbl_01(id int,name string,address string);
INFO  : Compiling command(queryId=hive_20221025222450_860859f9-02ae-44b3-86bb-3fd958789b09);USER_NAME:ndh_test_user;IP:172.30.2.213;QUERY_STRING: create table hive_tbl_01(id int,name string,address string)
INFO  : Semantic Analysis Completed
INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20221025222450_860859f9-02ae-44b3-86bb-3fd958789b09);USER_NAME:ndh_test_user;IP:172.30.2.213;Time taken: 0.164 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing command(queryId=hive_20221025222450_860859f9-02ae-44b3-86bb-3fd958789b09); USER_NAME=ndh_test_user;IP=172.30.2.213;QUERY_STRING:create table hive_tbl_01(id int,name string,address string)
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20221025222450_860859f9-02ae-44b3-86bb-3fd958789b09);USER_NAME=ndh_test_user;IP=172.30.2.213;Time taken: 0.084 seconds
INFO  : OK
No rows affected (0.261 seconds)
0: jdbc:hive2://xnode3.local:2182,xnode4.loca>
0: jdbc:hive2://xnode3.local:2182,xnode4.loca> insert into table hive_tbl_01 select 1,"joey","zj";
INFO  : Compiling command(queryId=hive_20221025222451_511cb5fa-4405-4cf7-957f-15ec1eedca51);USER_NAME:ndh_test_user;IP:172.30.2.213;QUERY_STRING: insert into table hive_tbl_01 select 1,"joey","zj"
INFO  : Semantic Analysis Completed
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, type:int, comment:null), FieldSchema(name:_c1, type:string, comment:null), FieldSchema(name:_c2, type:string, comment:null)], properties:null)
INFO  : Completed compiling command(queryId=hive_20221025222451_511cb5fa-4405-4cf7-957f-15ec1eedca51);USER_NAME:ndh_test_user;IP:172.30.2.213;Time taken: 0.482 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing command(queryId=hive_20221025222451_511cb5fa-4405-4cf7-957f-15ec1eedca51); USER_NAME=ndh_test_user;IP=172.30.2.213;QUERY_STRING:insert into table hive_tbl_01 select 1,"joey","zj"
WARN  : Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
INFO  : Query ID = hive_20221025222451_511cb5fa-4405-4cf7-957f-15ec1eedca51
INFO  : Total jobs = 3
INFO  : Launching Job 1 out of 3
INFO  : Starting task [Stage-1:MAPRED] in serial mode
INFO  : Number of reduce tasks is set to 0 since there's no reduce operator
INFO  : number of splits:1
INFO  : Submitting tokens for job: job_1666075306696_0016
INFO  : Executing with tokens: [Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:easyops-cluster, Ident: (token for ndh_test_user: HDFS_DELEGATION_TOKEN owner=ndh_test_user, renewer=yarn, realUser=hive/xnode3.local@BDMS.COM, issueDate=1666707891779, maxDate=1667312691779, sequenceNumber=28, masterKeyId=9), Kind: HIVE_DELEGATION_TOKEN, Service: HiveServer2ImpersonationToken, Ident: 00 0d 6e 64 68 5f 74 65 73 74 5f 75 73 65 72 0d 6e 64 68 5f 74 65 73 74 5f 75 73 65 72 1a 68 69 76 65 2f 78 6e 6f 64 65 33 2e 6c 6f 63 61 6c 40 42 44 4d 53 2e 43 4f 4d 8a 01 84 0f 88 16 f4 8a 01 84 1e fb 2a f4 8e 1e e1 15]
INFO  : The url to track the job: http://xnode4.local:8088/proxy/application_1666075306696_0016/
INFO  : Starting Job = job_1666075306696_0016, Tracking URL = http://xnode4.local:8088/proxy/application_1666075306696_0016/
INFO  : Kill Command = /home/bdms/yarn/default_yarn/client/20221018113442217309170ee/current/bin/hadoop job  -kill job_1666075306696_0016
INFO  : Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
INFO  : 2022-10-25 22:25:06,970 Stage-1 map = 0%,  reduce = 0%
INFO  : 2022-10-25 22:25:18,630 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.61 sec
INFO  : MapReduce Total cumulative CPU time: 2 seconds 610 msec
INFO  : Ended Job = job_1666075306696_0016
INFO  : Starting task [Stage-7:CONDITIONAL] in serial mode
INFO  : Stage-4 is selected by condition resolver.
INFO  : Stage-3 is filtered out by condition resolver.
INFO  : Stage-5 is filtered out by condition resolver.
INFO  : Starting task [Stage-4:MOVE] in serial mode
INFO  : Moving data to directory hdfs://easyops-cluster/user/warehouse/ndh_test_db.db/hive_tbl_01/.hive-staging_hive_2022-10-25_22-24-51_141_7072788565978475619-1/-ext-10000 from hdfs://easyops-cluster/user/warehouse/ndh_test_db.db/hive_tbl_01/.hive-staging_hive_2022-10-25_22-24-51_141_7072788565978475619-1/-ext-10002
INFO  : Starting task [Stage-0:MOVE] in serial mode
INFO  : Loading data to table ndh_test_db.hive_tbl_01 from hdfs://easyops-cluster/user/warehouse/ndh_test_db.db/hive_tbl_01/.hive-staging_hive_2022-10-25_22-24-51_141_7072788565978475619-1/-ext-10000
INFO  : Starting task [Stage-2:STATS] in serial mode
INFO  : Table ndh_test_db.hive_tbl_01 stats: [numFiles=1, numRows=1, totalSize=10, rawDataSize=9]
INFO  : MapReduce Jobs Launched:
INFO  : Stage-Stage-1: Map: 1   Cumulative CPU: 2.61 sec   HDFS Read: 5291 HDFS Write: 89 SUCCESS
INFO  : Total MapReduce CPU Time Spent: 2 seconds 610 msec
INFO  : Completed executing command(queryId=hive_20221025222451_511cb5fa-4405-4cf7-957f-15ec1eedca51);USER_NAME=ndh_test_user;IP=172.30.2.213;Time taken: 28.657 seconds
INFO  : OK
No rows affected (29.15 seconds)
0: jdbc:hive2://xnode3.local:2182,xnode4.loca>
0: jdbc:hive2://xnode3.local:2182,xnode4.loca> select * from hive_tbl_01;
INFO  : Compiling command(queryId=hive_20221025222520_c08b09a4-3c9c-4373-9a9d-5dcd80316a05);USER_NAME:ndh_test_user;IP:172.30.2.213;QUERY_STRING: select * from hive_tbl_01
INFO  : Semantic Analysis Completed
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:hive_tbl_01.id, type:int, comment:null), FieldSchema(name:hive_tbl_01.name, type:string, comment:null), FieldSchema(name:hive_tbl_01.address, type:string, comment:null)], properties:null)
INFO  : Completed compiling command(queryId=hive_20221025222520_c08b09a4-3c9c-4373-9a9d-5dcd80316a05);USER_NAME:ndh_test_user;IP:172.30.2.213;Time taken: 0.348 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing command(queryId=hive_20221025222520_c08b09a4-3c9c-4373-9a9d-5dcd80316a05); USER_NAME=ndh_test_user;IP=172.30.2.213;QUERY_STRING:select * from hive_tbl_01
INFO  : Completed executing command(queryId=hive_20221025222520_c08b09a4-3c9c-4373-9a9d-5dcd80316a05);USER_NAME=ndh_test_user;IP=172.30.2.213;Time taken: 0.0 seconds
INFO  : OK
+-----------------+-------------------+----------------------+
| hive_tbl_01.id  | hive_tbl_01.name  | hive_tbl_01.address  |
+-----------------+-------------------+----------------------+
| 1               | joey              | zj                   |
+-----------------+-------------------+----------------------+
1 row selected (0.479 seconds)
0: jdbc:hive2://xnode3.local:2182,xnode4.loca>
0: jdbc:hive2://xnode3.local:2182,xnode4.loca> select count(1) from hive_tbl_01;
INFO  : Compiling command(queryId=hive_20221025222520_23c739ab-f970-4371-bf43-3c75bc6d4c25);USER_NAME:ndh_test_user;IP:172.30.2.213;QUERY_STRING: select count(1) from hive_tbl_01
INFO  : Semantic Analysis Completed
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, type:bigint, comment:null)], properties:null)
INFO  : Completed compiling command(queryId=hive_20221025222520_23c739ab-f970-4371-bf43-3c75bc6d4c25);USER_NAME:ndh_test_user;IP:172.30.2.213;Time taken: 0.285 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing command(queryId=hive_20221025222520_23c739ab-f970-4371-bf43-3c75bc6d4c25); USER_NAME=ndh_test_user;IP=172.30.2.213;QUERY_STRING:select count(1) from hive_tbl_01
INFO  : Completed executing command(queryId=hive_20221025222520_23c739ab-f970-4371-bf43-3c75bc6d4c25);USER_NAME=ndh_test_user;IP=172.30.2.213;Time taken: 0.0 seconds
INFO  : OK
+------+
| _c0  |
+------+
| 1    |
+------+
1 row selected (0.301 seconds)
0: jdbc:hive2://xnode3.local:2182,xnode4.loca>
0: jdbc:hive2://xnode3.local:2182,xnode4.loca> drop table hive_tbl_01;
INFO  : Compiling command(queryId=hive_20221025222521_d851f513-dc69-47b5-b3a2-1c247b6c68f3);USER_NAME:ndh_test_user;IP:172.30.2.213;QUERY_STRING: drop table hive_tbl_01
INFO  : Semantic Analysis Completed
INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20221025222521_d851f513-dc69-47b5-b3a2-1c247b6c68f3);USER_NAME:ndh_test_user;IP:172.30.2.213;Time taken: 0.059 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing command(queryId=hive_20221025222521_d851f513-dc69-47b5-b3a2-1c247b6c68f3); USER_NAME=ndh_test_user;IP=172.30.2.213;QUERY_STRING:drop table hive_tbl_01
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20221025222521_d851f513-dc69-47b5-b3a2-1c247b6c68f3);USER_NAME=ndh_test_user;IP=172.30.2.213;Time taken: 0.141 seconds
INFO  : OK
No rows affected (0.21 seconds)
0: jdbc:hive2://xnode3.local:2182,xnode4.loca>

2.3.3 预期结果

1. SQL均执行成功；

2.4 Spark 服务功能测试

2.4.1 前置操作

1. 登录Spark Client部署服务器

2. 执行kinit认证操作：通过ndh_test_user keytab认证

# 执行klist查看Principal
klist -kt /etc/security/keytabs/ndh_test_user.keytab

# 获取Principal后执行kinit认证
kinit -kt /etc/security/keytabs/ndh_test_user.keytab ndh_test_user@BDMS.COM

如下图所示 3. 查看Spark Client部署目录

查看EasyOps服务部署根目录（BASE_DIR），如/home/bdms

进入BASE_DIR,并逐级找到Spark Client部署目录（其中202210181134422225d2ebefe为组件实例名，不同机器上不同），如下所示的/home/bdms/spark2/default_spark2/client/202210181134422225d2ebefe/

# 替换/home/bdms为上述获取的EasyOps服务部署的根目录BASE_DIR
[root@ndh10 bdms]# cd /home/bdms/
[root@ndh10 bdms]# ls
hadoop  hdfs  impala  java  knox  kyuubi  logs  monitor  nginx_ha  spark2  yarn  zookeeper
[root@ndh10 bdms]# cd spark2
[root@ndh10 spark2]# ls
default_spark2  package_shared
[root@ndh10 spark2]# cd ndh_spark2/
[root@ndh10 ndh_spark2]# ls
client  jobhistoryserver
[root@ndh10 ndh_spark2]# cd client/
[root@ndh10 client]# ls
202210181134422225d2ebefe
# 此处若组件安装多次，则选择目录创建时间最近的
[root@ndh10 client]# cd 202210181134422225d2ebefe/
[root@ndh10 202210181134422225d2ebefe]# ls
config  current  data  keytab  logs  package  pid  ranger-0.5.4-1.1.3.5-spark-plugin
[root@ndh10 202210181134422225d2ebefe]# pwd
/home/bdms/spark2/default_spark2/client/202210181134422225d2ebefe

2.4.2 验证步骤

1. 验证Spark client模式JAR任务运行，需通过--指定队列

# 指定SPARK_EXAMPLE_JAR_DIR为上述查看的Spark Client部署目录
SPARK_EXAMPLE_JAR_DIR=/home/bdms/spark2/default_spark2/client/202210181134422225d2ebefe
spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode client --driver-memory 1g --executor-memory 1g --executor-cores 1 --queue ndh_test ${SPARK_EXAMPLE_JAR_DIR}/current/examples/jars/spark-examples_*.jar 100

2. 验证Spark cluster模式JAR任务运行，需通过--指定队列

# 指定SPARK_EXAMPLE_JAR_DIR为上述查看的Spark Client部署目录
SPARK_EXAMPLE_JAR_DIR=/home/bdms/spark2/default_spark2/client/202210181134422225d2ebefe
spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster --driver-memory 1g --executor-memory 1g --executor-cores 1 --queue ndh_test ${SPARK_EXAMPLE_JAR_DIR}/current/examples/jars/spark-examples_*.jar 100

3. 验证spark-sql 任务执行，需要通过--指定队列

# 执行spark-sql 
spark-sql --queue ndh_test

# 执行以下SQL，ndh_test_db 前置操作已经创建
use ndh_test_db;
create table spark_tbl_01(id int,name string,address string);
insert into table spark_tbl_01 select 1,"joey","zj";
select * from spark_tbl_01;
select count(1) from spark_tbl_01;
drop table spark_tbl_01;

执行结果如下所示

spark-sql> use ndh_test_db;
Time taken: 0.086 seconds
spark-sql> create table spark_tbl_01(id int,name string,address string);
Time taken: 0.684 seconds
spark-sql> insert into table spark_tbl_01 select 1,"joey","zj";
2022-10-25T22:01:53.893+0800: [GC (Metadata GC Threshold)
Desired survivor size 53477376 bytes, new threshold 3 (max 15)
[PSYoungGen: 431034K->43212K(1073152K)] 465081K->82547K(1416192K), 0.0683764 secs] [Times: user=0.11 sys=0.09, real=0.06 secs]
2022-10-25T22:01:53.962+0800: [Full GC (Metadata GC Threshold) [PSYoungGen: 43212K->0K(1073152K)] [ParOldGen: 39334K->56988K(222720K)] 82547K->56988K(1295872K), [Metaspace: 122808K->122798K(1163264K)], 0.1926002 secs] [Times: user=0.67 sys=0.05, real=0.20 secs]
Time taken: 8.103 seconds
spark-sql> select * from spark_tbl_01;
1        joey        zj
Time taken: 1.181 seconds, Fetched 1 row(s)
spark-sql> select count(1) from spark_tbl_01;
1
Time taken: 0.822 seconds, Fetched 1 row(s)
spark-sql> drop table spark_tbl_01;
Time taken: 0.627 seconds

2.4.3 预期结果

1. 任务均正常执行成功。

2. 进入Spark HistoryServer 可正常查看已运行完成的任务

2.5 Kyuubi 服务功能测试

2.5.1 前置操作

1. 登录Spark Client部署服务器 2. 执行kinit认证操作：通过ndh_test_user keytab认证

# 执行klist查看Principal
klist -kt /etc/security/keytabs/ndh_test_user.keytab

# 获取Principal后执行kinit认证
kinit -kt /etc/security/keytabs/ndh_test_user.keytab ndh_test_user@BDMS.COM

如下图所示

查看Spark Client部署目录获取Kyuubi JDBC连接串通过EasyOps查看Kyuubi服务连接串，如下所示： kyuubi_jdbc_url："jdbc:hive2://xnode3.local:2182,xnode4.local:2182,xnode2.local:2182/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi-cluster;"

2.5.2 验证步骤

1. Kyuubi基本功能：

通过beeline连接Kyuubi，并执行以下SQL

# 通过beeline连接
beeline -u "jdbc:hive2://xnode3.local:2182,xnode4.local:2182,xnode2.local:2182/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi-cluster;principal=hive/_HOST@CDH.HUATAI.COM;#spark.yarn.queue=root.ndh_test"

# 连接成功后，分别执行以下SQL ndh_test_db 前置操作已经创建
use ndh_test_db;
create table kyuubi_tbl_01(id int,name string,address string);
insert into table kyuubi_tbl_01 select 1,"joey","zj";
select * from kyuubi_tbl_01;
select count(1) from kyuubi_tbl_01;
drop table kyuubi_tbl_01;

执行记录如下（已过滤部分日志）：

[root@xnode3 ~]# beeline -u "jdbc:hive2://xnode3.local:2182,xnode4.local:2182,xnode2.local:2182/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi-cluster;principal=hive/_HOST@CDH.HUATAI.COM;#spark.yarn.queue=root.ndh_test"
Connecting to jdbc:hive2://xnode3.local:2182,xnode4.local:2182,xnode2.local:2182/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi-cluster;principal=hive/_HOST@CDH.HUATAI.COM;#spark.yarn.queue=root.ndh_test
log4j:WARN No appenders could be found for logger (org.apache.hive.jdbc.Utils).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Connected to: Apache Kyuubi (Incubating) (version 1.4.1-incubating)
Driver: Hive JDBC (version 2.3.9)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 2.3.9 by Apache Hive
0: jdbc:hive2://xnode3.local:2182,xnode4.loca> use ndh_test_db;
2022-10-25 22:14:47.713 INFO credentials.HadoopFsDelegationTokenProvider: getting token owned by ndh_test_user for: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_465453460_206, ugi=ndh_test_user (auth:PROXY) via hive/xnode3.local@BDMS.COM (auth:KERBEROS)]]
+---------+
| Result  |
+---------+
+---------+
No rows selected (0.456 seconds)
0: jdbc:hive2://xnode3.local:2182,xnode4.loca> create table kyuubi_tbl_01(id int,name string,address string);
2022-10-25 22:14:54.578 INFO credentials.HadoopCredentialsManager: Send new credentials with epoch 0 to SQL engine through session 53ab3c3f-a179-43a5-b1a4-5fa9f70d5fb6
2022-10-25 22:14:54.625 INFO credentials.HadoopCredentialsManager: Update session credentials epoch from -1 to 0
+---------+
| Result  |
+---------+
+---------+
No rows selected (0.824 seconds)
0: jdbc:hive2://xnode3.local:2182,xnode4.loca>
0: jdbc:hive2://xnode3.local:2182,xnode4.loca> insert into table kyuubi_tbl_01 select 1,"joey","zj";
2022-10-25 22:14:54.653 INFO operation.ExecuteStatement: Processing ndh_test_user's query[55ed97c3-6e2d-4895-9a94-6d7e811a3140]: INITIALIZED_STATE -> PENDING_STATE, statement: insert into table kyuubi_tbl_01 select 1,"joey","zj"
2022-10-25 22:14:54.658 INFO operation.ExecuteStatement: Processing ndh_test_user's query[55ed97c3-6e2d-4895-9a94-6d7e811a3140]: PENDING_STATE -> RUNNING_STATE, statement: insert into table kyuubi_tbl_01 select 1,"joey","zj"
2022-10-25 22:14:59.388 INFO operation.ExecuteStatement: Processing ndh_test_user's query[55ed97c3-6e2d-4895-9a94-6d7e811a3140]: RUNNING_STATE -> FINISHED_STATE, statement: insert into table kyuubi_tbl_01 select 1,"joey","zj", time taken: 4.731 seconds
+---------+
| Result  |
+---------+
+---------+
No rows selected (4.761 seconds)
0: jdbc:hive2://xnode3.local:2182,xnode4.loca>
0: jdbc:hive2://xnode3.local:2182,xnode4.loca> select * from kyuubi_tbl_01;
2022-10-25 22:15:00.655 INFO operation.ExecuteStatement: Processing ndh_test_user's query[da7e9025-3be4-458e-829b-01d796493248]: RUNNING_STATE -> FINISHED_STATE, statement: select * from kyuubi_tbl_01, time taken: 1.224 seconds
+-----+-------+----------+
| id  | name  | address  |
+-----+-------+----------+
| 1   | joey  | zj       |
+-----+-------+----------+
1 row selected (1.256 seconds)
0: jdbc:hive2://xnode3.local:2182,xnode4.loca>
0: jdbc:hive2://xnode3.local:2182,xnode4.loca> select count(1) from kyuubi_tbl_01;
2022-09-23 09:50:35.358 INFO operation.ExecuteStatement: Processing hive's query[b68579be-bfd0-44ab-9ebf-520199ef5876]: RUNNING_STATE -> FINISHED_STATE, statement: select count(1) from kyuubi_tbl_01, time taken: 0.719 seconds
2022-10-25 22:15:01.640 INFO operation.ExecuteStatement: Query[effb801c-53ad-4c7b-adf1-0fe4f934756b] in FINISHED_STATE
2022-10-25 22:15:01.641 INFO operation.ExecuteStatement: Processing ndh_test_user's query[effb801c-53ad-4c7b-adf1-0fe4f934756b]: RUNNING_STATE -> FINISHED_STATE, statement: select count(1) from kyuubi_tbl_01, time taken: 0.939 seconds
+-----------+
| count(1)  |
+-----------+
| 1         |
+-----------+
1 row selected (0.974 seconds)
0: jdbc:hive2://xnode3.local:2182,xnode4.loca>
0: jdbc:hive2://xnode3.local:2182,xnode4.loca> drop table kyuubi_tbl_01;
+---------+
| Result  |
+---------+
+---------+
No rows selected (0.244 seconds)
0: jdbc:hive2://ndh10.huatai.com:2182,ndh11.h>

2.5.3 预期结果

1. SQL均执行成功；

2.6 Impala 服务功能测试

2.6.1 前置操作

1. 登录Hive Client 部署服务器

2. 执行kinit认证操作：通过ndh_test_user keytab认证

# 执行klist查看Principal
klist -kt /etc/security/keytabs/ndh_test_user.keytab

# 获取Principal后执行kinit认证
kinit -kt /etc/security/keytabs/ndh_test_user.keytab ndh_test_user@BDMS.COM

如下图所示 3. 通过EasyOps查看Impala Beeline服务连接串，如下所示： impala_link:"jdbc:hive2://xnode3.local:2182,xnode4.local:2182,xnode2.local:2182/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=impala-ha/hiveserver2;principal=impala/_HOST@BDMS.COM"

2.6.2 验证步骤

1. Impala基本功能验证

通过beeline连接Impalad，并执行以下SQL
# 通过beeline连接
beeline -u "jdbc:hive2://xnode3.local:2182,xnode4.local:2182,xnode2.local:2182/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=impala-ha/hiveserver2;principal=impala/_HOST@BDMS.COM"

# 连接成功后，分别执行以下SQL
use ndh_test_db;  
create table impala_tbl_01(id int,name string,address string);
insert into table impala_tbl_01 select 1,"joey","zj";
select * from impala_tbl_01;
select count(1) from impala_tbl_01;
drop table impala_tbl_01;

2.7 HBase 服务功能测试

2.7.1 前置操作

1. 登录HBase Client部署服务器

2. 执行kinit认证操作

# 执行klist查看Principal
klist -kt /etc/security/keytabs/hbase/hbase.keytab

# 获取Principal后执行kinit认证
kinit -kt /etc/security/keytabs/hbase/hbase.keytab hbase/xnode2.local@BDMS.COM

3. 查看HBase Client部署目录

查看EasyOps服务部署根目录（BASE_DIR），如/home/bdms 进入BASE_DIR,并逐级找到HBase Client部署目录（其中/20221018113442217309170ee为组件实例名，不同机器上不同），如下所示的/home/bdms/hbase/default_hbase/client/20221018113442223fde7b95f

# 替换/home/bdms为上述获取的EasyOps服务部署的根目录BASE_DIR
[root@xnode2 ~]# cd /home/bdms/
[root@xnode2 bdms]# ls
elasticsearch  hbase  hdfs  hive  java  kafka  kerberos  ldap  logs  monitor  ntesmysqlpaas  spark2  yarn  zookeeper
[root@xnode2 bdms]# cd hbase/
[root@xnode2 hbase]# ls
default_hbase  package_shared
[root@xnode2 hbase]# cd default_hbase/
[root@xnode2 default_hbase]# ls
client  regionserver
[root@xnode2 default_hbase]# cd client/
[root@xnode2 client]# ls
20221018113442223fde7b95f
[root@xnode2 client]# cd 20221018113442223fde7b95f/
[root@xnode2 20221018113442223fde7b95f]# ls
config  current  data  keytab  logs  package  pid
[root@xnode2 20221018113442223fde7b95f]# pwd
/home/bdms/hbase/default_hbase/client/20221018113442223fde7b95f

2.7.2 验证步骤

1. 通过hbase shell连接HBase服务

HBASE_HOME=/home/bdms/hbase/default_hbase/client/20221018113442223fde7b95f
export HBASE_CONF_DIR=${HBASE_HOME}/config
${HBASE_HOME}/current/bin/hbase shell

2. 验证HBase基本功能

# 1.创建namespaces:
create_namespace 'test_ndh_hbase'

# 2. 创建表：
create 'test_ndh_hbase:student','id','address','info'

# 3. 插入数据：
put 'test_ndh_hbase:student', 'Elvis', 'id', '22'
put 'test_ndh_hbase:student', 'Elvis','info:age', '26'
put 'test_ndh_hbase:student', 'Elvis','info:birthday', '1988-09-14 '
put 'test_ndh_hbase:student', 'Elvis','info:industry', 'it'
put 'test_ndh_hbase:student', 'Elvis','address:city', 'beijing'
put 'test_ndh_hbase:student', 'Elvis','address:country', 'china'

# 4. 查询表
scan 'test_ndh_hbase:student'

# 5. 删除表
disable 'test_ndh_hbase:student'
drop 'test_ndh_hbase:student'
exists 'test_ndh_hbase:student'

# 6. 删除namespaces
drop_namespace  'test_ndh_hbase'

执行记录如下：

[root@xnode2 bin]# HBASE_HOME=/home/bdms/hbase/default_hbase/client/20221018113442223fde7b95f
[root@xnode2 bin]# export HBASE_CONF_DIR=${HBASE_HOME}/config
[root@xnode2 bin]# ${HBASE_HOME}/current/bin/hbase shell
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/bdms/hdfs/package_shared/hadoop-3.3.0-1.3.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/bdms/hbase/package_shared/hbase-2.2.6/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
HBase Shell
Use "help" to get list of supported commands.
Use "exit" to quit this interactive shell.
For Reference, please visit: http://hbase.apache.org/2.0/book.html#shell
Version 2.2.6, r88c9a386176e2c2b5fd9915d0e9d3ce17d0e456e, Tue Sep 15 17:36:14 CST 2020
Took 0.0053 seconds
hbase(main):001:0> create_namespace 'test_ndh_hbase'
Took 1.1733 seconds
hbase(main):002:0> create 'test_ndh_hbase:student','id','address','info'
Created table test_ndh_hbase:student
Took 1.3291 seconds
=> Hbase::Table - test_ndh_hbase:student
hbase(main):003:0>
hbase(main):004:0*
hbase(main):005:0* put 'test_ndh_hbase:student', 'Elvis', 'id', '22'
put 'test_ndh_hbase:student', 'Elvis','info:age', '26'
put 'test_ndh_hbase:student', 'Elvis','info:birthday', '1988-09-14 '
put 'test_ndh_hbase:student', 'Elvis','info:industry', 'it'
put 'test_ndh_hbase:student', 'Elvis','address:city', 'beijing'
Took 0.2563 seconds
put 'test_ndh_hbase:student', 'Elvis','address:country', 'china'hbase(main):006:0> put 'test_ndh_hbase:student', 'Elvis','info:age', '26'
Took 0.0084 seconds
hbase(main):007:0> put 'test_ndh_hbase:student', 'Elvis','info:birthday', '1988-09-14 '
Took 0.0088 seconds
hbase(main):008:0> put 'test_ndh_hbase:student', 'Elvis','info:industry', 'it'
Took 0.0064 seconds
hbase(main):009:0> put 'test_ndh_hbase:student', 'Elvis','address:city', 'beijing'
Took 0.0081 seconds
hbase(main):010:0> put 'test_ndh_hbase:student', 'Elvis','address:country', 'china'
Took 0.0087 seconds
hbase(main):011:0>
hbase(main):012:0* scan 'test_ndh_hbase:student'
ROW                                      COLUMN+CELL
 Elvis                                   column=address:city, timestamp=1666719524530, value=beijing
 Elvis                                   column=address:country, timestamp=1666719525159, value=china
 Elvis                                   column=id:, timestamp=1666719524389, value=22
 Elvis                                   column=info:age, timestamp=1666719524430, value=26
 Elvis                                   column=info:birthday, timestamp=1666719524465, value=1988-09-14
 Elvis                                   column=info:industry, timestamp=1666719524502, value=it
1 row(s)
Took 0.0411 seconds
hbase(main):013:0> disable 'test_ndh_hbase:student'
drop 'test_ndh_hbase:student'
exists 'test_ndh_hbase:student'
Took 0.8262 seconds
hbase(main):014:0> drop 'test_ndh_hbase:student'
Took 0.4985 seconds
hbase(main):015:0> exists 'test_ndh_hbase:student'
Table test_ndh_hbase:student does not exist
Took 0.0120 seconds
=> false
hbase(main):016:0>
hbase(main):017:0* drop_namespace  'test_ndh_hbase'
Took 0.1972 seconds
hbase(main):018:0>

2.8 Kafka 服务功能测试

2.8.1 前置操作

1. 通过EasyOps查看Kafka Manager获取配置信息

Zookeepers xnode3.local:2182 xnode4.local:2182 xnode2.local:2182/kafka
Brokers xnode2.local:9092,xnode3.local:9092,xnode4.local:9092

2. 登录Kafka broker部署服务器,执行kinit认证操作

# 执行klist查看Principal
klist -kt /etc/security/keytabs/kafka/kafka.service.keytab

# 获取Principal后执行kinit认证
kinit -kt /etc/security/keytabs/kafka/kafka.service.keytab kafka/xnode3.local@BDMS.COM

3. 查看Kafka broker部署目录查看EasyOps服务部署根目录（BASE_DIR），如/home/bdms 进入BASE_DIR,并逐级找到Kafka broker部署目录（其中202210181134422111b6b1df3为组件实例名，不同机器上不同），如下所示的/home/bdms/kafka/default_kafka/broker/202210181134422111b6b1df3

# 替换/home/bdms为上述获取的EasyOps服务部署的根目录BASE_DIR
[root@xnode3 ~]# cd /home/bdms/
[root@xnode3 bdms]# ls
easy_ranger        elasticsearch  hbase  hive  kafka     knox    ldap  monitor        spark2  zookeeper
easy_ranger_admin  hadoop         hdfs   java  kerberos  kyuubi  logs  ntesmysqlpaas  yarn
[root@xnode3 bdms]# cd kafka/
[root@xnode3 kafka]# ls
default_kafka
[root@xnode3 kafka]# cd default_kafka/
[root@xnode3 default_kafka]# ls
broker
[root@xnode3 default_kafka]# cd broker/
[root@xnode3 broker]# ls
202210181134422111b6b1df3
[root@xnode3 broker]# cd 202210181134422111b6b1df3/
[root@xnode3 202210181134422111b6b1df3]# ls
config  current  data  keytab  logs  monitor  package  pid
[root@xnode3 202210181134422111b6b1df3]# pwd
/home/bdms/kafka/default_kafka/broker/202210181134422111b6b1df3
[root@xnode3 202210181134422111b6b1df3]#

2.8.2 验证步骤

1. 创建topic，生产数据

KAFKA_HOME=/home/bdms/kafka/default_kafka/broker/202210181134422111b6b1df3
# 创建topic，此处的zookeeper列表为前置步骤1中获取，通过','拼接
${KAFKA_HOME}/current/bin/kafka-topics.sh --create --zookeeper xnode3.local:2182,xnode4.local:2182,xnode2.local:2182/kafka --replication-factor 1 --partitions 1 --topic test_kafka_topic

# 生产数据，此处的broker list为前置步骤1中获取，通过','拼接
${KAFKA_HOME}/current/bin/kafka-console-producer.sh --broker-list xnode2.local:9092,xnode3.local:9092,xnode4.local:9092 --topic test_kafka_topic
# 任意输入数据

2. 登录其他broker节点，消费数据

# 获取该节点kafka broker部署目录
KAFKA_HOME=/home/bdms/kafka/default_kafka/broker/202210181134422111b6b1df3
# 消费数据，此处的bootstrap-server为前置步骤1中获取，通过','拼接
bin/kafka-console-consumer.sh --bootstrap-server xnode2.local:9092,xnode3.local:9092,xnode4.local:9092 --topic test_kafka_topic --from-beginning