日志文件格式及内容说明
注：日志格式说明：
ZCBUS内部日志分为-1到5六个级别日志 ，日志输出详细能力-1—-> 5 ,数字越高显示内容越详细
-1 不显示任何日志
0 显示基础运行日志
1 显示组件运行SQL日志
2 显示基础函数调用日志
3 显示数据转换调用日志
4 显示系统详细参数日志
5 显示操作系统详细参数日志
发布
日志启动 启动提示信息：
当配置实时信息的时候，会产生如下INFLUX CONFIG配置信息
influx config:
INFLUX_BUCKET_NAME=influxdb
INFLUX_HOST=http://172.17.104.186:8086
INFLUX_ORG=influxdb
INFLUX_TOKEN=mVr3RJWNhaCu235sqOnOivo59xp3V0ASwqKTH38bMMWpfNSzbp0kN3E-rm4hyGu27uUY1qKn6_6-hpYVRmMALw==
config:
auto_add_table_real_rule=1
data_migrate_mode=0
db_charset=
db_name=databus_v4
db_type=mysql
full_publish_interval=0
full_speed_limit=0
full_sync_buffer=100
full_sync_threads=6
host=172.17.58.146
land_file_reserve_cont=0
land_to_file=0
log_buffer_len=
log_extract_interval=1000
message_encrypt=1
message_max_len=0
not_compose_trans=0
not_filter_dml_in_ddl=0
password=1qaz!QAZ
port=3306
read_log_from_remote=1
real_export_buffer=0
real_export_by_where=0
real_export_threads=1
real_send_cache_buffer=200
real_send_queues=0
single_message_max_len=5242880
slave_server_id=35059
statistics_into_influxdb=1
trans_expire_time=0
user=zbomc
zcbusmq_compress=1
zcbusmq_mode=0
message_max_len adjust to 5242880全量发布日志格式：log.zcbus_full.${nodeid} 日志内容：
全量发布topic信息：
[LV1] THD[0] send bsdata: topic[37578.test.test.s.1] tag[0-37578] len[390] rows[5]
[INF] 2021-09-09 15:21:12: THD[0] send data statistics: topic[37578.test.test.s.1] tag[0-37578] length[585]
#全量发布进度：
exp rows 5/5 100.00%
total use time 2 s, send use time 1 s
[INF] 2021-09-09 15:21:12: THD[0] send bsdata: topic[37578.test.test.s.1] tag[0-37578] len[38]
#全量单表发布信息：
[INF] 2021-09-09 15:21:13: export table TEST.TEST success.
Topic: 37578.test.test.s.1
Tag: 0-37578
Exp rows: 5
Data: 0.0006 M
Use time: 2.094 s
Speed: 0.0003 M/s
#全量多表发布信息：
[INF] table export statictis:
+-------+----------+-------+-----+--------------------+--------+----------+---------+-----------+
|id |table |status |rows |topic |tag |total_len |use time |speed |
+-------+----------+-------+-----+--------------------+--------+----------+---------+-----------+
|164208 |TEST.TEST |OK |5 |37578.test.test.s.1 |0-37578 |623 |2 |0.0003 M/s |
+-------+----------+-------+-----+--------------------+--------+----------+---------+-----------+
|164209 |TEST.AAAA |OK |5 |37578.test.aaaa.s.1 |0-37578 |623 |2 |0.0003 M/s |
+-------+----------+-------+-----+--------------------+--------+----------+---------+-----------+
[INF] export 1 tables, 0 errors, 0 skips增量发布日志格式：log.zcbus_real.${nodeid}
日志内容：
#增量发布字典信息：
[LV0] get dict of table TEST.TEST...
TEST.TEST 400612/0 flags 0x0
+---+----+-----+-----+-----+-----+-------+-------+---------+--------+
|no |cid |name |type |flag |prec |length |offset |null_pos |bit_pos |
+---+----+-----+-----+-----+-----+-------+-------+---------+--------+
|0 |0 |ID |1 |4 |0 |22 |-1 |0 |0 |
|1 |1 |NAME |6 |0 |0 |10 |-1 |0 |0 |
|2 |2 |COL |1 |0 |0 |22 |-1 |0 |0 |
+---+----+-----+-----+-----+-----+-------+-------+---------+--------+
#增量发布数据库日志时间点信息：
[INF] 2021-09-09 15:31:37: LOG TIME: 2021-09-09 15:31:11
#增量发布kafka和缓存信息：
[INF] 2021-09-09 15:31:37: flush data to kafka...
[LV1] 2021-09-09 15:31:37: flush all cache data...
[INF] 2021-09-09 15:31:37: flush data to kafka ok.
#增量发布dml和ddl统计：
[INF] UPDATE[35] ---> INSERT[1] UPDATE[1] DELETE[1] DDL[1]
Send data: 0.0003 M
Use time: 1 s
[INF] 2021-09-09 15:31:37: extract end pos 1909687426.推送服务日志格式：log.zcbus_pump.${nodeid}
日志内容：
订阅
订阅管理进程日志格式：log.client_${custoemrid}
日志内容：
[LV1] SQL-> update bus_op_customer_alive set cntid=2941 where customerid=37579 and module='zcbus-client'
[LV1] SQL-> SELECT id,loadThread,loadStatus,dbid,loadtype,name FROM bus_client_main_control WHERE customerid=? order by id
2021-09-09 15:34:24: exec -> ps -eaf|grep -v grep|grep " -customer_id 37579 -ctlid 995 "|awk -F" " '{print $2}' >/busdata/zl/zcbus/2.0.1/cache/bus17bc97aab1b0.tmp传输通道日志格式：log.client_${customerid}.${ctlid}
日志内容：
全量部分
#全量topic信息打印：
[LV1] Message: topic 37578.test.test.s.1 [0], offset 0, 38 bytes, timestamp: log append time 2021-09-09 15:21:11(1631172071874)
[LV0] FULL CONSUME THD: recv message 0.00 M, topic[37578.test.test.s.1] offset[0]
#全量表信息和延迟时间打印：
[LV0] LOAD THD[1]: 2021-09-09 15:23:29: table[TEST1.TEST] bsd data time[2021-09-09 15:21:12], delay 137 s.
#全量订阅dml和ddl统计：
[LV0] 2021-09-09 15:23:29: LOAD THD[1] load FULL message ok, length[390] time[0 s]
TABLE[TEST1.TEST] INSERT[5] DDL[1]
[INF] 2021-09-09 15:23:29: [FULL] Total 1 messages, 0 errors.
Data: 0.00 M
INSERT: 5
DDL: 1
Use time: 0 s
Speed: 0.00 M/s增量部分
#增量topic信息打印：
[LV0] REAL CONSUME THD: recv message 0.00 M, topic[37578.test.test.r] offset[4]
[INF] REAL CONSUME THD: total 1 messages, start loading...
[LV0] 2021-09-09 15:31:42: LOAD THD[1] loading REAL message 0, table[TEST1.TEST] topic[37578.test.test.r] offset[3]...
[LV1] SQL-> START TRANSACTION
#增量表信息和延迟时间打印：
[LV0] LOAD THD[1]: 2021-09-09 15:31:42: table[TEST1.TEST] bsd data time[2021-09-09 15:30:47], delay 55 s.
#增量订阅dml和ddl统计：
[LV0] 2021-09-09 15:31:43: LOAD THD[1] load REAL message ok, length[331] time[1 s]
TABLE[TEST1.TEST] INSERT[1] UPDATE[1] DELETE[1] DDL[1]
[LV0] 2021-09-09 15:31:43: get full table list...
[INF] 2021-09-09 15:33:15: [REAL] Total 1 messages, 0 errors.
Data: 0.00 M
INSERT: 1
UPDATE: 1
DELETE: 1
DDL: 1
Use time: 2 s
Speed: 0.00 M/sETL
ETL管理进程日志格式：log.etl_${custoemrid}
日志内容：
2021-09-10 13:16:49: exec -> ps -eaf|grep -v grep|grep " -customer_id 37358 -ctlid 855 "|awk -F" " '{print $2}' >/busdata/zl/zcbus/2.0.1/cache/bus17bce230e7f0.tmpETL传输通道日志格式：log.etl_${customerid}.${ctlid}
日志内容：
ETL规则部分
#ETL规则信息打印：
[INF] add etl obj[754], object_id[23864], content: PASS(SQLTYPE=(ON=INSERT,ON=UPDATE,ON=DELETE),CONDITION=(flag=1))
[LV0] check columns of etl obj[23864] tableid[123921]...
etl object count: 1
object_id[23864]:
etl id: 754
dbid: 0
cache: 0
queue name: test_etl_oracle
full topic: 37358.zfl.test_etl.s.2.test_etl_oracle
real topic: 37358.zfl.test_etl.r.test_etl_oracle.37357
context: PASS(SQLTYPE=(ON=INSERT,ON=UPDATE,ON=DELETE),CONDITION=(flag=1))
parse result:
PASS(SQLTYPE=(ON=INSERT,ON=UPDATE,ON=DELETE),CONDITION=(flag = 1))
-------------全量部分
#全量topic信息打印：
[INF] 2021-09-10 11:24:27: FULL CONSUME THD init topic list...
[INF] FULL CONSUME THD: total topic count: 1
[LV0] FULL CONSUME THD: recv message 0.00 M, topic[37357.zfl.test_etl.s.3] offset[0]
[LV0] FULL CONSUME THD: recv message 0.00 M, topic[37357.zfl.test_etl.s.3] offset[1]
[LV0] FULL CONSUME THD: recv message 0.00 M, topic[37357.zfl.test_etl.s.3] offset[2]
[LV0] FULL CONSUME THD: recv message 0.00 M, topic[37357.zfl.test_etl.s.3] offset[3]
[INF] FULL CONSUME THD: total 4 messages, start loading...
[LV0] LOAD THD[1] loading FULL message 0, table ZFL.TEST_ETL...
[LV0] LOAD THD[1] process etl rule[754]...
#全量表信息和延迟时间打印：
[LV0] LOAD THD[1]: 2021-09-10 11:24:28: bsd data time[2021-09-10 11:23:59], delay 29 s.
#全量订阅dml和ddl统计：
[LV0] LOAD THD[1] send to kafka, topic[37358.zfl.test_etl.s.3.test_etl_oracle], tag[0-37358], length[41]...
[LV0] LOAD THD[1] process etl rule[754] ok ----> INSERT[0] DDL[0]
[LV0] LOAD THD[1] load FULL message ok, length[41] time[0 s]
TABLE[ZFL.TEST_ETL] INSERT[0] DDL[0]
FULL CONSUME THD: topic[37357.zfl.test_etl.s.3] detect full sync end flag.
[INF] 2021-09-10 11:24:29: [FULL] Total 4 messages, 0 errors.
Data: 0.00 M
INSERT: 6
DDL: 1
Use time: 1 s
Speed: 0.00 M/s增量部分
#增量topic信息打印：
[LV0] 2021-09-10 13:55:04: get real table list...
[INF] 2021-09-10 13:55:04: REAL CONSUME THD init topic list...
[INF] REAL CONSUME THD: total topic count: 1
[LV0] REAL CONSUME THD: recv message 0.00 M, topic[37357.zfl.test_etl.r] offset[28]
[LV0] REAL CONSUME THD: filter control message.
#增量表信息和延迟时间打印：
[LV0] LOAD THD[1]: 2021-09-10 13:55:05: bsd data time[2021-09-10 13:55:01], delay 4 s.
#增量订阅dml和ddl统计：
[LV0] LOAD THD[1] send to kafka, topic[37358.zfl.test_etl.r.test_etl_oracle.37357], tag[0-37358], length[380]...
[LV0] LOAD THD[1] process etl rule[754] ok ----> INSERT[2] UPDATE[0] DELETE[2] DDL[0]
[LV0] LOAD THD[1] load REAL message ok, length[307] time[0 s]
TABLE[ZFL.TEST_ETL] INSERT[1] UPDATE[1] DELETE[1] DDL[0]
[INF] 2021-09-10 13:55:06: [REAL] Total 1 messages, 0 errors.
Data: 0.00 M
INSERT: 1
UPDATE: 1
DELETE: 1
DDL: 0
Use time: 1 s
Speed: 0.00 M/s比对
比对管理进程日志格式：log.compare_${customerid}
日志内容：
2021-09-09 12:10:05: exec -> ps -eaf|grep -v grep|grep "/busdata/zl/zcbus/2.0.1/bin/zcbus_service -compare -customer_id 37579 -child 0 -log_level 1"|awk -F" " '{print $2}' >/busdata/zl/zcbus/2.0.1/cache/bus17bc8bf99fb0.tmp比对进程日志格式：log.compare_${customerid}.${n}[0-3]（4个比对服务）
日志内容：
#统计源库（oracle为例）表记录：
[INF] begin compare count of table[TEST.TEST] ---- [TEST1.TEST]...
[INF] NLS_LANG not set, set NLS_LANG=AMERICAN_AMERICA.AL32UTF8
[LV0] 2021-09-09 16:08:10 connect by tns:172.17.58.145:1521/oracle12c
[LV1] SQL-> alter session set nls_date_format='yyyy-mm-dd hh24:mi:ss'
[LV1] SQL-> alter session set nls_timestamp_format='yyyy-mm-dd hh24:mi:ss.ff9'
[LV1] SQL-> alter session set nls_timestamp_tz_format='yyyy-mm-dd hh24:mi:ss.ff9 tzh:tzm'
[LV1] SQL-> select count(*) from dba_tables where owner='TEST' and table_name='TEST'
[LV1] SQL-> select count(*) from "TEST"."TEST"
[LV1] SQL-> SELECT nvl(SUM(BYTES),0) FROM DBA_SEGMENTS WHERE OWNER = 'TEST' AND SEGMENT_NAME = 'TEST'
[LV1] SQL-> SELECT nvl(SUM(BYTES),0) FROM DBA_SEGMENTS DS, DBA_LOBS DL WHERE DS.OWNER = DL.OWNER AND DS.SEGMENT_NAME = DL.SEGMENT_NAME AND DL.OWNER = 'TEST' AND DL.TABLE_NAME = 'TEST'
#统计目标库（mysql为例）表记录：
[INF] set client character set utf8mb4...
[INF] new client character set: utf8mb4
[INF] MYSQL VERSION: 50729
[LV1] SQL-> select version()
[INF] MYSQL INFO: 5.7.29-log
SET SESSION sql_mode='ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION'
[LV1] SQL-> SET SESSION sql_mode='ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION'
[LV1] SQL-> select count(*) from information_schema.tables where TABLE_SCHEMA='TEST1' and TABLE_NAME ='TEST' and TABLE_TYPE like '%TABLE'
[LV1] SQL-> select count(*) from `TEST1`.`TEST`
[LV1] SQL-> select sum(data_length) from information_schema.tables where TABLE_SCHEMA = 'TEST1' and table_name = 'TEST'
#比对结果：
[INF] table count is same, 5 rows.
#实时检索比对表状态：
[LV1] SQL-> SELECT id,customerid FROM bus_client_check_table_list WHERE customerid=37579 AND STATUS=1
[INF] 2021-09-09 16:08:30: find table for compare...检查
容器内对象检查服务日志格式：log.check
日志内容：
[INF] 2021-09-03 12:22:13: check client service...
[INF] 2021-09-03 12:22:13: check client service ok.容器服务
容器服务日志格式：log.service
日志内容：
#自动清理日志：
[LV2] 2021-09-09 16:12:58: check total size of log archive dir /busdata/zl/zcbus/2.0.1/log/archive...
[LV2] remove log file /busdata/zl/zcbus/2.0.1/log/archive/log.client_32070.436.20210901214302.
#自动检查进程：
[INF] 2021-09-09 16:12:58: get service list...
[LV0] check process of service[3840]...
2021-09-09 16:13:01: exec -> ps auxw|grep -v grep|grep "/busdata/zl/zcbus/2.0.1/bin/zcbus_service -client -customer_id 37579 " | awk -F" " '{print $2" "$3" "$4}' >/busdata/zl/zcbus/2.0.1/cache/bus17bc99e03160.tmp
[LV0] process exsit, pid[4912] CPU[0.90%], MEM[0.30%]
#获取OS信息：
2021-09-09 16:13:02: exec -> df -m /busdata/zl/zcbus/2.0.1 |grep -v Filesystem|awk -F" " '{print $2" "$3}' > /busdata/zl/zcbus/2.0.1/cache/bus17bc99e07210.tmp
2021-09-09 16:13:02: exec -> du -s -m /busdata/zl/zcbus/2.0.1 > /busdata/zl/zcbus/2.0.1/cache/bus17bc99e07210.tmp
2021-09-09 16:13:03: exec -> ps aux |grep ${ZCBUS_HOME}|grep -v grep |awk 'BEGIN{cpu=0.0;mem=0.0;cnt=0}{cpu=cpu+$3;mem=mem+$4;cnt++}END{printf("%d %d %d",cpu,mem,cnt);}' > /busdata/zl/zcbus/2.0.1/cache/bus17bc99e0bff0.tmp
OS: MEM size: 7821M, DISK size[201457 M]
OS: CPU used[10%], MEM used[39%], DISK used[156441 M]
ZCBUS: CPU used[38%], MEM used[14%], DISK used[30059 M], process_count[99]混合云数据上传下发服务
上传服务日志格式：log.aux_up
日志内容：
下发服务日志格式：log.aux_down
日志内容：
文档更新时间: 2023-09-17 17:14 作者：阿力
