CDC 任务数据源配置说明

Kafka 数据源

kafka 数据源作为cdc 目标表时

kafka 对应的表序列化方式只支持选择debezium-json、canal-json、 maxwell-json这3种之一。 这3种不同的序列化方式输出的数据结构各不相同。具体如下,请根据数据使用情况进行选择。

  • debezium-json
    {
    "before": {
      "id": 111,
      "name": "scooter",
      "description": "Big 2-wheel scooter",
      "weight": 5.18
    },
    "after": {
      "id": 111,
      "name": "scooter",
      "description": "Big 2-wheel scooter",
      "weight": 5.15
    },
    "source": {...},
    "op": "u",
    "ts_ms": 1589362330904,
    "transaction": null
    }
  • canal-json

    {
    "data": [
      {
        "id": "111",
        "name": "scooter",
        "description": "Big 2-wheel scooter",
        "weight": "5.18"
      }
    ],
    "database": "inventory",
    "es": 1589373560000,
    "id": 9,
    "isDdl": false,
    "mysqlType": {
      "id": "INTEGER",
      "name": "VARCHAR(255)",
      "description": "VARCHAR(512)",
      "weight": "FLOAT"
    },
    "old": [
      {
        "weight": "5.15"
      }
    ],
    "pkNames": [
      "id"
    ],
    "sql": "",
    "sqlType": {
      "id": 4,
      "name": 12,
      "description": 12,
      "weight": 7
    },
    "table": "products",
    "ts": 1589373560798,
    "type": "UPDATE"
    }
  • maxwell-json

{
   "database":"test",
   "table":"e",
   "type":"insert",
   "ts":1477053217,
   "xid":23396,
   "commit":true,
   "position":"master.000006:800911",
   "server_id":23042,
   "thread_id":108,
   "primary_key": [1, "2016-10-21 05:33:37.523000"],
   "primary_key_columns": ["id", "c"],
   "data":{
     "id":111,
     "name":"scooter",
     "description":"Big 2-wheel scooter",
     "weight":5.15
   },
   "old":{
     "weight":5.18,
   }
}

Mysql 数据源

Mysql 作为cdc 源表
  • mysql 表中存在timestamp 类型需要同步时 需要在自定义参数里 额外增加参数’server-time-zone’ = ‘Asia/Shanghai’