canal(mysql数据库 binlog的增量订阅,消费组件)
2018-03-12 14:55
2651 查看
环境准备
1、jdk 环境变量配置
vim /etc/profile export JAVA_HOME=/root/data/jdk/jdk export PATH=$JAVA_HOME/bin:$PATH source /etc/profile
2、jdk配置校验
root@never-PAF4:~# date 2018年 03月 11日 星期日 00:00:30 CST root@never-PAF4:~# java -version java version "1.8.0_161" Java(TM) SE Runtime Environment (build 1.8.0_161-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode) root@never-PAF4:~#
3、canal
阿里巴巴mysql数据库 binlog的增量订阅,消费组件基于日志增量订阅和消费支持的业务
数据库镜像
数据库实时备份
多级索引
search build
业务cache刷新
价格变化等重要业务消息
定位:基于数据库增量日志解析,提供增量数据订阅 消费
工作原理
canal 模拟mysql slave的交互协议,伪装自己为myql slave ,向mysql master 发送 dump协议mysql master收到 dump请求,开始推送binary log 给slave (canal)
canal 解析 binary log 对象(原始为byte流)
下载
wget https://github.com/alibaba/canal/releases/download/canal-1.0.26-preview-1/canal.deployer-1.0.26-SNAPSHOT.tar.gz 解压:tar -xvf *.tar.gz
配置修改(使用默认配置需要对参数进行本地化,创建对应的测试库)
root@never-PAF4:~/data/canal/conf/example# pwd /root/data/canal/conf/example root@never-PAF4:~/data/canal/conf/example# vim instance.properties root@never-PAF4:~/data/canal/conf/example# cat instance.properties ################################################# ## mysql serverId canal.instance.mysql.slaveId=10 # position info canal.instance.master.address=127.0.0.1:3302 canal.instance.master.journal.name= canal.instance.master.position= canal.instance.master.timestamp= # table meta tsdb info canal.instance.tsdb.enable=true canal.instance.tsdb.dir=${canal.file.data.dir:../conf}/${canal.instance.destination:} canal.instance.tsdb.url=jdbc:h2:${canal.instance.tsdb.dir}/h2;CACHE_SIZE=1000;MODE=MYSQL; #canal.instance.tsdb.url=jdbc:mysql://127.0.0.1:3302/canal_tsdb canal.instance.tsdb.dbUsername=root canal.instance.tsdb.dbPassword=root #canal.instance.standby.address = #canal.instance.standby.journal.name = #canal.instance.standby.position = #canal.instance.standby.timestamp = # username/password canal.instance.dbUsername=root canal.instance.dbPassword=root canal.instance.defaultDatabaseName=test canal.instance.connectionCharset=UTF-8 # table regex canal.instance.filter.regex=.*\\..* # table black regex canal.instance.filter.black.regex= ################################################# root@never-PAF4:~/data/canal/conf/example#
docker 启动mysql(将本地的配置文件挂载到容器中..)
root@never-PAF4:~# docker images REPOSITORY TAG IMAGE ID CREATED SIZE mysql latest 5d4d51c57ea8 11 days ago 374MB root@never-PAF4:~# docker run -p 3306:3306 --name slave -e MYSQL_ROOT_PASSWORD=canal -d mysql 47292b980e11e2e4e6bfdee7722474303adc8b6a236a427713d1088550682be8 root@never-PAF4:~# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 47292b980e11 mysql "docker-entrypoint.s…" 5 seconds ago Up 4 seconds 0.0.0.0:3306->3306/tcp slave
进入镜像,修改配置文件,发现无法使用宿主机器的命令,选择构建新的镜像?还是将宿主机文件挂在容器中
root@never-PAF4:~# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES efb8f779ff41 mysql "docker-entrypoint.s…" 30 minutes ago Up 30 minutes 0.0.0.0:3305->3305/tcp, 3306/tcp master 47292b980e11 mysql "docker-entrypoint.s…" 31 minutes ago Up 31 minutes 0.0.0.0:3306->3306/tcp slave root@never-PAF4:~# docker exec -it master /bin/bash root@never-PAF4:~/data/mysql/master# docker run -p 3302:3306 --name ms4 -v ~/data/mysql/master/my.conf:/etc/mysql/my.cnf -e MYSQL_ROOT_PASSWORD=root -d mysql my.cnf root@never-PAF4:~/data/mysql/master# cat my.conf [mysqld] log-bin=mysql-bin binlog-format=ROW server_id=1 bind-address=0.0.0.0 root@never-PAF4:~/data/mysql/master#
监听的mysql服务器创建校验
root@never-PAF4:~/data/mysql/master# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES a7252065ea95 mysql "docker-entrypoint.s…" 2 minutes ago Up 2 minutes 0.0.0.0:3302->3306/tcp ms4 47292b980e11 mysql "docker-entrypoint.s…" About an hour ago Up About an hour 0.0.0.0:3306->3306/tcp slave root@never-PAF4:~/data/mysql/master# docker exec -it ms4 mysql -uroot -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 4 Server version: 5.7.21-log MySQL Community Server (GPL) Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> create user canal identified by 'canal'; Query OK, 0 rows affected (0.06 sec) mysql> grant select,replication slave,replication client on *.* to 'canal'@'%'; Query OK, 0 rows affected (0.05 sec) mysql> flush privileges -> -> ; Query OK, 0 rows affected (0.05 sec) mysql> mysql>
结果,binlog 发生变化,也就是所监听的数据库发生变化 canal 客户端就会监听到
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=96m; support was removed in 8.0 Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0 Java HotSpot(TM) 64-Bit Server VM warning: UseCMSCompactAtFullCollection is deprecated and will likely be removed in a future release. 2018-03-11 12:04:46.340 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## set default uncaught exception handler 2018-03-11 12:04:46.384 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## load canal configurations 2018-03-11 12:04:46.385 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## start the canal server. 2018-03-11 12:04:46.423 [main] INFO com.alibaba.otter.canal.deployer.CanalController - ## start the canal server[172.17.0.1:11111] 2018-03-11 12:04:46.838 [main] WARN o.s.beans.GenericTypeAwarePropertyDescriptor - Invalid JavaBean property 'connectionCharset' being accessed! Ambiguous write methods found next to actually used [public void com.alibaba.otter.canal.parse.inbound.mysql.AbstractMysqlEventParser.setConnectionCharset(java.nio.charset.Charset)]: [public void com.alibaba.otter.canal.parse.inbound.mysql.AbstractMysqlEventParser.setConnectionCharset(java.lang.String)] 2018-03-11 12:04:47.040 [main] ERROR com.alibaba.druid.pool.DruidDataSource - testWhileIdle is true, validationQuery not set 2018-03-11 12:04:47.321 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## the canal server is running now ...... 2018-03-11 12:04:48.222 [destination = example , address = /127.0.0.1:3302 , EventParser] WARN c.a.otter.canal.parse.inbound.mysql.MysqlEventParser - prepare to find start position just show master status 2018-03-11 12:53:51.976 [destination = example , address = /127.0.0.1:3302 , EventParser] WARN c.a.o.canal.parse.inbound.mysql.tsdb.DatabaseTableMeta - dup apply for sql : CREATE TABLE `a` ( `a` varchar(255) NULL )
相关文章推荐
- canal —— 阿里巴巴mysql数据库binlog的增量订阅&消费组件
- 【技术分享】mysql数据库binlog的增量订阅&消费组件:canal
- CanalSharp-mysql数据库binlog的增量订阅&消费组件Canal的.NET客户端
- 阿里巴巴开源项目: canal 基于mysql数据库binlog的增量订阅&消费
- [转]基于mysql数据库binlog的增量订阅&消费中间件:Canal
- 阿里巴巴开源项目: 基于mysql数据库binlog的增量订阅&消费
- 阿里巴巴开源项目: 基于mysql数据库binlog的增量订阅&消费
- 阿里巴巴开源项目: 基于mysql数据库binlog的增量订阅&消费
- 阿里巴巴开源项目: 基于mysql数据库binlog的增量订阅&消费
- 阿里巴巴开源项目: 基于mysql数据库binlog的增量订阅&消费
- 阿里巴巴开源项目: 基于mysql数据库binlog的增量订阅&消费
- 一个重大轮子: 基于mysql数据库binlog的增量订阅&消费
- 阿里巴巴开源项目: 基于mysql数据库binlog的增量订阅&消费
- 阿里巴巴开源项目: 基于mysql数据库binlog的增量订阅&消费
- 阿里巴巴开源项目: 基于mysql数据库binlog的增量订阅&消费
- 阿里巴巴开源项目: 基于mysql数据库binlog的增量订阅&消费
- 阿里巴巴开源项目: 基于mysql数据库binlog的增量订阅&消费
- 谈谈对Canal( 增量数据订阅与消费 )的理解
- canal的使用--日志增量订阅&消费
- MySQL增量订阅&消费组件Canal POC