ElasticSearch中使用reiver-jdbc从数据库导入数据
2015-03-29 00:00
1256 查看
摘要: ElasticSearch中使用reiver-jdbc从数据库导入数据
ElasticSearch中提供了River模块来从其他数据源中获取数据,该项功能以插件的形式存在,目前已有的River插件包括:
CouchDB River Plugin
RabbitMQ River Plugin
Twitter River Plugin
Wikipedia River Plugin
Supported by the community
ActiveMQ River Plugin (by Dominik Dorn)
Amazon SQS River Plugin (by Alex Bogdanovski)
CSV River Plugin (by Martin Bednar)
Dropbox River Plugin (by David Pilato)
FileSystem River Plugin (by David Pilato)
Git River Plugin (by Olivier Bazoud)
GitHub River Plugin (by uberVU)
Hazelcast River Plugin (by Steve Samuel)
JDBC River Plugin (by Jörg Prante)
JMS River Plugin (by Steve Sarandos)
Kafka River Plugin (by Endgame Inc.)
Kafka River Plugin 2 (by Mariam Hakobyan)
LDAP River Plugin (by Tanguy Leroux)
MongoDB River Plugin (by Richard Louapre)
Neo4j River Plugin (by Steve Samuel)
Open Archives Initiative (OAI) River Plugin (by Jörg Prante)
Redis River Plugin (by Steve Samuel)
RethinkDB River Plugin (by RethinkDB)
RSS River Plugin (by David Pilato)
Sofa River Plugin (by adamlofts)
Solr River Plugin (by Luca Cavanna)
St9 River Plugin (by Sunny Gleason)
Subversion River Plugin (by Pascal Lombard)
DynamoDB River Plugin (by Kevin Wang)
IMAP/POP3 Email River Plugin (by Hendrik Saly)
Web River Plugin (by CodeLibs Project)
EEA ElasticSearch RDF River Plugin (by the European Environment Agency)
Amazon S3 River Plugin (by Laurent Broudoux)
Google Drive River Plugin (by Laurent Broudoux)
可以看出,已经覆盖了大部分的数据源,特别是针对关系型数据库提供了统一的jdbc-river来进行数据操作。elasticsearch-river-jdbc的源码在:github.com/jprante/elasticsearch-river-jdbc,该项目提供了详细的文档,下面以SQL Server为例简单说明使用方法。
首先,需要安装elasticsearch-river-jdbc,在elasticsearch目录下执行:
./bin/plugin --install jdbc --url http://xbib.org/repository/org/xbib/elasticsearch/plugin/elasticsearch-river-jdbc/1.5.0.0/elasticsearch-river-jdbc-1.5.0.0.zip
然后,安装SQLServer的JDBC库,链接为: Microsoft JDBC Driver.把其中的 'sqljdbc4.jar'复制到elasticsearch安装目录的lib文件夹下。
考虑到elasticsearch集群,以上两个步骤在每个节点上都需要执行。
最后也是最关键的一步,在elasticsearch中建立river,让elasticsearch自动从SQLServer中获取数据。
PUT /_river/mytest_river/_meta
{
"type" : "jdbc",
"jdbc" : {
"driver":"com.microsoft.sqlserver.jdbc.SQLServerDriver",
"url":"jdbc:sqlserver://MYSQLSERVERNAME;databaseName=MYProductDatabase",
"user":"admin","password":"Password",
"sql":"select ProductID as _id, CategoryID,ManufacturerID,MfName,ProductTitle,MfgPartNumber from MyProductsTable(nolock)",
"poll":"10m",
"strategy" : "simple",
"index" : "myinventory",
"type" : "product",
"bulk_size" : 100,
"max_retries": 5,
"max_retries_wait":"30s",
"max_bulk_requests" : 5,
"bulk_flush_interval" : "5s"
}
}
ElasticSearch中提供了River模块来从其他数据源中获取数据,该项功能以插件的形式存在,目前已有的River插件包括:
River Pluginsedit
Supported by ElasticsearchCouchDB River Plugin
RabbitMQ River Plugin
Twitter River Plugin
Wikipedia River Plugin
Supported by the community
ActiveMQ River Plugin (by Dominik Dorn)
Amazon SQS River Plugin (by Alex Bogdanovski)
CSV River Plugin (by Martin Bednar)
Dropbox River Plugin (by David Pilato)
FileSystem River Plugin (by David Pilato)
Git River Plugin (by Olivier Bazoud)
GitHub River Plugin (by uberVU)
Hazelcast River Plugin (by Steve Samuel)
JDBC River Plugin (by Jörg Prante)
JMS River Plugin (by Steve Sarandos)
Kafka River Plugin (by Endgame Inc.)
Kafka River Plugin 2 (by Mariam Hakobyan)
LDAP River Plugin (by Tanguy Leroux)
MongoDB River Plugin (by Richard Louapre)
Neo4j River Plugin (by Steve Samuel)
Open Archives Initiative (OAI) River Plugin (by Jörg Prante)
Redis River Plugin (by Steve Samuel)
RethinkDB River Plugin (by RethinkDB)
RSS River Plugin (by David Pilato)
Sofa River Plugin (by adamlofts)
Solr River Plugin (by Luca Cavanna)
St9 River Plugin (by Sunny Gleason)
Subversion River Plugin (by Pascal Lombard)
DynamoDB River Plugin (by Kevin Wang)
IMAP/POP3 Email River Plugin (by Hendrik Saly)
Web River Plugin (by CodeLibs Project)
EEA ElasticSearch RDF River Plugin (by the European Environment Agency)
Amazon S3 River Plugin (by Laurent Broudoux)
Google Drive River Plugin (by Laurent Broudoux)
可以看出,已经覆盖了大部分的数据源,特别是针对关系型数据库提供了统一的jdbc-river来进行数据操作。elasticsearch-river-jdbc的源码在:github.com/jprante/elasticsearch-river-jdbc,该项目提供了详细的文档,下面以SQL Server为例简单说明使用方法。
首先,需要安装elasticsearch-river-jdbc,在elasticsearch目录下执行:
./bin/plugin --install jdbc --url http://xbib.org/repository/org/xbib/elasticsearch/plugin/elasticsearch-river-jdbc/1.5.0.0/elasticsearch-river-jdbc-1.5.0.0.zip
然后,安装SQLServer的JDBC库,链接为: Microsoft JDBC Driver.把其中的 'sqljdbc4.jar'复制到elasticsearch安装目录的lib文件夹下。
考虑到elasticsearch集群,以上两个步骤在每个节点上都需要执行。
最后也是最关键的一步,在elasticsearch中建立river,让elasticsearch自动从SQLServer中获取数据。
PUT /_river/mytest_river/_meta
{
"type" : "jdbc",
"jdbc" : {
"driver":"com.microsoft.sqlserver.jdbc.SQLServerDriver",
"url":"jdbc:sqlserver://MYSQLSERVERNAME;databaseName=MYProductDatabase",
"user":"admin","password":"Password",
"sql":"select ProductID as _id, CategoryID,ManufacturerID,MfName,ProductTitle,MfgPartNumber from MyProductsTable(nolock)",
"poll":"10m",
"strategy" : "simple",
"index" : "myinventory",
"type" : "product",
"bulk_size" : 100,
"max_retries": 5,
"max_retries_wait":"30s",
"max_bulk_requests" : 5,
"bulk_flush_interval" : "5s"
}
}
相关文章推荐
- Elasticsearch中使用reiver-jdbc导入数据
- Elasticsearch中使用reiver-jdbc导入数据
- elasticsearch-jdbc 使用数据库数据创建索引
- 使用C#将数据库中的数据导入Excel2003
- 使用OLEDB将数据库的数据导入Excel文件
- ElasticSearch使用jdbc-river同步数据库出现dateOptionalTime不能识别的问题解决办法一则
- Java 使用poi把数据库中数据导入Excel的解决方法
- 使用回调函数来实现jdbc操作数据库数据
- Oracle txt文件直接使用sqldlr将数据导入到数据库表中
- 使用数据泵expdp命令和impdp命令对数据库的数据进行导入导出
- 《项目经验》--简单三层使用DataTable向数据库表批量导入数据---向SqlServer一张表中导入数据
- Java jdbc将sql数据导入数据库
- php中使用ExcelFileParser处理excel获得数据(可作批量导入到数据库使用)
- 使用Java 的jxl 批量导入数据到数据库
- 使用navicat 8实现创建数据库和导入数据 管理用户与权限[图文方法]
- 《项目经验》--简单三层使用DataTable向数据库表批量导入数据---向SqlServer多张张表中导入数据
- 使用navicat8实现创建数据库和导入数据[图文]
- oracle中,使用sqlldr将文件中的数据导入到数据库
- 在oracle中,使用sqlldr将文件中的数据导入到数据库
- 使用OPENDATASOURCE和OPENROWSET在不同类型的数据库之间导入导出数据