spark中读取elasticsearch数据
2015-05-14 15:41
363 查看
在spark中读取es的数据
pom.xml
esRDDTest.scala
pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.test</groupId> <artifactId>spark</artifactId> <version>0.0.1-SNAPSHOT</version> <packaging>jar</packaging> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <scala.version>2.11.6</scala.version> <scala.maven.version>2.11.6</scala.maven.version> </properties> <dependencies> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.10</artifactId> <version>1.3.1</version> </dependency> <!--<dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>2.6.0</version> </dependency>--> <dependency> <groupId>org.elasticsearch</groupId> <artifactId>elasticsearch-spark_2.10</artifactId> <version>2.1.0.Beta4</version> </dependency> </dependencies> <build> <sourceDirectory>src/main/scala</sourceDirectory> <testSourceDirectory>src/test/scala</testSourceDirectory> <plugins> <plugin> <groupId>org.scala-tools</groupId> <artifactId>maven-scala-plugin</artifactId> <version>2.15.2</version> <executions> <execution> <goals> <goal>compile</goal> <!--<goal>testCompile</goal>--> </goals> </execution> </executions> <configuration> <scalaVersion>${scala.version}</scalaVersion> </configuration> </plugin> <plugin> <artifactId>maven-assembly-plugin</artifactId> <version>2.4</version> <configuration> <skipTests>true</skipTests> <descriptorRefs> <descriptorRef>jar-with-dependencies</descriptorRef> </descriptorRefs> </configuration> <executions> <execution> <id>make-assembly</id> <phase>package</phase> <goals> <goal>single</goal> </goals> </execution> </executions> </plugin> </plugins> </build> </project>
esRDDTest.scala
package spark import org.apache.spark.SparkConf import org.apache.spark.SparkContext import org.apache.spark.SparkContext._ import org.elasticsearch.spark._ object esRDDTest { def main(args: Array[String]) { val conf = new SparkConf().setAppName("esRDDtest") conf.set("es.nodes", "127.0.0.1") conf.set("es.port", "8200") val sc = new SparkContext(conf) val resource = args(0) val query = args(1) val eslogs = sc.esRDD(resource, query) //…… } }
相关文章推荐
- spark中读取elasticsearch数据
- spark中读取elasticsearch数据
- sparksql 从oracle读取数据然后整合到elasticsearch
- Spark通过https的方式读取elasticsearch中的数据
- spark 读取elasticsearch中数据不完整问题
- 关于spark读取elasticsearch中数据,但是无法实现过滤数据的问题
- 24-java版Spark程序读取ElasticSearch数据
- Spark Streaming 读取Kafka数据写入Elasticsearch
- 使用spark访问elasticsearch的数据
- spark-sql(四)---读写elasticsearch数据
- Spark使用Java读取mysql数据和保存数据到mysql
- Spark在Hadoop的HDFS中读取数据
- spark 读取hbase数据并转化为dataFrame
- spark-shell读取外部数据源
- Spark读取csv数据
- Ssh框架与大数据平台(Spark)集成 读取mysql关系型数据库关键技术说明
- spark SQL (五)数据源 Data Source----json hive jdbc等数据的的读取与加载
- Spark读取Hbase中的数据
- Spark读取Hbase中的数据_云帆大数据分享
- spark join shuffle 数据文件的读取