在hadoop环境下用spark跑wordcount（没有安装scala）

在spark和hdfs上运行wordcount：

一、单机单节点安装spark：

1、解压

2、配置conf，cp，， spark-env.sh，写路径，sbin跑动

[root@localhost spark-1.6.1-bin-hadoop1]# cd sbin

[root@localhost sbin]# ls

slaves.sh start-slaves.sh

spark-config.sh start-thriftserver.sh

spark-daemon.sh stop-all.sh

spark-daemons.sh stop-history-server.sh

start-all.sh stop-master.sh

start-history-server.sh stop-mesos-dispatcher.sh

start-master.sh stop-mesos-shuffle-service.sh

start-mesos-dispatcher.sh stop-shuffle-service.sh

start-mesos-shuffle-service.sh stop-slave.sh

start-shuffle-service.sh stop-slaves.sh

start-slave.sh stop-thriftserver.sh

[root@localhost sbin]# start-all.sh

starting org.apache.spark.deploy.master.Master, logging to /usr/lib/spark/spark-1.6.1-bin-hadoop1/logs/spark-root-org.apache.spark.deploy.master.Master-1-localhost.localdomain.out

localhost: starting org.apache.spark.deploy.worker.Worker, logging to /usr/lib/spark/spark-1.6.1-bin-hadoop1/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-localhost.localdomain.out

localhost: failed to launch org.apache.spark.deploy.worker.Worker:

localhost: Spark Command: /usr/lib/jvm/jdk1.7.0_45/bin/java -cp /usr/lib/spark/spark-1.6.1-bin-hadoop1/conf/:/usr/lib/spark/spark-1.6.1-bin-hadoop1/lib/spark-assembly-1.6.1-hadoop1.2.1.jar:/usr/lib/spark/spark-1.6.1-bin-hadoop1/lib/datanucleus-rdbms-3.2.9.jar:/usr/lib/spark/spark-1.6.1-bin-hadoop1/lib/datanucleus-core-3.2.10.jar:/usr/lib/spark/spark-1.6.1-bin-hadoop1/lib/datanucleus-api-jdo-3.2.6.jar:/usr/lib/hadoop/hadoop-1.2.1/etc/hadoop
-Xms1g -Xmx1g -XX:MaxPermSize=256m org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://localhost.localdomain:7077

localhost: ========================================

localhost: full log in /usr/lib/spark/spark-1.6.1-bin-hadoop1/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-localhost.localdomain.out

[root@localhost sbin]# jps

4035

5268 Master

5354 Jps

5322 Worker

二、

再跑动hadoop

[root@localhost sbin]# cd ..

[root@localhost spark-1.6.1-bin-hadoop1]# start-all.sh

starting namenode, logging to /usr/lib/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-root-namenode-localhost.localdomain.out

localhost: starting datanode, logging to /usr/lib/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-root-datanode-localhost.localdomain.out

localhost: starting secondarynamenode, logging to /usr/lib/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-root-secondarynamenode-localhost.localdomain.out

starting jobtracker, logging to /usr/lib/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-root-jobtracker-localhost.localdomain.out

localhost: starting tasktracker, logging to /usr/lib/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-root-tasktracker-localhost.localdomain.out

[root@localhost spark-1.6.1-bin-hadoop1]# jps

4035

6364 TaskTracker

6432 Jps

6244 JobTracker

6064 DataNode

5268 Master

6166 SecondaryNameNode

5959 NameNode

5322 Worker

三、上传文档

[root@localhost hadoop-1.2.1]# hadoop fs -copyFromLocal README.txt input

四、

没有装scala，想用纯java环境跑动wordcount

（一）进入/bin/spark-shell

[root@localhost bin]# spark-shell

log4j:WARN No appenders could be found for logger (org.apache.hadoop.security.Groups).

log4j:WARN Please initialize the log4j system properly.

log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

Using Spark's repl log4j profile: org/apache/spark/log4j-defaults-repl.properties

To adjust logging level use sc.setLogLevel("INFO")

Welcome to

____ __

/ __/__ ___ _____/ /__

_\ \/ _ \/ _ `/ __/ '_/

/___/ .__/\_,_/_/ /_/\_\ version 1.6.1

/_/

Using Scala version 2.10.5 (Java HotSpot(TM) Client VM, Java 1.7.0_45)

Type in expressions to have them evaluated.

Type :help for more information.

16/04/29 08:03:24 WARN Utils: Your hostname, localhost.localdomain resolves to a loopback address: 127.0.0.1; using 192.168.86.135 instead (on interface eth0)

16/04/29 08:03:24 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address

Spark context available as sc.

16/04/29 08:03:57 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)

16/04/29 08:04:06 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)

16/04/29 08:04:28 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0

16/04/29 08:04:30 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException

16/04/29 08:05:06 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

16/04/29 08:05:23 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)

16/04/29 08:05:26 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)

SQL context available as sqlContext.

scala>

(二)指定运行文档

scala> val file=sc.textFile("hdfs://192.168.86.135:9000/user/root/input/README.txt")

16/04/29 08:07:31 WARN SizeEstimator: Failed to check whether UseCompressedOops is set; assuming yes

file: org.apache.spark.rdd.RDD[String] = hdfs://192.168.86.135:9000/user/root/input/README.txt MapPartitionsRDD[1] at textFile at <console>:27

（三）

进行运算输出格式编辑

scala> val count=file.flatMap(line => line.split(" ")).map(word => (word,1)).reduceByKey(_+_)

16/04/29 08:12:31 WARN LoadSnappy: Snappy native library not loaded

count: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[4] at reduceByKey at <console>:29

（四）

输出

scala> count.collect()

res0: Array[(String, Int)] = Array((Hadoop,1), (Commodity,1), (For,1), (this,3), (country,1), (under,1), (it,1), (The,4), (Jetty,1), (Software,2), (Technology,1), (<http://www.wassenaar.org/>,1), (have,1), (http://wiki.apache.org/hadoop/,1), (BIS,1), (classified,1),
(This,1), (following,1), (which,2), (security,1), (See,1), (encryption,3), (Number,1), (export,1), (reside,1), (for,3), ((BIS),,1), (any,1), (at:,2), (software,2), (makes,1), (algorithms.,1), (re-export,2), (latest,1), (your,1), (SSL,1), (the,8), (Administration,1),
(includes,2), (import,,2), (provides,1), (Unrestricted,1), (country's,1), (if,1), (740.13),1), (Commerce,,1), (country,,1), (software.,2), (concerning,1), (laws,,1), (source,1), (possession,,2), (Apache,1), (our,2), (written,1), (as,1), (License,1), (regulations,...

scala>