spark load file的几种方式

spark load file的几种方式：

1、直接导入localfile，而不是HDFS

sc.textFile("file:///path to the file/")

如sc.textFile("file:///home/spark/Desktop/README.md")

注意：

当设置了HADOOP_CONF_DIR的时候，即配置了集群环境的时候，如果直接sc.textFile("path/README.md")

路径会自动变成： hdfs://master:9000/user/spark/README.md
这个时候如果HDFS中没有，就会说，input path does not exist

2、给hdfs 的路径也可以

相关内容：

1、

Spark Quick Start - call to open README.md needs explicit fs prefix

Good catch; the Spark cluster on EC2 is configured to use HDFS as its default filesystem, so

it can’t find this file. The quick start was written to run on a single machine with an

out-of-the-box install. If you’d like to upload this file to the HDFS cluster on EC2, use

the following command:

2、

This has been discussed into spark mailing list, and please refer this mail.

You should use hadoop fs -put <localsrc> ... <dst> copy the file into hdfs:

${HADOOP_COMMON_HOME}/bin/hadoop fs -put /path/to/README.md README.md

于是我 /bin/hadoop -fs -put /home/spark/Desktop/README.md README.md
但这种方法怎么试都不行，说no such file or directory，还在研究