Hadoop基准测试

其实就是从网络上copy的吧，在这里做一下记录

这个是看一下有哪些测试方式：

hadoop jar /opt/cloudera/parcels/CDH-5.3.6-1.cdh5.3.6.p0.11/jars/hadoop-mapreduce-client-jobclient-2.5.0-cdh5.3.6-tests.jar

测试hadoop写的速度

向HDFS文件系统中写入数据,30个文件,每个文件100MB,文件存放到/benchmarks/TestDFSIO/io_data中

hadoop jar /opt/cloudera/parcels/CDH-5.3.6-1.cdh5.3.6.p0.11/jars/hadoop-mapreduce-client-jobclient-2.5.0-cdh5.3.6-tests.jar TestDFSIO -write -nrFiles 30 -fileSize 100MB

然后查看结果：cat TestDFSIO_results.log

我的集群基准测试结果如下 做了两次，有两个不同的结果，从结果上来看，为什么变化这么大：

----- TestDFSIO ----- : write
Date & time: Thu Sep 17 16:45:03 CST 2015
Number of files: 10
Total MBytes processed: 100.0
Throughput mb/sec: 27.51031636863824
Average IO rate mb/sec: 30.240123748779297
IO rate std deviation: 8.554948120135029
Test exec time sec: 30.227

----- TestDFSIO ----- : write
Date & time: Thu Sep 17 16:49:53 CST 2015
Number of files: 30
Total MBytes processed: 3000.0
Throughput mb/sec: 7.770168768065642
Average IO rate mb/sec: 8.027955055236816
IO rate std deviation: 1.629595948634101
Test exec time sec: 41.057

测试一下读的速度

hadoop jar /opt/cloudera/parcels/CDH-5.3.6-1.cdh5.3.6.p0.11/jars/hadoop-mapreduce-client-jobclient-2.5.0-cdh5.3.6-tests.jar TestDFSIO -read -nrFiles 30 -fileSize 100MB

结果如下：

----- TestDFSIO ----- : read
Date & time: Thu Sep 17 16:55:26 CST 2015
Number of files: 30
Total MBytes processed: 3000.0
Throughput mb/sec: 55.33115697449234
Average IO rate mb/sec: 215.3984375
IO rate std deviation: 181.40860904339297
Test exec time sec: 27.108

清除一下测试数据：

hadoop jar /opt/cloudera/parcels/CDH-5.3.6-1.cdh5.3.6.p0.11/jars/hadoop-mapreduce-client-jobclient-2.5.0-cdh5.3.6-tests.jar TestDFSIO -clean