Spark生态之Alluxio学习25--spark从HDFS和Alluxio读取时间比较
2016-12-15 18:42
375 查看
更多代码请见:https://github.com/xubo245/SparkLearning
Spark生态之Alluxio学习 版本:alluxio-1.3.0(tachyon),spark-1.5.2,hadoop-2.6.0
参考
Spark生态之Alluxio学习 版本:alluxio-1.3.0(tachyon),spark-1.5.2,hadoop-2.6.0
1.解释
想要分析alluxio加速效果,发现alluxio会出现长尾效应,导致有些task特别耗时,相对于HDFS,并没有明显优势。
2.代码:
#~/cloud/alluxio-1.3.0/bin/alluxio fs mkdir -p /xubo/project/SparkSW/input/ for i in 1 2 3 4 5 6 7 8 9 10 do echo 'time:'$i for j in 1 2 3 4 5 6 7 8 9 10 do sh test.sh 10G hdfs://Master:9000/xubo/project/SparkSW/input/'D'$i'Line.fasta' hdfs://Master:9000/xubo/project/alluxio/output/TimeCount #sh test.sh 10G alluxio://Master:19998/xubo/alluxio/uniref/'D'$j'Line.fasta' hdfs://Master:9000/xubo/project/alluxio/output/TimeCount #sh test.sh 10G alluxio://Master:19998/xubo/project/SparkSW/input/'D'$j'Line.fasta' alluxio://Master:19998/xubo/project/alluxio/output/TimeCount done for j in 1 2 3 4 5 6 7 8 9 10 do sh test.sh 10G alluxio://Master:19998/xubo/project/SparkSW/input/'D'$i'Line.fasta' hdfs://Master:9000/xubo/project/alluxio/output/TimeCount done done
3.结果:
hadoop@Master:~/disk2/xubo/project/alluxio/timeCount$ tail -f runtime201611231839.txt time:1 input:hdfs://Master:9000/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:3.918s input:hdfs://Master:9000/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:3.923s input:hdfs://Master:9000/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:3.848s input:hdfs://Master:9000/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:3.959s input:hdfs://Master:9000/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:3.85s input:hdfs://Master:9000/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:3.816s input:hdfs://Master:9000/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:4.208s input:hdfs://Master:9000/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:3.925s input:hdfs://Master:9000/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:3.001s input:hdfs://Master:9000/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:2.817s input:alluxio://Master:19998/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:3.466s input:alluxio://Master:19998/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:2.83s input:alluxio://Master:19998/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:2.757s input:alluxio://Master:19998/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:2.824s input:alluxio://Master:19998/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:2.813s input:alluxio://Master:19998/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:2.802s input:alluxio://Master:19998/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:2.847s input:alluxio://Master:19998/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:2.88s input:alluxio://Master:19998/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:3.963s input:alluxio://Master:19998/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:2.788s time:2 input:hdfs://Master:9000/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:3.074s input:hdfs://Master:9000/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:2.945s input:hdfs://Master:9000/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:3.023s input:hdfs://Master:9000/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:3.999s input:hdfs://Master:9000/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:3.073s input:hdfs://Master:9000/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:4.206s input:hdfs://Master:9000/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:3.028s input:hdfs://Master:9000/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:4.049s input:hdfs://Master:9000/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:3.927s input:hdfs://Master:9000/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:3.964s input:alluxio://Master:19998/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:3.875s input:alluxio://Master:19998/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:6.505s input:alluxio://Master:19998/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:3.001s input:alluxio://Master:19998/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:3.276s input:alluxio://Master:19998/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:3.047s input:alluxio://Master:19998/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:3.081s input:alluxio://Master:19998/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:3.365s input:alluxio://Master:19998/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:3.083s input:alluxio://Master:19998/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:3.141s input:alluxio://Master:19998/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:3.027s time:3 input:hdfs://Master:9000/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:4.572s input:hdfs://Master:9000/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:4.45s input:hdfs://Master:9000/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:4.342s input:hdfs://Master:9000/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:4.336s input:hdfs://Master:9000/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:4.508s input:hdfs://Master:9000/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:4.268s input:hdfs://Master:9000/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:4.558s input:hdfs://Master:9000/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:4.401s input:hdfs://Master:9000/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:4.457s input:hdfs://Master:9000/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:4.513s input:alluxio://Master:19998/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:5.515s input:alluxio://Master:19998/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:4.846s input:alluxio://Master:19998/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:4.215s input:alluxio://Master:19998/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:4.242s input:alluxio://Master:19998/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:4.181s input:alluxio://Master:19998/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:4.236s input:alluxio://Master:19998/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:3.047s input:alluxio://Master:19998/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:4.178s input:alluxio://Master:19998/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:4.24s input:alluxio://Master:19998/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:4.178s time:4 input:hdfs://Master:9000/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:7.852s input:hdfs://Master:9000/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:6.148s input:hdfs://Master:9000/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:7.575s input:hdfs://Master:9000/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:16.612s input:hdfs://Master:9000/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:8.009s input:hdfs://Master:9000/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:8.02s input:hdfs://Master:9000/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:7.937s input:hdfs://Master:9000/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:8.007s input:hdfs://Master:9000/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:16.949s input:hdfs://Master:9000/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:7.795s input:alluxio://Master:19998/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:7.182s input:alluxio://Master:19998/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:4.506s input:alluxio://Master:19998/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:3.463s input:alluxio://Master:19998/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:3.501s input:alluxio://Master:19998/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:3.544s input:alluxio://Master:19998/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:3.747s input:alluxio://Master:19998/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:3.523s input:alluxio://Master:19998/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:3.4s input:alluxio://Master:19998/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:3.325s input:alluxio://Master:19998/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:3.519s time:5 input:hdfs://Master:9000/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:3.988s input:hdfs://Master:9000/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:4.004s input:hdfs://Master:9000/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:5.494s input:hdfs://Master:9000/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:4.916s input:hdfs://Master:9000/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:5.479s 16/11/23 18:56:20 WARN QueuedThreadPool: 4 threads could not be stopped input:hdfs://Master:9000/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:3.891s input:hdfs://Master:9000/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:4.73s input:hdfs://Master:9000/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:4.851s input:hdfs://Master:9000/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:4.9s input:hdfs://Master:9000/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:4.966s input:alluxio://Master:19998/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:7.924s input:alluxio://Master:19998/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:6.396s input:alluxio://Master:19998/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:3.487s input:alluxio://Master:19998/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:5.417s input:alluxio://Master:19998/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:4.272s input:alluxio://Master:19998/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:3.465s input:alluxio://Master:19998/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:3.437s input:alluxio://Master:19998/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:4.067s input:alluxio://Master:19998/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:6.194s input:alluxio://Master:19998/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:5.352s time:6 input:hdfs://Master:9000/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:4.883s input:hdfs://Master:9000/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:4.119s 16/11/23 18:59:47 WARN QueuedThreadPool: 4 threads could not be stopped input:hdfs://Master:9000/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:4.172s input:hdfs://Master:9000/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:5.09s input:hdfs://Master:9000/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:7.57s input:hdfs://Master:9000/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:4.578s input:hdfs://Master:9000/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:7.317s input:hdfs://Master:9000/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:4.857s 16/11/23 19:01:03 WARN QueuedThreadPool: 1 threads could not be stopped input:hdfs://Master:9000/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:4.854s 16/11/23 19:01:16 WARN QueuedThreadPool: 1 threads could not be stopped input:hdfs://Master:9000/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:3.994s input:alluxio://Master:19998/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:15.504s input:alluxio://Master:19998/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:6.336s input:alluxio://Master:19998/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:6.456s input:alluxio://Master:19998/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:4.951s 16/11/23 19:02:27 WARN QueuedThreadPool: 2 threads could not be stopped input:alluxio://Master:19998/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:5.525s input:alluxio://Master:19998/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:15.943s input:alluxio://Master:19998/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:4.665s input:alluxio://Master:19998/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:4.814s input:alluxio://Master:19998/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:15.965s input:alluxio://Master:19998/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:4.915s time:7 input:hdfs://Master:9000/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:9.814s input:hdfs://Master:9000/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:9.196s input:hdfs://Master:9000/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:9.434s input:hdfs://Master:9000/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:9.172s input:hdfs://Master:9000/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:20.193s input:hdfs://Master:9000/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:10.454s input:hdfs://Master:9000/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:9.551s input:hdfs://Master:9000/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:9.321s input:hdfs://Master:9000/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:10.313s input:hdfs://Master:9000/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:9.707s input:alluxio://Master:19998/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:52.331s input:alluxio://Master:19998/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:6.583s input:alluxio://Master:19998/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:16.998s input:alluxio://Master:19998/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:16.55s input:alluxio://Master:19998/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:6.496s input:alluxio://Master:19998/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:6.781s input:alluxio://Master:19998/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:7.477s input:alluxio://Master:19998/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:17.51s input:alluxio://Master:19998/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:6.494s input:alluxio://Master:19998/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:17.511s time:8 input:hdfs://Master:9000/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:10.427s input:hdfs://Master:9000/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:9.993s input:hdfs://Master:9000/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:11.242s input:hdfs://Master:9000/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:10.868s input:hdfs://Master:9000/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:7.354s input:hdfs://Master:9000/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:7.015s input:hdfs://Master:9000/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:7.344s input:hdfs://Master:9000/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:7.182s input:hdfs://Master:9000/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:7.262s input:hdfs://Master:9000/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:8.282s input:alluxio://Master:19998/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:42.406s input:alluxio://Master:19998/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:16.769s input:alluxio://Master:19998/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:18.197s input:alluxio://Master:19998/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:7.881s input:alluxio://Master:19998/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:8.913s input:alluxio://Master:19998/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:8.174s input:alluxio://Master:19998/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:20.971s input:alluxio://Master:19998/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:8.434s input:alluxio://Master:19998/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:8.316s input:alluxio://Master:19998/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:16.767s time:9 input:hdfs://Master:9000/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:32.624s input:hdfs://Master:9000/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:19.688s input:hdfs://Master:9000/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:17.345s input:hdfs://Master:9000/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:20.98s input:hdfs://Master:9000/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:13.478s input:hdfs://Master:9000/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:27.69s input:hdfs://Master:9000/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:15.567s input:hdfs://Master:9000/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:14.237s input:hdfs://Master:9000/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:16.413s input:hdfs://Master:9000/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:10.537s input:alluxio://Master:19998/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:99.776s input:alluxio://Master:19998/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:28.482s input:alluxio://Master:19998/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:29.184s input:alluxio://Master:19998/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:17.949s input:alluxio://Master:19998/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:17.29s input:alluxio://Master:19998/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:28.279s input:alluxio://Master:19998/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:30.892s input:alluxio://Master:19998/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:18.895s input:alluxio://Master:19998/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:28.351s input:alluxio://Master:19998/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:20.001s time:10 input:hdfs://Master:9000/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:38.227s input:hdfs://Master:9000/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:25.727s input:hdfs://Master:9000/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:23.691s input:hdfs://Master:9000/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:19.047s input:hdfs://Master:9000/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:19.611s input:hdfs://Master:9000/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:14.391s input:hdfs://Master:9000/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:16.036s input:hdfs://Master:9000/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:16.249s input:hdfs://Master:9000/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:15.324s input:hdfs://Master:9000/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:13.936s input:alluxio://Master:19998/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:151.861s input:alluxio://Master:19998/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:46.322s input:alluxio://Master:19998/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:31.237s input:alluxio://Master:19998/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:31.318s input:alluxio://Master:19998/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:19.097s input:alluxio://Master:19998/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:19.043s input:alluxio://Master:19998/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:30.663s input:alluxio://Master:19998/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:43.809s input:alluxio://Master:19998/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:20.463s input:alluxio://Master:19998/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:31.099s
参考
【1】https://github.com/Alluxio/alluxio 【2】http://www.alluxio.org/ 【3】http://spark.apache.org/docs/1.5.2/programming-guide.html 【4】https://github.com/xubo245/SparkLearning
相关文章推荐
- Spark生态之Alluxio学习24--分别读取HDFS和Alluxio的数据进行line count比较分析
- Spark生态之Alluxio学习19--alluxio性能提升分析与实验1
- Spark生态之Alluxio学习15--alluxio性能分析和加速方式
- Spark生态之Alluxio学习2---Spark从tachyon中读取文件
- Spark生态之Alluxio学习20--疑问:为什么D7-D9到最后的count时间都是30s左右?
- Spark生态之Alluxio学习14--alluxio内存文件加载方式和分布情况分析
- Spark生态之Alluxio学习3---机器重启后数据存储位置的变化
- Spark生态之Alluxio学习8---集群版搭建问题之集群无法全部启动
- Spark生态之Alluxio学习21--疑问:为什么master看不到D9,而work节点有D9部分数据?
- Spark生态之Alluxio学习4---下载源码通过maven install安装失败记录
- Spark生态之Alluxio学习16--将alluxio文件的block大小修改
- Spark生态之Alluxio学习22--saveAsTextFile alluxio后count (有bug)
- Spark生态之Alluxio学习18--alluxio work 3000出现问题
- Spark生态之Alluxio学习6---集群版搭建和运行(Tachyon)
- Spark生态之Alluxio学习10--集群问法全部启动问题解决
- Spark生态之Alluxio学习7--下载源码通过maven安装成功
- Spark生态之Alluxio学习12--spark调用alluxio-1.3.0配置
- Spark生态之Alluxio学习13--Alluxio-1.3.0启动疑问
- spark 读取alluxio: Caused by: java.io.IOException: Block 38416150429696 is not available in Alluxio
- Spark生态之Alluxio学习17--多次运行Space Usage增加