基因数据处理57之BWA-MEM运行single-end(1千万条100bp的reads)
2017-12-18 23:47
1411 查看
``` hadoop@Master:~/cloud/adam/xubo/data/GRCH38Sub/cs-bwamem$ bwa mem GRCH38BWAindex/GRCH38chr1L3556522.fasta g38L100c10000000Nhs20.fq > g38L100c10000000Nhs20.bwamem.sam [M::bwa_idx_load_from_disk] read 0 ALT contigs [M::process] read 100000 sequences (10000000 bp)... [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.827 CPU sec, 20.815 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.876 CPU sec, 20.775 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 22.159 CPU sec, 22.063 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.952 CPU sec, 21.857 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.909 CPU sec, 21.797 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.732 CPU sec, 20.628 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.763 CPU sec, 20.657 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.511 CPU sec, 20.405 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.114 CPU sec, 21.001 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 22.102 CPU sec, 22.016 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.821 CPU sec, 21.708 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 22.081 CPU sec, 21.978 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.888 CPU sec, 20.791 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.097 CPU sec, 21.004 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.717 CPU sec, 21.614 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.871 CPU sec, 21.782 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 22.280 CPU sec, 22.179 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 22.211 CPU sec, 22.113 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.901 CPU sec, 20.804 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.728 CPU sec, 21.648 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.704 CPU sec, 21.596 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 22.287 CPU sec, 22.188 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.724 CPU sec, 21.609 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.171 CPU sec, 21.084 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.989 CPU sec, 20.878 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.866 CPU sec, 20.771 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.136 CPU sec, 21.037 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.849 CPU sec, 20.741 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.833 CPU sec, 21.729 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.794 CPU sec, 21.694 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.526 CPU sec, 20.424 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.460 CPU sec, 20.355 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.228 CPU sec, 21.115 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 22.191 CPU sec, 22.102 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.857 CPU sec, 21.742 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.860 CPU sec, 20.771 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.009 CPU sec, 20.895 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.996 CPU sec, 20.891 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.845 CPU sec, 21.748 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.937 CPU sec, 21.843 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.731 CPU sec, 20.624 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.741 CPU sec, 20.636 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.906 CPU sec, 21.818 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.945 CPU sec, 21.845 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.902 CPU sec, 21.790 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.832 CPU sec, 20.739 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.842 CPU sec, 20.742 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.651 CPU sec, 20.553 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.574 CPU sec, 20.457 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.832 CPU sec, 20.737 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.767 CPU sec, 21.673 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.862 CPU sec, 21.759 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.979 CPU sec, 21.885 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.832 CPU sec, 21.751 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.003 CPU sec, 20.897 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.735 CPU sec, 20.629 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.845 CPU sec, 20.731 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.683 CPU sec, 20.579 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.907 CPU sec, 20.794 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.862 CPU sec, 21.772 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.971 CPU sec, 20.862 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 22.224 CPU sec, 22.138 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.741 CPU sec, 20.634 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.083 CPU sec, 20.982 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.823 CPU sec, 20.708 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.709 CPU sec, 21.618 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 22.415 CPU sec, 22.318 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.697 CPU sec, 20.598 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.518 CPU sec, 20.401 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.708 CPU sec, 20.603 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.118 CPU sec, 21.009 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.871 CPU sec, 20.766 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.978 CPU sec, 20.870 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.396 CPU sec, 20.288 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.998 CPU sec, 21.901 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.119 CPU sec, 21.026 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.878 CPU sec, 20.771 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.180 CPU sec, 21.086 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 22.058 CPU sec, 21.955 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.019 CPU sec, 20.934 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.577 CPU sec, 20.472 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.847 CPU sec, 20.741 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.248 CPU sec, 21.135 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.024 CPU sec, 20.940 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.932 CPU sec, 20.825 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 20.660 CPU sec, 20.556 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.877 CPU sec, 21.780 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 22.084 CPU sec, 21.981 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.708 CPU sec, 21.596 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.616 CPU sec, 21.547 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_process_seqs] Processed 100000 reads in 21.034 CPU sec, 20.935 real sec [M::process] read 57464 sequences (5746400 bp)... [M::mem_process_seqs] Processed 100000 reads in 22.099 CPU sec, 22.032 real sec [M::mem_process_seqs] Processed 57464 reads in 11.989 CPU sec, 11.941 real sec [main] Version: 0.7.13-r1126 [main] CMD: bwa mem GRCH38BWAindex/GRCH38chr1L3556522.fasta g38L100c10000000Nhs20.fq [main] Real time: 1964.274 sec; CPU: 1972.944 sec ```
参考
【1】https://github.com/xubo245/AdamLearning 【2】https://github.com/bigdatagenomics/adam/ 【3】https://github.com/xubo245/SparkLearning 【4】http://spark.apache.org 【5】http://stackoverflow.com/questions/28166667/how-to-pass-d-parameter-or-environment-variable-to-spark-job 【6】http://stackoverflow.com/questions/28840438/how-to-override-sparks-log4j-properties-per-driver
研究成果：
【1】 [BIBM] Bo Xu, Changlong Li, Hang Zhuang, Jiali Wang, Qingfeng Wang, Chao Wang, and Xuehai Zhou, "Distributed Gene Clinical Decision Support System Based on Cloud Computing", in IEEE International Conference on Bioinformatics and Biomedicine. (BIBM 2017, CCF B) 【2】 [IEEE CLOUD] Bo Xu, Changlong Li, Hang Zhuang, Jiali Wang, Qingfeng Wang, Xuehai Zhou. Efficient Distributed Smith-Waterman Algorithm Based on Apache Spark (CLOUD 2017, CCF-C). 【3】 [CCGrid] Bo Xu, Changlong Li, Hang Zhuang, Jiali Wang, Qingfeng Wang, Jinhong Zhou, Xuehai Zhou. DSA: Scalable Distributed Sequence Alignment System Using SIMD Instructions. (CCGrid 2017, CCF-C). 【4】more: https://github.com/xubo245/Publications
Help
If you have any questions or suggestions, please write it in the issue of this project or send an e-mail to me: xubo245@mail.ustc.edu.cn Wechat: xu601450868 QQ: 601450868
内容来自用户分享和网络整理，不保证内容的准确性，如有侵权内容，可联系管理员处理 
相关文章推荐
- 基因数据处理54之bwa-mem运行paird-end（1千万条100bp的reads）
- 基因数据处理60之bwa运行single-end（1千万条100bp的reads）
- 基因数据处理59之snap运行single-end（1千万条100bp的reads）
- 基因数据处理56之bwa运行paird-end（1千万条100bp的reads）.md
- 基因数据处理53之cs-bwamem集群版运行paird-end（1千万条100bp的reads）
- 基因数据处理58之snap运行paired-end（1千万条100bp的reads对）
- 基因数据处理61之idea运行cs-bwamem处理single-end（1条100bp的reads）
- 基因数据处理106之bwa-mem运行paird-end（1千万条100bp的reads g38L100c10000000Nhs20Paired12）
- 基因数据处理102之SparkBWA本地运行100万条paired-reads实例
- 基因数据处理7之BWA_MEM运行太长
- 基因数据处理50之cs-bwamem、bwa、snap、bwa-mem与art比较
- 基因数据处理95之sparkBWA运行问题
- 基因数据处理96之sparkBWA运行问题（yarn）
- 基因数据处理105之SparkBWAYarn模式运行1000万条paired-reads实例g38L100c10000000Nhs20Paired12YarnPartition
- 基因数据处理14之BWA三种方式bwa、BWA_SW、BWA_MEM使用
- 基因数据处理52之cs-bwamem集群版运行（1千万条100bp的reads）
- 基因数据处理15之scala对BWA运行结果进行时间提取
- 基因数据处理8之BWA_MEM小数据集处理（成功）
- 基因数据处理17之使用scala对BWA运行结果进行各阶段程序时间提取和统计求和
- 基因数据处理6之BWA_MEM无法分配内存